Thread 8754302

This is a thread for those interested in downloading NSFW content for local consumption.

Here we share tools, scripts and tips on how to download the content and organize it.

>How do i find the best sites?
https://everythingmoe.com/ (click on NSFW)


Some tools:
>gallery-dl
https://github.com/mikf/gallery-dl

>Grabber
https://github.com/Bionus/imgbrd-grabber

>yt-dlp
https://github.com/yt-dlp/yt-dlp


>why not make this thread on /g/?
/g/ had a series thread named "coom/g/". But the janitors delete any attempt on making this now.


I start by asking a question: Mangayomi or Suwayomi for desktop tachiyomi-based reader?

>>8754302 (OP)
Anything for mass conversion of webp into civilized formats?

>>8754570
it's trivial to script tho

>>8754302 (OP)
Does anyone know of a way to grab Irodori's third-party titles for offline viewing? It's for those doujins/manga that requires you to view it on their website instead of downloading a DRM-free copy.

>>8754570
You can probably do it with ImageMagick. I know webp is a supported format so it shouldn't be an issue.

Gotta plug Hydrus Network here, probably the best scraping and tagging solution for coom (posting image to keep it /h/)

https://hydrusnetwork.github.io/hydrus/index.html

>>8754570
Xnconvert if you want a GUI
A script that uses FFMPEG or ImageMagick if you want a CLI

>>8754653
>Irodori
If gallery-dl doesn't have it, you're unlikely to find it anywhere else. You have to write a scrapper for a unsupported site. I used gemini-cli to write a scrapper. It smashes through python.

>>8754302 (OP)
I've been using Suwayomi. You can also use Suwayomi as a server so you can connect it to Mihon/Tachiyomi on your phone. Also if you don't like the built-in Suwayomi frontend then you can use others.

>>8755110
Is there a difference between using Gemini-CLI versus just using Gemini straight from Google's website?

>>8755490
Gemini-CLI is a agent. It runs on the console and it's better for programming. You get 1000 daily requests (i hardly go over 100 in 1-2 hours).
>create a folder
>copy some examples/code/repositories it can use as reference to that folder
>write a GEMINI.MD file with general instructions
>run and tell it to follow GEMINI file instructions

Ever since bakabt i haven't downloaded hentai
whats are the best options these days

I want to download stuff from the following sites:
Rule34xxx
DeviantArt
Kemono
Hitomi.la
Ex-Hentai
Pixiv

Since Rule34xxx is a booru, i can just use Hydrus Network. But what about the others?

Must contain the following:
>Process a Url list one by one
So i can just keep a list of URLs and run a script to fetch them.

>Skip already downloaded files/galleries by checking a sqlite db
A sqlite db that stores the metadata from the galleries/files, such as tags, artists, etc. So that i could use it to rename the folders/files later by using a script.

>Stop fetching more if it skips X amount of galleries/files
e.g. if i search a tag in Hitomi.la, and it downloads thousands of pages, next time it process that i don't want it to go through thousands of pages again.

>A GUI
Something like hydrus that keep track of tags + thumbnails.

>Customizable folder/file naming
e.g.
[Artist]/
[Artist] (Release Date) Gallery Title [Language] (Site Gallery ID)/
[Artist] (Release Date) Gallery Title [Language] (Site Gallery ID) (1 of 20).jpg
[Artist] (Release Date) Gallery Title [Language] (Site Gallery ID) (2 of 20).jpg


I was thinking about Suwayomi for Hitomi.la, Ex-Hentai and Kemono. It has a GUI with thumbnails and tags, and i can read it from PC and phone. But it would be a pain to fetch new content since i would need to add it manually.

>>8755989
sukebei.nyaa.si

>>8755477
I came to the same conclusion after using both. Suwayomi came out on top by ease of set up and usage.

>>8754302 (OP)
https://github.com/ififfy/flipflip

>>8755994
After some researching and asking around i found that gallery-dl is the perfect tool for the job:
>skips already downloaded files
>dumps json with the metadata (no need to download the image/video files, it can skip them)
>aborts fetching if it skips X items
I just need to download the metadata, convert it into a sqlite database with a script, then make another script to query the database and build the gallery-dl command for downloading. Then make another script to create a Suwayomi/Tachiyomi xml file for each gallery, and now trere's a UI to read things from! It's more complicated than just downloading, but being able to use a database with sql queries is worth it. I post the scripts here later.

>>8755928
I tried that but eventually it figured out the site was blocked by Cloudflare and it couldn't login to my account. It straight up just said it doesn't know how to bypass it.

Well, I guess we're SOL until someone more competent takes over. Thanks for the help anyway! Didn't know such a thing existed and it's pretty cool.

Do you guys know any slideshow software or something that can help me pull out images at random?
I don't have so few images saved that I could just remember every single one of them, so I'd like some way of surprising myself with thing I've forgotten.

>>8757460 >>8756846

>>8757071
2 things you can do:
1. Tell it to make the scrapper using selenium in python, and to use cookies to try to bypass cloudflare.
2. Tell it to make a violentmonkey script to scrape the current page you're viewing. Since you're logged in and can view the content, it should be easier. You may need to dump the html files for that page (right click > save page as..) in the project directory, so that it may use them as a example.

>>8756880
gallery-dl saves metadata in a retarded way because it re-downloads the same metadata.

>>8757522
Damn, it turns out clicking random 4chan links with no description is actually pretty helpful.

i need something to download all my pixiv bookmakrs, cuz of the threat of the recent changes
can anyone help?

>>8756846
thanks

>>8755989
>bakabt
that's a name i haven't heard in a long time

>>8754570
https://github.com/JacobDev1/xl-converter

you can obviously also save size by using lossless jpeg xl with this

tks for the links
Do you know where I can find mirrors from vip and secrethentai sites ? Went to the everthingmoe and they don't list them
They were taken down a while ago and they don't seem to have any more valid sites

Or at least any site that mirrored or dumped their older posts
Pure L hentai has newer stuff, but the older stuff that appeared only on those sites that vanished and could not find any real sites
Some crap domains with terms like that only open scam ads and dating shit

Tks

>>8759195
Gallery-dl

>>8762125
>secrethentai
Do you mean ex-hentai? Because that's where everything that gets pruned from e-hentai goes. There's also hitomi.la, which just clones whatever is posted on ex-hentai.

>>8762429
Actually not from those, they used to look like this
Found some stuff similar but it was a site with paid password and crap/less content and the other one had some premium file hosting stuff, but mostly posts had a lot of images
They were like that

>>8754791
thanks

.

.

>>8754570
Brother this is so trivial that even an LLM can get it right on the first try.

..

I'm using HDoujin DL, and I managed to get it to work for sadpanda, but for some reason it won't download from Mangadex or any site like that.
What am I doing wrong?

>>8754302 (OP)
>Grabber
It's a shame Rule34xxx stopped working with Grabber recently

>>8764797
Must be a bug. Make a bug report or use something else like Suwayomi.

...