Thread 105985070

Whats the policy on scraping 4chan? I'm getting into the hobby of using ai to transcribe long documentaries for me so I can summarize them with AI and make study guides with NotebookLLM. I want to do the same for posts on /x/ (I'm studying occultism in religion), I only want to scrape text, not media.

>Summarizing with AI
>Making study guys with an LLM

>>105985084
guides*

>>105985084
some videos just ramble on for too long.

>>105985070 (OP)
idk learn to code its easy. scrape the archive not 4chan obviously.

>>105985070 (OP)
#14 here https://4chan.org/rules#global
says that language is prohibited, so perhaps use this instead: https://github.com/4chan/4chan-API

>>105985107
>>105985098
appreciate it

>>105985107
Here is a simple script in Python modified by another the REAL Jordan used to find the desktop thread here on /g/. You could probably repurpose it to find threads on /x/ that fit your requirements, then download those threads using the API with another script using curl or wget. You can easily ignore images, but many times the image is important and adds needed context to posts, so consider including them.

>>105985143
I'll find downloading images will quickly use up all my storage. In the future I will

>>105985143
thank you btw

>>105985183
>>105985113
You are very welcome, and I wish you success no matter how you chose to go ahead.

>>105985245
>pic rel
??? who the fuck do you think youre talking to lol

>>pic rel
>??? who the fuck do you think youre talking to lo

>>105985558
The speech bubble on the stick really shows a level of genius I didn't expect.

I invested 100$ in palantir

>>105985070 (OP)
>what's the policy on
Who gives a fuck you stupid fucking retard faggot

Ive used both the frontend when I was a retard and the api and was slamming tons of reqs/s and didnt get banned, judging by the 4chan leaks they dont meter for that

>>105985898
>old ass imageboard with 2004 ui doesnt keep up with the tech
waow really tell us more nostradamus. Dog.

>>105986153
yes