Thread 105680498

>website has 100 pages
>cf showing 10,000 unique hits per hour, increasing
these are bots. but why are they bots? what the fuck is going on

Jeets programming botnet scrapers, getting rewarded by jeet managers for scrapes per minute.

>>105680598
>botnet scrapers
what is that
> rewarded by jeet managers for scrapes per minute
i could probably do it better than they can how do i do this for a job

>>105680598
>scraper
ok. so at 100 pages, my entire website is being scraped 72,000 times over every month? perpetually? are there 72,000 different scrapers all looking at my important website?

>>105680498 (OP)
People's minds have been uploaded to the matrix but most people were robots anyway so now they are more robots than humans.

>>105680754
there's no universe in which matrix robots care this mouth about my website

>>105680498 (OP)
What are you going to do about it?

>>105680498 (OP)
AI garbage looking for useful content.

>>105680498 (OP)
Your site is being scraped by AI companies to feed their LLMs.
Block them with Anubis: https://github.com/TecharoHQ/anubis

>>105680823
scrapers looking to feed their AI garbage*

fixed.

>>105680819
PoW seems to be the only effective tool in slowing it down, sort of like Xe's Anubis.

why not DDoS the entire web so that everybody puts up annoying captchas so people end up using ChatGPT instead of searching?

>>105680829
>>105680823
>>105680831
>its scrapers!
72,000 times they scrape my whole site every month. i can't understand this philosophy of scraping im sorry i don't get it

>>105680860
literally:
a) chink botnets looking for backdoors (all the /phpMyAdmin like requests)
b) AI companies desperate for training data
c) some faggot doing a research paper for his bachelors' thesis or some other edu institution.

look son you're doing great just make sure we keep scraping that small website with the erotic pokemon fanfiction 24 hours a day. put more servers on it

>>105680860
They look for updates in the pages.
If there is updated text they will save the new text.

>>105680889
if they did this once a month, there would still be 72,000 of them

>>105680829
>enbyware
>CoCked

At least it's MIT

>>105680893
Jeet coders. They are not efficient at scraping.
Let's say there are 50 AI companies, they all scrape 1 time a day. If there are 72 000 requests a month:
72000 ÷ 30 = 2400 requests a day.
2400 ÷ 50 = 48.
An average of 48 requests per day per AI company is not that much desu.

Use Anubis, it's not hard to set up. If you use a load balancer deploy it there.

>>105680936
Yeah, the author is a tranny but at least he built something useful.

>>105680860
it's called being based and making webshits seethe. if you can't deal with the ancient problem of "HIGH INCOMING TRAFFIC" then you don't deserve to host a website.

>>105680726
>what is that
Some scrapers use rented IPs ranges for scraping so they don't get blocked. It's basically a botnet, almost impossible to block.

>>105680893
>if they did this once a month, there would still be 72,000 of them
You underestimate their incompetence.

>>105681048
incompetence is fine but that's a waste of money

Holy shit enable ads and make some money.

>>105681069
yea i should do it. i bet the thousands of scraper bots would click on all the ads and make me rich

>>105680860
it's AI scrapers. they hit up everything they can as frequently as they can to harvest data for LLM training. It gets worse wen your site has "expensive" endpoints (e.g. git blame on the web). They do it a few times a day: https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html

>>105680829
>may inhibit "good bots" like the Internet Arc
ah that won't work in my case unfortunately. need to be friendly to search engines and IA

>>105681625
I believe Internet Archive is whitelisted by default now via their hardcoded IP addresses.

>>105680856
yea

>>105680829
lmao another snake oil "solution" that can be circumvented with few lines of code?

>>105681042
You can block ASN IP blocks who belong to cheap VPS services, the lists are publicly available just be sure to update them from time to time, or set a cron job, pull the list from an API and update the ranges automatically.

>>105682633
Seems to be working.
Git forge instances and wiki instances are the most affected and they all praise how well Anubis is protecting their services.
https://anubis.techaro.lol/docs/user/known-instances

>>105680498 (OP)
240k hits a day?
Tf are you running, userbenchmark?
Just get adsense and get yourself a threadripper