Thread 105739969

Why does stackoverflow and github now block AI scraping attempts when their entire body of text has already been used as training material by all the major LLMs and no new stackoverflow questions have been allowed on the site since 2013?

>>105739969 (OP)
To virtue signal of course

>>105739969 (OP)
To prevent new and upcoming independent LLMs from scraping them too so only the big dogs can benefit.

>>105739969 (OP)
Because llm scraping is basically a ddos

All the big boys already got all the data they need you are not allowed to compete, be faster next time

why would you scrape a site that has its entire database dump uploaded to archive.org in the first place?

>>105740042
SO could release an archive torrent of all the old posts, but that's not the real reason they block scaping

>>105740201
It should be mandated by law to make archives of all public posts easily accessible to everyone.

>>105741158
I don't like mandating someone has to do more work. This should only be required if you implement anti scraping measures

>>105741504
Don't like it? Don't host a website where 100% of your data is user generated.

>>105741517
Or you could stop being lazy. I'm basically a no coder and whipped up a web scraper ina couple of days. Luckily the site's I was going after work without JS

>>105741739
So don't cry about web scraping then, subhuman.

>>105741846
Some websites make it nearly impossible to scrape. In those cases I support easily accessible archive. Big sites don't want people accessing that data all, cause in their eyes data = money

>>105741923
No such thing as unscrapeable website.

>>105741960
You have horrible reading comprehension.
You are either:
>1. a retard
>2. esl
>3. a paid disruptor/bot

>>105742008
I accept your concession.

>>105739969 (OP)
Because it's a pay to play market, fuck-o

>>105739969 (OP)
You can easily bypass that using kiwix.
Online kiwix library at https://browse.library.kiwix.org/viewer#stackoverflow.com_en_all_2023-11/questions
...Or install it locally, be aware the entire stackoverflow weights 80Gb

How do they even stop scrapers.. why wouldn't you just write a bot with human characteristics ie sleep times.

>>105743750
You just add a small proof of work cryptominer on each page load small enough that people will tolerate it but significant enough that you are essentially paying to scrape

>>105743706
>the entire stackoverflow weights 80Gb
I wonder how much it is once you delete all comments (worthless), HTML (non-information) and all accepted answers (always wrong)

bump