>>105589471
>scraping non-static sites
We are talking about websites here that do ridiculous shit like generating zip files for each commit of a repository on request.

Someone did the math and found out that you can download tens of millions of dynamically generated zip archives from the GNOME gitlab.
This is crazy. Some single person could write a shitty shell script and it would put the whole thing down.
Whoever implemented this and/or enabled this, never considered that there could be people out there trying to put his service down.
There was never any consideration about making those platforms stable and resilient.

The first issue is:
MAKE YOUR FUCKING WEBSITE BETTER