LLM crawlers continue to DDoS SourceHut | sr.ht status
status.sr.ht
SourceHut continues to face disruptions due to aggressive LLM crawlers. We are continuously working to deploy mitigations. We have deployed a number of mitigations which are keeping the problem contained for now. However, some of our mitigations may impact end-users. In particular, we have deployed Nepenthes to certain routes which are associated with large volumes of LLM-related traffic. You may encounter certain pages which are not usable as a result, especially if you are not logged in.

SourceHut continues to face disruptions due to aggressive LLM crawlers. We are continuously working to deploy mitigations. We have deployed a number of mitigations which are keeping the problem contained for now. However, some of our mitigations may impact end-users.

  • mesamune@lemmy.worldEnglish
    14·
    1 day ago

    I had the same issue. OpenAI was just slamming my tiny little server, ignoring the robots.txt. I had to install a LLM black hole and put a very basic password protection around my git server frontend, since it kept getting slammed by the crawler.

    As much as I dont like google, I did see them come in, look at the robot.txt and no other calls for a week. Thats how it should work.