Will talk about Linux, plants, space, retro games, and anything else I find interesting.

Also mesamunefire@piefed.social over on Piefed.

  • 5 Posts
  • 3 Comments
Joined 2 years ago
cake
Cake day: June 11th, 2023

help-circle

  • I had the same issue. OpenAI was just slamming my tiny little server, ignoring the robots.txt. I had to install a LLM black hole and put a very basic password protection around my git server frontend, since it kept getting slammed by the crawler.

    As much as I dont like google, I did see them come in, look at the robot.txt and no other calls for a week. Thats how it should work.