Doesn’t Microsoft even explicitly say their Bing bot outright ignores your robots.txt? Basically only Google honors it at this point.
I have a whole table full of noncompliant bots and their IPs by now, all of which get shunted straight into the bin.
The most amusing part is that it would be absolutely trivial for these guys to make their bot traffic blend in with regular users but they’re always too stupid (greedy, impatient, pick any three) to do so. Spoof the user agent string of a legitimate desktop browser. Run some manner of Javascript environment to pass agent checks. Don’t bombard my server with dozens/hundreds/thousands of requests per minute in behavior paths that would be impossible for human users to click through. The last one is usually where they stumble. If these dweebs had any chill it would be trivial for them not to be caught. But they don’t. No, instead they want to try to fork bomb my site by clicking on every link simultaneously in parallel.
![Scrapist [PizzaCake]](https://slrpnk.net/pictrs/image/9ac42d45-d20f-4c72-ac94-1269c5dd363d.png){kind=link}
Bots ignoring robots.txt, a tale as old as time
Doesn’t Microsoft even explicitly say their Bing bot outright ignores your robots.txt? Basically only Google honors it at this point.
I have a whole table full of noncompliant bots and their IPs by now, all of which get shunted straight into the bin.
The most amusing part is that it would be absolutely trivial for these guys to make their bot traffic blend in with regular users but they’re always too stupid (greedy, impatient, pick any three) to do so. Spoof the user agent string of a legitimate desktop browser. Run some manner of Javascript environment to pass agent checks. Don’t bombard my server with dozens/hundreds/thousands of requests per minute in behavior paths that would be impossible for human users to click through. The last one is usually where they stumble. If these dweebs had any chill it would be trivial for them not to be caught. But they don’t. No, instead they want to try to fork bomb my site by clicking on every link simultaneously in parallel.
So, 403. No page for you!
Any reason you give them an immediate failure instead of wasting their time with something like nepenthes?
Laziness. Maybe some day I’ll implement that. Not today.