• Em Adespoton@lemmy.ca
    link
    fedilink
    English
    arrow-up
    1
    ·
    19 hours ago

    I don’t; I think googlebot is paying attention to it, and AI sites get a LOT of their initial visibility and site ranking from the Google index.

    Case in point: the sites I spend time on that set robots.txt to deny all bots before the LLM wave started still aren’t getting hammered with AI bot requests, while the ones that didn’t are.

    I can see it on my own websites as well; I’ve got one that’s in Google’s index, and it gets a constant low volume traffic from AI bots. The ones that aren’t listed? They just get traffic from exploit scanners.