• buddascrayon@lemmy.world
    link
    fedilink
    English
    arrow-up
    3
    ·
    edit-2
    19 小时前

    Just a quick correction. Robots.txt doesn’t block anything, it simply politely asks the search engine to not look here or use what is contained within. A lot of the search engines ignore this and catalogue the contents of the website anyway. And pretty much all of the AI database downloaders ignore it with impunity.