Disclaimer: I don’t have a background in computer science
I recently heard about a lightweight opensource software called Anubis.
Anubis is designed to stop AI crawlers that download a lot of data to train artificial intelligence models
Several websites have deployed Anubis:
https://gitlab.gnome.org/GNOME
10 websites have deployed. 10 websites. Out of millions of websites.
My question is extremely simple.
If this software is so damn great, why isn’t it everywhere?
Seriously. Why isn’t it used on Lemmy? On Wikipedia? On CBC?
I believe, (far too) much of the commercial world relies on Cloudflare to solve that problem.
And as for Wikipedia, any AI trainer worth their salt should know that they don’t need to crawl it, because you can actually just download the whole Wikipedia dataset.