minus-squarepewter@lemmy.worldtoTechnology@lemmy.world•AI crawlers cause Wikimedia(The umbrella organization of Wikipedia and a dozen or so other crowdsourced knowledge projects) Commons bandwidth demands to surge 50%.linkfedilinkEnglisharrow-up1·2 days agoThere’s a chance this isn’t being done by someone who only wants Wikipedia’s data. As the amount of websites you scrape increases, your desire to use the easy tools loses out to creating the most general tool that can look at most webpages. linkfedilink
minus-squarepewter@lemmy.worldtoMemes@lemmy.ml•Once you realiselinkfedilinkarrow-up2·10 days agoIf they’re withdrawing to their bunkers, the surface probably isn’t that great. linkfedilink
There’s a chance this isn’t being done by someone who only wants Wikipedia’s data. As the amount of websites you scrape increases, your desire to use the easy tools loses out to creating the most general tool that can look at most webpages.