Problem is scraper bots are way more aggressive and harder to block. If they were ignoring Reddit because they were taking content from IA but IA is willing to obey robots.txt whereas scraper bots are not, they just shifted the load of serving the bots or playing whack-a-mole with their block evading mechanisms. They aren’t going to stop the bots. It may result in being able to negotiate a license with the bigger guys, but that’s likely not going to make up for the money they spend on dealing with the bots in the long run. Of course companies like this don’t really think long term, it just looks good to investors this quarter.
Problem is scraper bots are way more aggressive and harder to block. If they were ignoring Reddit because they were taking content from IA but IA is willing to obey robots.txt whereas scraper bots are not, they just shifted the load of serving the bots or playing whack-a-mole with their block evading mechanisms. They aren’t going to stop the bots. It may result in being able to negotiate a license with the bigger guys, but that’s likely not going to make up for the money they spend on dealing with the bots in the long run. Of course companies like this don’t really think long term, it just looks good to investors this quarter.