It’s hard to put this into words succinctly. But when I was a kid before the Internet, the library was the main source of human knowledge, and it was very organized through the Dewey Decimal System which imposed a kind of tree-like hierarchy across a wide spectrum of topics. And while I imagined that one day this might all wind up served by computers, I thought this organization would survive the transition. But it didn’t.
I suppose in the early days of the Internet, they tried? Yahoo was kind of an Internet directory in the beginning, and the Whole Internet catalog was another such effort. But these eventually fell apart and we had to resort to search engines to find anything. Frankly, it’s a bit like the early days of personal computing when we used flat file systems instead of hierarchical ones with proper directory trees. When you were limited to what could fit on a floppy, this wasn’t such a big burden. But the Internet as it stands might as well be a giant flat file system.
Today, even the search engines are failing us, and we are turning to AI. In the pre-Internet era, we had sort of an equivalent to this also. They were called librarians. But a librarian’s job was made easier by the fact that all the books were carefully organized by topic. For AI, it’s as though the library were just a massive pile of books tossed around haphazardly and the librarian had to make sense of it enough to pull whatever you’re looking for out of the chaos. Is it at all surprising, then, that it takes a huge amount of energy and computing resources to make any of this work?
Apologies if I’m reading too much into what you’ve said here, but it reads to me as if you’re implying search engines couldn’t keep up with the volume of the internet and AI is the logical next progression.
Search engines actually worked pretty amazingly at one point, but then were kneecapped because Google et al were more interested in advertising money than useful search results. Then when they needed an excuse to justify ludicrous spending on AI, they shoehorned AI into web search where it is very much the wrong tool for the job if you want accurate, high relevance results.
It’s more like the librarian has taken it upon themself (for some reason) to write a book that is the mathematical average of every other book in the library. Then when any patron comes looking for a specific book, the librarian just hands them this slag instead hoping they won’t notice the difference.
Now that said, I 100% agree that running a good “old-style” search engine does still take a huge amount of computing resources, but I’d be surprised if the requirements were anywhere near what is being thrown at training LLMs.
Honestly, best description of LLMs for search I’ve seen so far.
Ah, so you’re saying it’s more of an enshitification thing that killed the search engines? I can believe that to an extent.
But as the amount of content on the Internet has increased, it’s also become more difficult to narrow the queries to get what you’re really looking for. The main thing for me with AI is it allows you to refine the search without discarding the context of what you have already tried.
If the content were more organized to begin with, though, I suspect you could drill a bit deeper into the knowledge tree and then apply your conventional (non-AI) search over a more relevant branch?
It’s not just increase in content. Search engines don’t even do searches based on keywords or logic operators anymore. They run some NLP on the query input and do who knows what to give you results. If we could still make searcher that contain specifically certain terms or exclude others, it would be still possible to navigate into the mess
Google has a financial incentive for you to spend more time on their site. If you are gone in one click, you see less ads. Same problem with dating platforms. The user’s interest is in conflict with the platform’s.
There are leaks/interviews how their search and ads teams were at a conflict internally, ofc money won.
Does the ad revenue account for people who hardly realize the ads are even there?
Let’s not forget Search Engine Optimization (SEO) and content farms which were actively corrupting (adding entropy to) search engine results even before LLMs, yay capitalism. They’ve dialed it to twelve with LLMs.
Heh, I had actually forgotten about that. It seems almost quaint compared to where we’re at now.
Search algorithms are meant to work regardless of the amount of content or sorting. Page rank used to be a game changer back then to sort websites by relevance.
AI has no search algorithms. It’s about 1000% less efficient, because its algorithms are built for token prediction, not sifting through websites.
So yeah, my money is on enshitification.
AI is very much the right tool for sorting through the high volume of garbage search results looking for nuggets of relevance. Google built it into search because if they didn’t, they would be cut out of search other than as an MCP provider. I’m sure they’d rather not spend compute running an AI summary of every single query, but they would be out of business the moment someone else provided that service.
Now is it perfect? Of course not. Verify from sources, but you get to those relevant sources faster and easier with AI doing the sifting because Google search has grown so fucking worthless.
AI isn’t better than Google was, but it’s better than Google is.
Google is not at fault for search becoming so terrible. Everyone who engaged in SEO is. The internet would have sloppified much sooner if Google and other search engines had not relentlessly pushed back against SEO and continuously modified their algorithms. Lots of people definitely underestimate the extreme negative impact of SEO, and the lengths that search engine operators have to go to to have their service be at least somewhat useful. As soon as you have a search algorithm, people will attempt to manipulate it.