LESSON 9

You still can ask Google such questions— but these familiar blue links no longer appear at the top of your search query. To reach them you have to scroll past an “AI Overview” written by Google’s Gemini LLM.

Worse still, like all LLMs, Gemini often makes up utter nonsense about the most basic matters of fact. Take our query about US presidents. Google incorrectly reports that four—Washignton, Jefferson, Monroe, and Tyler—graduated from the University of Washington. Three of the four died before the UW was even founded.

This is a big problem, because people tend to trust Google and if you're not an expert, you are unlikely to question what it tells you. Or perhaps you are an expert, but don't realize how carefully you have to double-check everything Google tells you these days.

Sourcing. When LLMs pull information directly from their training data, their responses don't give you any way to check sources. They don't know their sources — and in fact they don't have sources in the way we typically think about them. They just have a vast soup of training data they use to predict what word should come next.

More recent models can also pull information from web searches and use these search queries to provide sources. At present, however, this process suffers from the same fabrications and confident false assertions that characterize LLMs more generally. One study found that commercial LLMs cite incorrect or non-existent sources from 37% (Perplexity.ai) to 94% (Grok) of the time.

This is a big problem. A key theme in our book Calling Bullshit is the need to investigate claims by tracking them back to their sources. But the whole point of replacing search engines with large language models is to eliminate sourcing. The CEO of the Perplexity AI search platform argues that AI has made blue links obsolete.

Consistency. Information retrieval systems need to be consistent. LLM's are anything but.

Imagine an enchanted encyclopedia that told you something different each time you looked up a topic. It might be fun to play with, but it wouldn't be a reliable information source.

When you query ChatGPT for facts about the world, this is essentially what you are getting. Ask the same question three times and you may get three different answers.

Recall that LLMs work by predicting likely next words in a string of text. It turns out that if you let them always pick the single most likely next response, their prose becomes stilted, they get stuck in loops, and they don't do a good job of creating human-like responses. The work-around is to add some randomness to the process. Sometimes they pick the most likely next word, a little less often they pick second-most likely etc., and so on.

This may be desirable when using a language model for a conversational partner, but it's not what you want an information retrieval system to do.

Some ChatGPT 4.o queries yield different answers each time you run them.

Fragility. LLMs can be extraordinarily fragile with respect to the precise inputs they are given. Here I ask Google for help fact-checking a claim.

Google's AI overview erroneously assures me that Ed Sheeran has a song called "The Crow".

Asking the same question but removing the quotation marks around the song title, I get the opposite answer.

In this case, the difference is replicable. We asked repeatedly, and each time Google's Gemini said yes if we put "the crow" in quotes and no otherwise.

Some people will argue that LLMs are great for information retrieval, and if you aren't getting good answers it is because you don't know how to ask the right questions. This strikes us a powerful counterexample. How would anyone know, in advance, that you have to omit quotation marks around a title?

So in general, LLMs are the wrong tool for information retrieval.

Yet they can be very helpful under the right circumstances. Sometimes you know what you're looking for, but you don't know what it's called or what keywords to use. An LLM can help you out. Describe what you're looking for in plain language, and ask what term to use in a search query.

Here's an example of how this approach helped us find a mathematical concept we vaguely remembered from college. Once we learned that the key term was "power mean", it was easy to find a detailed and trustworthy explanation online.