LESSON 16

Dealing with rare situations is difficult or impossible without an underlying logical model, concepts of embodiment, etc.

The problem is that extreme tail events won't be in the training data. While human intelligence deals with these naturally, AIs often don't know what to do.

Ted Chiang writes: "Self-driving cars trained on millions of miles of driving can still crash into an overturned trailer truck, because such things are not commonly found in their training data, whereas humans taking their first driving class will know to stop. More than our ability to solve algebraic equations, our ability to cope with unfamiliar situations is a fundamental part of why we consider humans intelligent."

We'll continue to need a human in the loop.

Jack of all trades, master of none. LLMs such as ChatGPT, Claude, and Gemini are capable of conversing in a variety of languages. But when you train a model to use many different languages, its performance at any individual language suffers. This is known as the curse of multilinguality.

The same basic problem arises more broadly. Current commercial LLMs are general-purpose tools trained on enormous volumes of data, scraped wholesale from the internet. That's great if you want a single model that can do everything, but for any specific task, you will probably do better with a tightly scoped model trained on curated data and specifically aligned for that task through reinforcement learning with human feedback.

Model collapse. Current large language models are trained on enormous volumes of text on the internet.

But every month, an increasing fraction of the text on the internet was itself generated by LLMs. The way things are headed, the training data for future models will include a substantial proportion of text generated by previous models.

That's a huge problem.

Imagine that you take the first page of The Hobbit and xerox it. Then copy the copy. Copy the copy of the copy. And so on. Do this a hundred times or so, and you'll end up with something that looks like television static.

Something similar happens when you iteratively train an AI on the output of previous AIs. After a few iterations, the output degrades markedly as the later models no longer have an accurate representation of the original training data. This phenomenon is known as model collapse.

As illustrated here, after a few generations of recursive training, models designed to write digits begin to produce homogeneous, blurry forms. Models trained to produce human faces quickly lose the ability to represent human diversity. And LLMs trained recursively begin to produce repetitive nonsense.

If there were definitive ways to detect AI-produced text and images when training new generative AI systems, this would not be a serious problem. These could simply be discarded from the training data. Indeed, OpenAi, Google, and others are exploring digital watermarking technologies that discreetly label text, images, or videos as AI-generated. But these technologies have not been deployed in the leading generative AI systems as yet.

Instead, the companies producing LLMs now are pulling up the ladder behind themselves, so to speak, contributing to an internet where human-created text is lost in a sea of machine-created content.

Flooding the zone. In addition to creating huge problems for training future LLMs, the explosion of LLM-created websites creates problems for current LLMs that rely on retrieval-augmented generation (RAG). Google's Gemini, for example, does this by searching the web based on your query and then providing an AI summary. The problem is, it lacks the capacity to discriminate between high and low quality sources.

Here we asked Google to convert hourly to annual wages. The answer is clearly nonsense. But why?

Google AI Overview for "how much annually is 50/hour"

Clicking on the links that Google provides, we are taken to a website called ZipRecruiter. It's not clear where they get their data or whether LLMs were involved—but it is clear the the data make little sense.

Indeed, if you replace the URL that Google references with just about any job description, however silly or implausible, ZipRecruiter will still give you a distribution of salaries, supposedly drawn from "employer job postings and third-party data sources."

(In light of these data, we recommend working as a modern-day oracle rather than a bullshit machine operator. )

It’s a remarkable quarter-century arc for an infotech company to go from revolutionizing search by using PageRank as a way of prioritizing results from influential and trusted sites, to grabbing any old random bullshit from the web and offering that up to its users in the guise of an AI Overview.