Mind Like a Sieve
Saying I have the memory of a goldfish is more of an insult to the goldfish than to me. I search calendar invites for the names of people I spent hours talking to the previous evening. I lookup formulas that I had used hundreds of times while in college. My memories from last year consist of 224 notes and 884 photos neatly organized on my phone.
I don’t have a photographic memory. It’s more like an impressionist painting on good days and an abstract one on bad. I empathize with the AI in conversations about hallucinations.
Here is an excerpt from the GPT-4 announcement last week:
Despite its capabilities, GPT-4 has similar limitations as earlier GPT models. Most importantly, it still is not fully reliable (it “hallucinates” facts and makes reasoning errors).
Yeah, me too.
My question is whether we should consider hallucinations a feature or a bug. This paper defines hallucinations as “generated content that is nonsensical or unfaithful to the provided source content.” Said another way, hallucinations are outputs that do not appear consistent with model training data.
Forgetful Together
OpenAI has yet to say how many parameters are behind GPT-4. However, we know GPT-3 had 175 billion. That may sound like a lot, but it’s impossible to store all the training data perfectly within the model. GPT-3 was trained on nearly 500 billion tokens, which individually contain more information than a model parameter. The model encodes patterns in the training data, not the actual data.
An AI model is an imperfect store of data. If it were perfect, we would call it a database. The 175 billion parameters can only be configured in so many ways. The model can reproduce the most common patterns it learns from the training data, but it cannot reproduce all the patterns.
Humans are no better. Each of us can configure our 86 billion neurons and 1 trillion synapses in a myriad of ways. But the possibilities are finite. We cannot retain every piece of “training data” we receive through our senses of vision, hearing, touch, smell, and taste. We encode representations of patterns and reproduce those patterns as memories.
Our inability to store the training data we receive with perfect fidelity is well-documented. Ebbinghaus’s forgetting curve, developed in the 1880s, was confirmed as recently as 2015.
We can strengthen our neural connections through repetition or tap into the high-capacity visual cortex using techniques like a Memory Palace, but our encoding is still lossy. We do not have enough synapses to “store” every pattern in the data processed by our brains.
Confidently Incorrect
Several years ago, one of my best friends stated, “the difference between confidence and arrogance is ability.” We shun arrogant AI for the same reason we shun arrogant humans. Confident people provide trustworthy information. So do arrogant people, but it’s hard to tell what is fact and what is bullshit. Arrogance and hallucination are two sides of the same coin.
The difference between humans and machines is that we realize our brains are lossy and act accordingly. We use hedging language (e.g., “I think”) and body language (e.g., shifting in our chair) to signal uncertainty. We even incorporate the reactions of others into our uncertainty calculations (e.g., pausing in response to furrowed eyebrows).
Early language models lacked the ability to communicate uncertainty in a way humans could intuit. The uncertainty was there if you looked, but most of us did not bother to check the machine’s work. We simply took the output as fact and accused the machine of hallucinating if those facts were wrong.
Here is an example. I asked the “text-davinci-003” model (based on GPT-3) whether Silicon Valley Bank stock was a good investment. This is a question the model cannot answer with certainty because the training data only goes through 2021 and language models are bad at math.
Why is Silicon Valley Bank a good investment?
I turned on “show probabilities” so I could see the predictions for each token. Unsurprisingly, the first prediction was “Sil” with a 99.97% probability.
The model predicted the name of the bank with near certainty. That was the easy part. I included the name of the bank in my prompt. The next prediction is more interesting.
In parentheses, the model predicted “SIVB,” which is the stock ticker for Silicon Valley Bank. However, the top prediction was actually “SVB,” an acronym used when referring to the bank. When I set the “temperature” to zero (fully deterministic), I receive the “SVB” acronym rather than the stock ticker, “SIVB.”
“SIVB” is a micro-hallucination. The model failed to indicate those letters were a stock ticker rather than an acronym. The training data contained references to both SVB and SIVB, but the model did not encode the nuance between them.
The Difference is Ability
I ran the same experiment with GPT-4. Here is the first part of the response.
As an Al language model, I cannot provide real-time stock advice or guarantee that any stock will be a good investment. However, I can provide you with some factors to consider when evaluating Silicon Valley Bank (SVB) stock or any other stock for potential investment.
The model knows “SVB” is an acronym rather than a ticker. It also provides hedging language to signal uncertainty. The model goes on to produce a complete response.
Silicon Valley Bank is a subsidiary of SVB Financial Group (NASDAQ: SIVB) and is a leading provider of financial services to startups, technology, and life science companies. Here are some potential reasons why some investors might consider it a good investment:
1. Niche market focus: Silicon Valley Bank specializes in catering to the unique financial needs of high-growth tech and life sciences companies, which could position them well for growth as these industries continue to expand.
2. Strong network: SVB has a well-established network in the tech and startup industries, giving it access to potential high-growth clients and creating opportunities for partnerships and collaborations.
3. Diversified services: In addition to traditional banking services, SVB offers venture capital, private equity, asset management, and other financial services, providing multiple avenues for revenue generation.
4. Innovation: SVB is known for its innovative approach to banking, utilizing technology and a deep understanding of its clients' industries to stay ahead of the competition.
5. Solid financials: Historically, SVB has demonstrated strong financial performance with robust revenue growth and a healthy balance sheet. Analyzing the company's financial statements and ratios can provide further insight into its financial health.
6. Positive outlook: Some investors may have a positive outlook on the technology and life sciences sectors, which could lead them to believe that SVB is well-positioned to benefit from the growth of these industries.
It is essential to remember that investing in individual stocks comes with risks, and past performance is not indicative of future results. Conduct thorough research, consult with a financial advisor, and consider your own investment goals and risk tolerance before making any investment decisions.
Note the correct citation for “SIVB” along with additional hedging language. Has the model become sentient and recognized it is capable of forgetfulness? No, it has simply encoded more patterns from the training data and learned when to use hedging language.
The large language models from OpenAI are becoming less arrogant and more confident. The difference is ability.
Feature or Bug?
What makes large language models incredible is their ability to answer new prompts using patterns learned from training data. Few of us are amazed by databases that store gigabytes of information with perfect fidelity, but we are fascinated by humans who can recite the first 100 digits of pi.
Hallucinations are a byproduct of intelligence. There is no hallucination problem to solve. As AI models grow, more patterns will be encoded. At the limit, neural networks become overly-complex databases. That defeats the purpose. We will always walk a fine line between an acceptable level of hallucination and overfitting the training data.
That said, we should absolutely solve the communication problem. GPT-4 has made progress with respect to hedging language. Perhaps, someday, the model will even incorporate feedback from our responses to update its uncertainty estimates. We accept uncertainty in humans when it is on full display. We need to hold machines to the same standard.
Hallucinations are a feature, not a bug.