Context Beats Computation (For Now)
My feeds are inundated with posts about ChatGPT. People seem surprised by the latest advancements in large language models and image generators like Midjourney. How did we progress in a few short years from chatbots that could barely understand simple commands to know-it-all assistants capable of explaining string theory, drafting cover letters, and telling better dad jokes than me?
I believe the answer lies in the power of context. As humans, we like to think of the world as neatly ordered. Writing an email is distinct from painting a picture or baking bread. In reality, the world is messy. The same neurons we use for painting are also used for writing an email. You think of an “S” as a building block of language. In reality, it’s also a squiggly line processed by your visual cortex.
For years, neuroscientists operated on a theory of localization. Each part of the brain was responsible for specific tasks, and studying the parts would help explain the whole. That theory has been largely debunked. The connections between parts of the brain may be less numerous but play a central role in how we process information.
What I’m describing is context. Our brains seem “magic” because we apply patterns learned in one domain to other domains. Context is the difference between ChatGPT and the annoying chatbot employed by your phone company. Do you need to read the entirety of Wikipedia to help with a phone bill? No, but context helps when conversations go off-script.
What machines lack in context, they make up for in computation. ChatGPT does not “know” what a dog is in the same way as you. It cannot generate an image of a dog, reproduce a dog’s sound, or recognize the smell of a dog coming in from the rain. However, ChatGPT has read more about dogs than you. If you ask ChatGPT how many legs a dog has, it will answer correctly. The AI lacks a contextual understanding but has seen enough data to accurately predict what should come after, “How many legs does a dog have?”
The battle between context and computation is central to the future of automation. Self-driving cars are one example. Self-driving algorithms are trained to predict steering, brake, and throttle inputs using information from the environment. The algorithms have access to more data and can make faster decisions than any human. Self-driving cars are already superior in terms of computation.
However, the task of driving also benefits from context. If you are driving down a road at dusk and spot a deer in a field, you will likely slow down, watch the side of the road, and move to the inner lane. You use your contextual understanding of “deer” to adjust steering, throttle, and braking inputs. The machine sees the deer as well but does not have the benefit of context. The machine does not know that deer frequently run in front of cars and that a single deer could mean other deer are waiting in the forest ahead.
Machines attempt to solve the self-driving problem using computation. The algorithm can recognize a deer jumping in front of the car and apply the brake faster than you. However, your understanding of deer allows you to take preventative measures beyond the reach of the algorithms. The power of computation is no match for the power of context.
What does this mean for the future of AI? In my view, context is simply a matter of building larger models. Language models and image generators are separate today but need not be in the future. As models become larger, contextual understanding should increase. Get ready for more AI surprises. Computation is a beginning, not an end.