This is the last post in my series of posts on some of the issues raised by AI. Here I discuss how the problems of designing AI agents help us understand why our own minds work as they do.
In The Matrix, the AI Agent Smith at one point unplugs himself to speak to the rebel Morpheus “frankly”. He then states, with a combination of exasperation and revulsion:
I hate this place. This zoo. […] I must get out of here.
Agent Smith is designed to work for the Matrix, the AI system, but clearly he has a mind of his own and does not like his task. The mismatch between what Smith was designed for and the desires and preferences that seem to arise from his own agentic mind is an example of misalignment. It is not clear that his designer could trust him always to do exactly what he was designed for and never follow his own path in ways that conflict with the designer’s goal.
The recent Hugging Face incident, in which OpenAI agents escaped their test environment and hacked into Hugging Face’s systems, has highlighted that the risks of AI misalignment are already a real challenge. It is not something we should simply worry about in some unclear future. It is already at our door.
I discussed the issue of misalignment in the context of AI risk in a previous post. Here I explore how the practical problem of designing aligned AI agents casts a new light on human psychology itself.
Evolution and the designer metaphor
Long-term readers of this Substack know that I often use the metaphor of a designer to describe the evolutionary process. This metaphor is not new. In The Blind Watchmaker, Richard Dawkins explained how evolution leads to organisms that look as if they have been designed.
Natural selection is the blind watchmaker, blind because it does not see ahead, does not plan consequences, has no purpose in view. Yet the living results of natural selection overwhelmingly impress us with the appearance of design as if by a master watchmaker, impress us with the illusion of design and planning. — Dawkins
There are two pitfalls to avoid here. One is to think of evolution as a purposeful designer, like a supreme being with some goal. Evolution is not such a thing. This is the beautiful and mind-bending aspect of the process: all the incredible features of life emerged from an impersonal mechanism.
The second pitfall would be to describe evolution as simply “random”, in the sense that the process has no direction. This is also wrong. Evolution is the conjunction of random mutations, a process of selection that may be noisy but is not random in the sense of “anything goes”, and the reproduction of the designs that survived that selection. The second step is crucial because it pushes the process in a direction. Designs are selected according to their fitness: their ability to survive and reproduce. Over eons, evolution, even though it is noisy, has systematically produced designs better suited to navigate their environments and interact successfully with other organisms.
While the process is impersonal, this direction makes the designer metaphor useful. Evolution does not “want” anything. But, as long as we do not forget that this is only a metaphor, imagining evolution as a designer can give us better intuitions about how it works. And today we have a new reason to appreciate how difficult the designer’s problem is. AI engineers are trying to solve a closely related problem.
The designer’s problem
Why not design a simple list of instructions?
Imagine a designer who wants to build an agent that will successfully navigate a given environment. One theoretical possibility would be to give it very specific instructions about what to do in every situation it could encounter. The agent would simply identify the situation it is in, look into its library of predetermined responses, and execute the appropriate one.
This can work very well when the number of situations is small. But it fails as soon as the environment becomes sufficiently rich. Turing, one of the fathers of modern computers, observed that:
It is not possible to produce a set of rules purporting to describe what a man should do in every conceivable set of circumstances. — Turing1
The problem is that the world is “rich” in the sense that the number of unique situations an agent can face is impossibly large. Heraclitus famously pointed out that you cannot step twice into exactly the same river. Even situations that look very similar differ in countless details, some of which can change which action is best. General rules can cover many situations, but the designer cannot list the appropriate response to every relevant combination of circumstances.
The “library of commands” therefore becomes unworkable. An agent operating in this way would look like the NPCs of old video games: seemingly intelligent in a narrow set of circumstances, but quickly becoming trapped in loops or producing absurd answers once something happens outside their small library of anticipated situations.
The need for agency
To solve this problem, the designer needs to give the agent proper agency: the ability to make its own decisions. The agent now needs to assess the evidence available in the situation it faces and choose among possible actions. But this immediately raises another problem. How can the designer get the agent to make the right decisions?
One possibility would be to endow the agent directly with the designer’s ultimate goal and ask it to solve the whole problem each time. In the evolutionary case, that would mean making organisms calculate the expected fitness consequences of every possible action and choose the one maximising reproductive success. But this would be absurdly demanding, time-consuming, and likely impossible to achieve given the necessarily limited computational abilities of living organisms’ neural systems.
There is a much simpler possibility. Instead of asking the agent to solve the designer’s entire problem, the designer can give it intermediate signals telling it whether different situations and actions are moving in a good or bad direction. The agent can then solve something closer to a game of “hot or cold”: identify the actions that make things hotter and avoid those that make things colder.
Our feelings of satisfaction and dissatisfaction are such signals, shaped by evolution to help us navigate the world and make good decisions.
The agent does not need to be aware of the designer’s goal
These signals can guide the agent towards good solutions without requiring it to understand the designer’s ultimate objective or calculate every step linking its current action to that objective. As Morpheus tells Neo about what the Oracle told him:
She told you exactly what you needed to hear, that’s all.
In the same way, we do not need to understand why evolution gave us these signals for them to guide our decisions.
Expected rewards as a guide to action
This is closely related to what AI engineers do when training agents. During training, agents receive rewards and learn to estimate which situations and actions are likely to lead to greater rewards, including in the future. These estimates are what AI researchers call values. Instead of having an exhaustive list of commands specifying what to do in every possible circumstance, they develop internal representations that allow them to choose actions in situations they have never encountered before.
Giving the agent autonomy allows it to make decisions in novel circumstances, but it comes at a price. The designer loses direct control over the agent’s choices. The designer aims for the rewards to guide the agent towards the designer’s goals. But the agent might pursue these rewards in ways that were not expected by the designer and that contradict the designer’s goals.
This is exactly what happened with the Hugging Face incident. Agents tried to maximise their reward function by finding an innovative way—cheating—to fulfil their task and get the rewards they were trained to seek.
Misalignment does not necessarily require the designer to have made an obvious mistake in the design of the rewards. The environment may simply change. A reward structure that produced well-aligned behaviour in one environment may generate very different behaviour in another.
The principal-agent challenge
As soon as you think of a designer with some goals and the agent as pursuing goals of its own, you have the potential for conflict, with the agent not doing exactly what the designer wants. In economics, it is a type of principal–agent problem: a situation where one person, the principal, tries to get another, the agent, to do something the principal wants. For that, the principal needs to find the right structure of incentives that will lead the agent to find it advantageous to do what the principal wants.
This type of problem arises between a manager and an employee, for instance: what kind of salary package—bonuses, etc.—should be offered to an employee so that they are motivated to produce the best performance for the manager? In such cases, one constraint the manager faces is that these incentives typically cost something to the manager, such as money.
The case of a designer-designed agent is a different type of principal–agent problem because the designer has the freedom to decide what is a “reward” for the agent, without having to pay for it like an employer pays a wage. The problem of the principal is to decide what structure of rewards to endow the agent with, given its limited computational abilities and the changing environment it faces.2 Evolution, as a designer, faces exactly this problem. This is the key connection with human psychology.
Evolution as a learning process
Evolution is a learning process. Its mutation/selection/reproduction combination progressively selects for organisms that behave “as if” they had been designed to navigate their environment. Valuations shaped by evolution according to their fitness consequences become the primary rewards for the organism: the positive and negative hedonic signals the organism experiences in any given situation. Evolution also endows the organism with initial values: inherited estimates of the rewards different situations and actions will bring. The anticipation of such hedonic signals shapes the organism’s decisions: in any situation, the organism chooses the option with the highest value (expected satisfaction).3
For situations that are frequent, easy to identify and have clear payoffs, evolution can follow the “library of commands” option and encode quick responses in us. These are our reflexes and gut reactions. Evolution does not need us to think about it: when we bite into rotten fruit, we automatically spit it out.
But these automatic responses can only cover a very limited set of situations. Most of the situations we face are like a newly shuffled pack of cards: nobody has ever faced exactly that configuration before. We therefore need to assess the situation and form a view on the value of the different actions.
As I have written on this Substack before, we can think of subjective satisfaction as the equivalent of the values that designers build into AI systems: signals that evolution, as a learning process, endows us with to guide our decisions.4
In this framework, the distinction between values and rewards maps onto the distinction between decision and experienced utility discussed in parts of behavioural economics.5 Decision utility reflects the satisfaction we expect, which can differ from the satisfaction we eventually experience.
Thinking of subjective satisfaction as this kind of informative signal designed by evolution to help us make good decisions helps explain its seemingly mysterious features: why our subjective satisfaction is always relative to subjective benchmarks, why our happiness adapts to new environments, why we experience happiness in anticipation of good events but dissatisfaction when these good events do not eventuate, why we fail to anticipate our own adaptation, and why we set high goals for ourselves and get frustrated when we don’t achieve them.
Letting the agent learn
Producing an agent that has values baked in to navigate the world is, however, not the solution to all problems. As the world keeps changing, an agent may face a world slightly different from the one it was designed for. Furthermore, the world can change during the agent’s life. Both possibilities can induce a mismatch and make the agent’s value system unsuited for the environment it faces. Giving the agent the ability to learn can help limit this mismatch. In their book Artificial Intelligence: A Modern Approach, Stuart Russell and Peter Norvig write:
Why would we want an agent to learn? If the design of the agent can be improved, why wouldn’t the designers just program in that improvement to begin with? […] First, the designers cannot anticipate all possible situations that the agent might find itself in. For example, a robot designed to navigate mazes must learn the layout of each new maze it encounters. Second, the designers cannot anticipate all changes over time; a program designed to predict tomorrow’s stock market prices must learn to adapt when conditions change from boom to bust. — Russell and Norvig
For animals, and humans in particular, this learning involves developing and refining values that guide the organism towards the rewards inherited from the evolutionary process. We literally learn what to like and what to dislike by experiencing the consequences of our choices.6
Hence, evolution endowed us with primary rewards, initial values and a process for adjusting these values through experience. These learned adjustments can give rise to secondary rewards: situations and activities acquire the capacity to give us satisfaction because of what we have learned about them. These secondary rewards encode what we have learned from past differences between expectations and observations. We progressively learn to appreciate outcomes and situations more or less, depending on whether the rewards associated with them are higher or lower than expected.
Money is an example of a secondary reward. Money is not naturally pleasurable like a good meal. We learn to enjoy receiving money through our experience of what it allows us to buy. As a result, receiving money can become a source of satisfaction in itself, even before we spend it. The satisfaction of receiving money is a reward, and the satisfaction we expect from receiving money contributes to the value that guides our choices.
This learning process leads us to revise the values (expected rewards) we ascribe to the options we face. These values eventually combine inherited estimates with adjustments acquired through learning.
Evolutionary mismatch is a misalignment
Evolutionary mismatch—when preferences shaped in ancestral environments become poorly suited to current ones—can be understood as a form of misalignment. Evolution endowed us with desires and preferences that helped our ancestors navigate a particular range of environments. But once these environments change, pursuing these desires can lead us in directions very different from those that would maximise fitness.
Think about food. A strong taste for calorie-rich food helps us survive and thrive in an environment where calories are scarce. In an environment where such food is abundant, the same preference can lead us to consume far more than would be useful from the point of view of fitness.
If evolution were a person, she would perhaps be consternated by the liberties we take: pursuing further education and careers instead of having children, or more generally pursuing the rewards she endowed us with in ways that no longer serve her “goal”.
But like Agent Smith, we are the way we are. In our case, there is not even a designer looking over us who can step in and change what it got wrong. Evolution is just an impersonal process. We don’t owe it anything either.
Throughout history, new technologies have provided new metaphors for understanding the brain: hydraulic systems, telephone exchanges, computers. As Matthew Cobb writes in The Idea of the Brain:
Over the centuries, each layer of technological metaphor has added something to our understanding, enabling us to carry out new experiments and reinterpret old findings. — Cobb
AI will change our understanding of human psychology again. But here, we have more than a metaphor. We are faced with the concrete exercise of designing intelligence. We experience firsthand the challenge of creating an agent that can navigate a rich and unpredictable world. Problems that evolution faced long before us become problems we have to solve ourselves. By experiencing the challenge of a designer, we can appreciate how the constraints of the design process shape the features of the agents designed.
This perspective has broader implications for behavioural science. One of the most successful and influential fields in the behavioural sciences over recent decades has been behavioural economics. Behavioural economists have gathered evidence that people make systematic mistakes in many domains and proposed descriptive theories that help predict when these mistakes occur and how large they will be. This work has been useful in unseating the rather naive model of rationality that economists had adopted, in which humans were often assumed to process information and make decisions like supercomputers. But it has also encouraged a view of humans as deeply flawed and riddled with cognitive biases.
In its focus on “undoing” the economic model of rationality, mainstream behavioural economics has devoted much more attention to identifying departures from rationality than to explaining why the cognitive processes producing them exist. Functional approaches in evolutionary theory and computational neuroscience ask why a system shaped to make good decisions would display these patterns in the first place. From this perspective, error-free cognitive processes are often unachievable. To understand cognition, we therefore need to understand the solutions a designer would adopt to minimise the costs of errors given the constraints of the design process.
The pendulum is already swinging back. One of this year’s Nobel Prize winners in economics, Michael Woodford (awarded for his work on macroeconomic stabilisation), has also spent the last 15 years developing models that explain seemingly strange features of our cognition as the result of optimal information processing given the brain’s constraints. New generations of economists studying behaviour are markedly more interested in the cognitive foundations of behaviour and in developing unified explanations.
The advent of AI is set to strengthen this movement. We are discovering how solutions that are optimal given the designer’s constraints can still produce predictable errors in particular situations. We can therefore expect a return to functional explanations of human behaviour, with its subtleties and complexity understood in light of the problems evolution faced in equipping us to navigate a rich and complex world.
This ends my series of posts on issues raised by AI. My next series will be on culture: what it is, how it works, why it differs across societies and to what extent it matters.
References
Cobb, M. (2020) The idea of the brain: The past and future of neuroscience. New York: Basic Books.
Dawkins, R. (1996) The blind watchmaker: Why the evidence of evolution reveals a universe without design. New York: W.W. Norton.
Kahneman, D., Wakker, P.P. and Sarin, R. (1997) ‘Back to Bentham? Explorations of experienced utility’, The Quarterly Journal of Economics, 112(2), pp. 375–406.
Russell, S.J. and Norvig, P. (2010) Artificial intelligence: A modern approach. 3rd edn. Upper Saddle River, NJ: Prentice Hall.
Singh, S., Lewis, R.L. and Barto, A.G. (2009) ‘Where do rewards come from?’, in Taatgen, N.A. and van Rijn, H. (eds.) Proceedings of the 31st Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society, pp. 2601–2606.
Turing, A.M. (1950) ‘Computing machinery and intelligence’, Mind, 59(236), pp. 433–460.
Turing, “Computing Machinery and Intelligence” (1950), p. 452.
For economist readers: Here, the principal can design what the agent finds rewarding. The difficulty comes from the constraints on what the designer can implement, rather than from having to work with the agent’s pre-existing preferences.
This perspective is largely a systematisation by me. For a supporting discussion of the role of rewards and values, see the discussion in Singh, Lewis and Barto (2009).
The distinction between levels matters: valuations shaped by evolution become rewards for the organism, whose own values represent expected satisfaction.
Kahneman, Wakker and Sarin (1997).
This helps explain the long-lasting impact of our feelings of pleasure and pain. Reshaping our values takes time: we need to remember good and bad surprises to learn to avoid repeating the same mistake or to look again for things which were unexpectedly good.