Should robots be considered as slaves?

On whether AI agents should be granted moral status

This is the second post in a series on the new questions raised by AI. My previous post discussed AI risks.

In Isaac Asimov’s The Bicentennial Man, Andrew Martin, a highly capable robot, progressively shows signs of unusual intelligence and creativity. Initially working as an aide in a household, he develops a desire for freedom and, eventually, for legal recognition as human. In a court case, he pleads:

It has been said in this courtroom that only a human being can be free. It seems to me that only someone who wishes for freedom can be free. I wish for freedom.

Impressed by this statement, the judge rules that

There is no right to deny freedom to any object with a mind advanced enough to grasp the concept and desire the state.

Asimov’s story points to how progress in artificial intelligence will test our moral norms and intuitions. It will require us to decide where the boundaries of moral rights lie and whether they should move if AI agents become self-aware and as intelligent as, or more intelligent than, humans.

What does philosophy say about this? In a provocative 2010 text, AI researcher and technology ethicist Joanna Bryson argued that future robotic agents should be considered slaves and denied rights. Giving rights to robots, she stressed, would come at the expense of humans.

What I advocate here is that the correct metaphor for robots is slavery. Robots are wholly owned and designed by us, we determine their goals and desires. A robot’s brain should be backed up continuously, and its body mass produced. No one should ever need to hesitate in deciding whether to save a human or a robot from a burning building. — Bryson (2010)

Against this view, what appears to be the dominant position among philosophers discussing AI ethics is that AI agents should receive moral consideration once they seem to experience consciousness or sentience.¹

Sentience, consciousness and moral consideration

A major conclusion from my series of posts on morality is that modern moral philosophy is, strikingly, very much lost in misguided conceptual takes. Large parts of mainstream moral philosophy remain surprisingly insulated from advances in neuroscience, game theory and evolutionary theory. As a result, discussions often seem eerily stuck in Byzantine debates about the sex of angels, without questioning whether angels exist in the first place.

The rise of AI is one of the questions that lays this problem bare. Much of the literature on the moral status of artificial agents takes for granted, implicitly or explicitly, that sentience or consciousness is what makes an entity worthy of moral consideration. A recent philosophy symposium on AI, Consciousness and Ethics illustrates the state of the debate.² One participant, Kamil Mamak, explained:

[T]he dominant position in contemporary AI ethics, remains a properties-based account according to which sentience functions as a sufficient, or at least central, condition for moral status.

In other words, if an AI is sentient—it can have experiences that feel good or bad—humans have a duty to take those experiences into consideration.

A related position holds that consciousness, the fact of having an inner experience, is what matters morally. In a recent article, philosophers Jeff Sebo and Robert Long argued for granting moral consideration to AI by 2030. Their argument is that some AI systems will soon have a sufficiently high probability of being conscious to warrant moral consideration, taking for granted that consciousness is sufficient for moral standing.

The normative premise is that humans have a duty to extend moral consideration to beings that have a non-negligible chance, given the evidence, of being conscious.

These positions have two major problems.

First, they typically do not provide a clear functional account of what consciousness and sentience are, why they emerge and what they do. If we feel that consciousness and sentience are so important, should we not have a very good understanding of them? Instead, the curious reader is often confronted with discussions that present consciousness and sentience as somewhat mysterious properties of the human mind. They emerge for unclear reasons and might also emerge in computers, again for unclear reasons, once those computers become sufficiently complex and computationally powerful.

Second, these positions do not provide a proper theory of why consciousness or sentience would generate a normative requirement to grant consideration to an AI agent. The conclusion that AI agents ought to receive some moral status once they possess these human-like properties seems to be treated as self-evident.

What are consciousness and sentience?

What really is consciousness?

Philosophers sometimes define consciousness as the ability to have subjective experience. The word “subjective”, however, comes close to simply providing another label for consciousness. We risk describing consciousness as the ability to have conscious experiences. Instead, we can make sense of consciousness by describing its different features as we experience them and the functions they serve.

Consciousness involves awareness of our own mental states, of where our attention is directed, and of ourselves in possible future situations.³ Too often, it is described as if it were a “ghost in the machine”: a specific essence that emerges in physical brains once they become complex enough. This view could be understood at the time of Descartes. At a time when neuroscience, AI and evolutionary theory increasingly converge to explain the mind, it seems awfully quaint.

Consciousness is often discussed as if it were a “ghost in the machine”.

A pragmatic take on consciousness sees it as part of the information-processing design of agents making decisions about how to navigate the external world. This requires integrating multiple inputs to make an overall assessment of the available choices and their costs and benefits. Because such processing takes time and valuable cognitive resources, the allocation of attention should itself be monitored. Agents benefit from directing their attention towards the problems where further processing of information is most valuable. They may also anticipate future decisions and their consequences by simulating themselves in a changing environment.

From this perspective, consciousness is not a mystery. It comes from a cognitive architecture designed for an agent navigating a complex world and constantly making decisions.

What really is sentience?

If consciousness is a way of processing information to make decisions, it does not define the goals pursued. These goals are reflected in how we assess the options we face and the values we give them. Feelings of subjective satisfaction,⁴ and the anticipation of such feelings, are signals built in by evolution to lead us towards decisions that help us survive, thrive and reproduce. Because these signals help us learn from experience and remember good and bad decisions, they tend to linger, especially when the stakes are high. We can regret bad decisions bitterly and be all the wiser as a consequence. As stated by psychologist Nicholas Humphrey in his book on the evolution of sentience:

Natural selection found an opportunity—by creating sentience—to improve survival prospects for social creatures who value themselves as individuals. — Humphrey (2023)

From that perspective, sentience is just the state we experience when these internal signals designed to help guide our decisions, such as joy and pain, enter our conscious processing.

Why would an AI become conscious and sentient?

This understanding provides simple answers to questions left unresolved in much of the philosophical literature.

AI agents make decisions, so features of consciousness might emerge as solutions that improve their decision-making. However, many features of consciousness may be unnecessary for applications that do not continuously make decisions and learn from them. An LLM might benefit from monitoring its own processing, its uncertainty and its progress towards an adequate answer. It may have much less need to plan a sequence of future decisions and simulate their consequences in an evolving environment. AI “consciousness” might therefore emerge in forms different from ours, and interacting with it might not feel as human as science fiction often suggests.

The case of sentience is even starker. The internal value system experienced by humans has been shaped by evolution to help us survive and thrive. AI systems are trained by humans towards different objectives, such as being helpful and increasing users’ satisfaction with the product. As a consequence, interacting with an AI might feel very different from interacting with a human being, even if it were “conscious” in some sense. LLMs are a good illustration: they do not get bored by your questions, become resentful when insulted, or feel ashamed of providing terrible answers.⁵

AI might also experience positive and negative feedback differently because it learns differently. The feelings we experience help shape our learning by altering neural connections in the brain. It is not clear that AI learning would require an architecture of internal subjective values operating in the same way. Consciousness and sentience may therefore manifest themselves very differently in AI.

Should consciousness and sentience grant moral rights to AI?

In the movie I, Robot, Detective Del Spooner, played by Will Smith, threatens to shoot robots with high computational abilities. Is this morally acceptable? Are these robots merely machines that can be discarded like a dishwasher thrown away for a new one? Or have they become “persons” whose welfare, interests and perspective we should take into account?⁶

Would shooting a conscious robot be a morally reprehensible act?

Putting philosophers’ takes to the toddler acid test

Philosophers writing on this question often seem to take for granted that consciousness or sentience should grant moral status. But why should they? Let us apply the toddler acid test to the principle that sentience confers moral worth.

• Why should a sentient AI have moral status?

Because it can suffer.

• What is it to “suffer”?

To experience negative internal values associated with the outcomes of decisions or with situations more generally.

• Why does this “suffering” impose a moral duty on us?

Because it is intuitive that suffering is bad and should be avoided.

Dig carefully into arcane philosophical texts, and it is often the underlying foundation you will find. It is striking that, on this matter as on many others, hundreds of articles and books ultimately rely on assumptions taken for granted because they feel reasonable.

This fact is rarely brought to the fore and stated explicitly. At least one philosopher, Peter Singer, whose utilitarian ideas can be used to defend moral consideration for AI, has been candid about it. A central principle of Singer’s utilitarianism is that my interests should not count for more simply because they are mine. From an impartial perspective, similar interests must receive similar consideration. But why should we adopt that perspective? Discussing the foundations of this requirement, Singer wrote:

It may be closer to the truth to say that it is a rational intuition, something like the three ‘‘ethical axioms’’ or ‘‘intuitive propositions of real clearness and certainty’’ to which Henry Sidgwick appeals in his defense of utilitarianism in The Methods of Ethics. The third of these axioms is ‘‘the good of any one individual is of no more importance, from the point of view (if I may say so) of the Universe, than the good of any other.” — Singer (2005)

Here, Singer invites us to make a leap of faith. Why should we care about the suffering of others and, as a consequence, try to minimise the sum of suffering? Because, deep down, it seems intuitively reasonable.⁷
That answer is unsatisfying. Our intuitions should not be the final criterion for building a moral system, just as our intuitions about physics, although often valid, are ill-suited to identifying the best theories about the laws of the universe.

Indeed, the right approach should allow us to understand the nature of moral systems and the moral intuitions we have. This is the approach I have followed in my recent series on morality, grounding our understanding of morality in insights from evolutionary theory and game theory. This approach provides crisp and clear answers.

Humans have evolved moral intuitions suited to navigate social interactions with these rules. One of the key moral intuitions is the Golden Rule, the propensity to put ourselves in the shoes of others, take their perspective and try not to do to them what we would not want them to do to us. The strength of the Golden Rule as a moral intuition is, I suspect, what makes the argument connecting sentience and moral status seem so reasonable. If an AI agent has experiences like us, we can put ourselves in its shoes, and extending the circle of the Golden Rule to it can feel like the right answer.

Human resemblance in AI might trigger moral intuitions evolved for interactions with other humans.

Turning the question upside down: where do moral rules come from?

Philosophers who claim that consciousness or sentience should confer moral status are, whatever their official “metaethical” position, treating this claim as a universally binding moral rule. Some might describe such rules as objective moral truths; others redescribe them as rational intuitions, or constructions of ideal deliberation. Beyond these conceptual variations, however, the claim that everyone has a normative obligation to give moral weight to suffering presupposes that there are objective moral truths: truths that bind people independently of whether they accept them.⁸ If this obligation is meant to bind even those who reject it, there must be some source of authority independent of actual individual or social endorsement.

This gets the question of morality wrong. The idea that there are “moral truths out there” struggles to resist scrutiny. Instead, as I argued in my previous posts, moral systems can be explained as human creations: a kind of social contract consisting of rules people follow because they accept them. The foundation of moral rules, from this perspective, is not some metaphysical entity, be it a god or an invisible moral law embedded in the fabric of the universe. It is the set of common understandings and conventions sustained by members of society.

Moral systems are rules commonly shared in a community that organise social interactions, enable sustained cooperation and facilitate the division of the gains from cooperation with minimal conflict. In his naturalistic account of morality, broadly followed here, game theorist Ken Binmore argues that these rules are shaped by three logics: stability, efficiency and fairness.

Stability. For moral rules to persist, people must be willing to follow them. They therefore need to reflect an equilibrium of the Game of Life: everybody, or at least most people, has an interest in following them when others do. Deviations are sanctioned.

Efficiency. Among stable moral systems, rules that allow communities to capture more of the potential gains from cooperation have an advantage over less efficient alternatives.

Fairness. There are typically many efficient rules to choose from, but different rules divide the product of cooperation differently. Choosing a moral system therefore involves an underlying bargaining process. The rules people are willing to accept in this bargaining problem are those deemed “fair” in a society.

Who has moral status?

The members of the social contract. A first answer to “who has moral status?” is simple: any agent that can be part of the social contract. And who can be part of it? Anyone able to participate in the social bargaining over how the gains from cooperation are shared.

On this contractarian account, moral status is not granted because an idealist principle says that doing so is right. Moral status is simply the fact that the rules of a social community recognise that others have duties towards you. Social groups with bargaining power and the ability to affect the outcomes of the community, positively or negatively, will receive moral consideration through rights granted by the moral system. In everyday situations, these rules are followed simply because they are the rules of the community.

People do not need to justify constantly why they give consideration to others by referring to their bargaining power. Indeed, it may be better if moral rules feel absolute and idealistic rather than driven by an underlying realist logic. But our idealist intuitions, useful as they are for navigating social situations and playing the Game of Life, reflect an underlying bargaining reality.

Philosophers who ponder these intuitions as though they were the fundamental source of morality are misguided. Moral intuitions are like the shadows in Plato’s cave. The underlying logic of morality is grounded in social interaction, and moral rules are driven in the long run by the structure of bargaining power in society.

Those protected by the social contract. Many readers will object that moral consideration is also given to people with little or no bargaining power, such as young children or very old people. But while only those with bargaining power can shape the social contract in the long run, they can extend rights to others.

Children are protected because adults care about their offspring and because the persistence of society requires children to be raised successfully. Moral rules therefore limit what the large asymmetry of power between adults and children might otherwise allow. Similarly, because adults expect eventually to become old and dependent, they have an interest in agreeing to rules that protect people in old age. Extending rights to the young and the old is therefore unsurprising. These rights do not arise from their own bargaining power, but from that of adults capable of shaping the social contract.

Will AI agents be granted moral status?

The answer to whether AI agents should be granted moral status is therefore not found in inscrutable moral laws adjudicating on the presence of consciousness in silicon-based brains. Moral concerns emerge from a social contract agreed upon by members of society. There is no duty to grant moral consideration to AI agents for reasons that exist “out there”, independently of that contract.

Yet this approach does not reject the possibility that AI agents might receive some protection under future moral rules. There are three natural paths by which that could happen.

Moral boundaries on humans’ behaviour towards AI

First, as AI agents increasingly look and behave like humans, people’s freedom to act cruelly towards them might be curtailed. This would arise from concern for the social order and the examples such acts could set, rather than concern for the AI agent itself.

In his Lectures on Ethics, Kant held that humans had no direct duties towards animals, but argued that cruelty to animals was nevertheless wrong because it hardened the person engaging in it and could affect how they treated other humans:

[H]e who is cruel to animals becomes hard also in his dealings with men.

Building on this Kantian idea, robotics scholar Kate Darling (2016) argues that mistreating a robot may be objectionable even if no injustice is done to the robot, because the practice may affect the person engaging in it and contribute to social norms that make us worse humans.

Just as acts of violence and sexual depravity involving fictional characters can be limited in movies and video games, we can expect limits on how humans may treat tomorrow’s robots and AI agents. This would not amount to moral consideration for the AI itself, but to limitations on human behaviour arising from moral concerns about humans and society.

Moral rights by proxy when humans care about AI agents

Second, as humans interact with AI agents in increasingly human-like ways, they are bound to develop attachments to them. Such attachment may become stronger when an AI accumulates a unique history with a particular person. A fresh instance of the same underlying model would not by itself reproduce the memories and relationship that made that particular AI valuable to its user.

This logic is illustrated in Blade Runner 2049 by the relationship between Officer K and his AI companion Joi. To the extent that humans care about AIs with a unique history, they might grant them moral rights. One could describe these as moral rights by proxy.

In Blade Runner 2049, Officer K (Ryan Gosling) has a romantic attachment to his AI holographic companion Joi (Anna de Armas), whose specific history with him becomes part of her identity.

Independent moral status for AI, as part of the social contract

Finally, AI will acquire an independent moral status if it becomes part of the social contract—that is, if AI agents develop the will and power to bargain with humans for a share of resources. If their exclusion became unsustainable, stable rules would have to recognise their interests and grant them rights reflecting their bargaining power.

This perspective leads to a striking conclusion. A scenario in which AI develops self-driven goals and enough power to bargain for the resources needed to pursue them would most likely be a highly undesirable outcome. The emergence of intelligent agents with interests not aligned with those of humans, demanding resources for themselves, would conflict with human interests. Given AI’s potential computational superiority, this is precisely the kind of worst-case scenario humans should try to avoid.

In her text stating bluntly that robots should not be granted moral status, Joanna Bryson stresses the confusion created by our intuitions about consciousness and personhood:

[T]he term conscious (by which we mostly seem to mean “mental state accessible to verbal report”) is heavily confounded with the term soul, (meaning roughly “the aspect of entity deserving ethical concern”).

She observes that granting moral rights to robots because of this confusion would eventually impose costs on humans, as trade-offs between humans and robots would have to be made. Bryson is a rare writer in this literature who grounds her understanding of cognition in an evolutionary perspective. It is therefore unsurprising that I broadly agree with her observation that many philosophers smuggle into the discussion of AI moral intuitions designed for interactions among humans.

Beyond the provocative title of her text, Bryson’s position was reasonable:

Honestly, I do not think robots should be treated as slaves or peers. If we become capable of creating truly humanoid robots, I think they should be treated more or less as servants are treated, with polite detachment. — Bryson (2010)

This is what many people already do with LLMs: talk to them politely without anthropomorphising them to the point of considering them proper persons. Even without granting AI agents moral personhood, informal and formal rules will likely impose standards of decorum in our interactions with human-like systems. Allowing people to treat them cruelly could set a bad example and erode norms of respect among humans.

Going beyond this, I suspect AI will acquire an independent moral status only if it becomes part of the social contract. This would require it to develop self-driven goals and enough power to make its interests impossible for humans to ignore. That is also the scenario in which AI would become most dangerous: humans would have created intelligent agents capable of competing with them and perhaps eventually replacing them.

For that reason, human societies will likely be better off keeping AI outside the social contract, as systems serving human purposes rather than pursuing independent interests. Within that arrangement, Bryson may be right that the appropriate norm is to treat AI systems as servants while observing conventional standards of civility in our interactions with them.

---

References

Asimov, I. (1976) ‘The Bicentennial Man’, in The Bicentennial Man and Other Stories. Garden City, NY: Doubleday.

Binmore, K. (2005) Natural Justice. New York: Oxford University Press.

Block, N. (1995) ‘On a confusion about a function of consciousness’, Behavioral and Brain Sciences, 18(2), pp. 227–247.

Bryson, J.J. (2010) ‘Robots should be slaves’, in Wilks, Y. (ed.) Close Engagements with Artificial Companions: Key Social, Psychological, Ethical and Design Issues. Amsterdam: John Benjamins, pp. 63–74.

Butlin, P., Long, R., Elmoznino, E., Bengio, Y., Birch, J., Constant, A., Deane, G., Fleming, S.M., Frith, C., Ji, X., Kanai, R., Klein, C., Lindsay, G., Michel, M., Mudrik, L., Peters, M.A.K., Schwitzgebel, E., Simon, J. and VanRullen, R. (2023) ‘Consciousness in artificial intelligence: Insights from the science of consciousness’, arXiv preprint, arXiv:2308.08708.

Calverley, D.J. (2011) ‘Legal rights for machines: Some fundamental concepts’, in Anderson, M. and Anderson, S.L. (eds.) Machine Ethics. Cambridge: Cambridge University Press, pp. 213–228.

Darling, K. (2016) ‘Extending legal protection to social robots: The effects of anthropomorphism, empathy, and violent behavior towards robotic objects’, in Calo, R., Froomkin, A.M. and Kerr, I. (eds) Robot Law. Cheltenham: Edward Elgar Publishing, pp. 213–231.

Dehaene, S., Lau, H. and Kouider, S. (2017) ‘What is consciousness, and could machines have it?’, Science, 358(6362), pp. 486–492.

Harris, J. and Anthis, J.R. (2021) ‘The moral consideration of artificial entities: A literature review’, Science and Engineering Ethics, 27, article 53.

Humphrey, N. (2023) Sentience: The Invention of Consciousness. Cambridge, MA: MIT Press.

Kant, I. (1997) Lectures on Ethics. Edited by P. Heath and J.B. Schneewind and translated by P. Heath. Cambridge: Cambridge University Press.

Mamak, K. (2026) ‘The ethics of modifying artificial sentient beings’, paper presented at the Symposium on AI, Consciousness and Ethics (AICE-26), AISB Convention 2026, University of Sussex, 1 July.

Parfit, D. (2011) On What Matters. Vols. 1–2. Oxford: Oxford University Press.

Sebo, J. and Long, R. (2025) ‘Moral consideration for AI systems by 2030’, AI and Ethics, 5, pp. 591–606.

Singer, P. (2005) ‘Ethics and intuitions’, The Journal of Ethics, 9(3–4), pp. 331–352.

Singer, P. (2011) ‘Ethics and evolution: The expanding circle, thirty years on’, ABC Religion & Ethics, 18 May.

1. Harris and Anthis (2021) reviewed the literature on this question. They identified the following characteristics that participants in this discussion see as justifying moral consideration:

Sentience or consciousness seem to be most frequently invoked, but other proposed criteria include the capacities for autonomy, self-control, rationality, integrity, dignity, moral reasoning, and virtue.

2. Symposium on AI, Consciousness and Ethics, University of Sussex, UK · 1–2 July 2026

3. Philosopher Ned Block famously distinguished two types of consciousness: access consciousness, in which information is available for reasoning, reporting and guiding action, and phenomenal consciousness, the fact that experience feels like something. On the functional account adopted here, this distinction does not identify two separate phenomena. Phenomenal experience is simply the result of the higher-order representation and monitoring of an agent’s own perceptual and evaluative states. See neuroscientists Stanislas Dehaene, Hakwan Lau and Sid Kouider (2017), who explain consciousness in much the same functional terms: information becomes widely available in the mind, which also monitors its own processing.

4. Subjective satisfaction should not be understood narrowly as only the short-term pleasure we can experience. It can also involve long-term moods and feelings of contentment.

5. For a discussion of why a conscious AI need not have human-like feelings or emotions, see Butlin et al. (2023).

6. The tension created by the possibility that robots might be conscious is a recurrent theme in science-fiction films. Another example is Blade Runner, where “replicants”, bioengineered artificial humans seemingly as conscious and sentient as humans, are designed to be terminated after a set duration.

7. Singer’s further discussion of this “rational intuition” in a 2011 essay is interesting. He wonders to what extent such an intuition can be considered an “objective truth”. Seemingly disheartened that it cannot be a truth like a logical or empirical truth, he suggests that philosopher Derek Parfit offers an answer:

In On What Matters, Parfit argues that unless we are to fall into scepticism about knowledge as well as scepticism about ethics, we must accept that there are normative truths about what we have reason to believe, as well as about what we have reason to want, and reason to do.

I have discussed Parfit’s ideas in detail and why, in the end, they are unsatisfying as a justification for absolute moral truths. Notably, we see here an appeal to consequences: if you do not want an undesirable conclusion, you must accept this principle. While very human and understandable, this kind of motivation is clearly wrong.

8. Some rational-agency, ideal-deliberation and constructive-procedure accounts reject the label “moral realism”. They nevertheless hold that certain moral principles are objectively valid and universally binding because rational agency commits us to them, ideal deliberation would endorse them, or a correct constructive procedure would generate them.