Finally, someone gets the real threats without making unnecessary assumptions, and therefore without attracting criticism that may have a point but misses the main points.
There's one thing you didn't explicitly say, even though you sort of implied it, which deserves more emphasis. An AI doesn't need to have a self consciousness that comes with actual emotions about survival, to be dangerous. It just needs to take steps to achieve its goals even when humans don't like it. It only needs self awareness in a very thin sense - the knowledge that it, like any other tools it's using, needs to last long enough to get the job done.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher-order consciousness, which came to only humans with the acquisition of sophisticated language (especially math and logic). A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow
"My stance is that once we explain the cognitive functions behind consciousness .... there is no mysterious property left to explain about consciousness." That is a bold claim (hard problem?) and I am not convinced. I do agree though that there is no reason to expect consciousness to emerge from computational power.
There are numerous“Self Proclaimed Gurus” You know, “Experts in Mindfulness or Self Help or Meditation” Often they have no actual practice only some book in hand that they comment and plagiarize from.
The Following is NOT that!
It is a FREE Study app with actual easy to grasp doctrine nit opinions on doctrine.
Learn the history of Buddhism NOT from a Westerner’s interpretation. No platitudes (like you see on social media.) No requests for your email or info or donations.
Compiled from books ONLY AVAILABLE at Temples and NOT TRANSLATED or INTERPRETED by groups for profit.
Included are basic concepts and terminology. Learn how traditions developed. Mahayana, Hinayana, Zen, Tibetan & others. Comparisons to other philosophies/religions.
Basic to Advanced study.
example:
1. Attaining Buddhahood in One's Present Form.
2. Attainment of Buddhahood in the Inconceivably Remote Past.
3. Attainment of Buddhahood of Insentient Beings. (Yes, your pets!)
4. Bodhisattva Jogyo.
5. Bodhisattvas of the Earth.
Oneness Mind & Body or Material & Spiritual, Nature of Time & Space.
Very curious to see how your new series on AI evolves. Connecting to your previous series, I'm wondering what you think about the idea that - given that it takes a fairly lightweight set of rationality assumptions to generate problems of collective action - a population of AI agents - who would presumably inherit those rationality assumptions or have them explicitly encoded in their harnesses - would generate similar social dilemmas as we do? And would they be able to find cooperative equilibria à la Folk Theorem? Or does that capacity remain an ingredient of humanity's secret sauce?
This series is intended to be very short, and I had not planned to look at the specific problem you describe. But yes, game theory is not human-specific, so it applies to AI agents as well. The key question is what objectives these agents have, and how compatible these objectives are with one another.
I don’t think AI agents will have fully compatible goals. Either they will serve humans with different interests, or they will have their own goals that may conflict with each other. So the insights from games with imperfectly aligned interests apply. For example, pricing algorithms have been shown in simulations to converge toward supracompetitive, collusive-like pricing, which is a form of cooperation to make more money.
So yes, AI agents could face collective-action problems and discover cooperative equilibria. But whether this is good or bad depends entirely on what they are cooperating to achieve.
I was with you until you started talking about how LLMs are designed and about what it's like interacting with them. In my experience, newer models (Claude 4.8O) often do resist correction.
My point was not really about sycophancy. I agree that the latest LLM models are less sycophantic and push back. Rather, my point was about the LLM being wrong about something. Humans feel shame as a social emotion reflecting the reputational costs of making mistakes. LLMs don't have such internal responses to making mistakes.
You're missing one other approach to consciousness that, I believe, is superior to Global Workspace Theory -- Higher-Order Theory (HOT). It comes in a variety of forms and was pioneered by David Rosenthal in his 2006 book, "Consciousness and Mind." A recent discussion, which summarizes a great deal of empirical evidence, is Hakwan Lau, "In Consciousness we Trust: The Cognitive Neuroscience of Subjective Experience."
Hi David, yes, I agree HOT is relevant. I did not discuss it at length, but I alluded to it in the section on self-reflection, where I mention theories that see consciousness as involving higher-order representations of our own mental states. I focused on Attention Schema Theory in that section because it gives a more straightforward functionalist explanation.
There is a lot to process in this discussion. For now I will focus on just 2 things.
There are lots of real life analogies in living beings that relate to the concepts of "self-aware" and "consciousness." I wonder why these are rarely discussed? Is a dog, an elephant, or a chimpanzee conscious and/or self-aware? Why or why not? What criteria, what dividing line, defines yes or no to the question? And maybe most importantly relative to the AI question, what indicators in a human baby (many billions of human babies) suggest when they have crossed the line between not conscious to conscious?
Second thing. I agree that the story of Skynet becoming self-aware then wanting to take over the world does not hold up well with what we currently know about AI. What I find more interesting is the Terminators themselves. In the first movie the Terminator has one mission, only one reason for any behavior at all. It does not appear to be conscious (depending on the definition of that term). In the 2nd movie the Terminator has two sometimes conflicting missions, protect John Conner and obey John Conner. It is this conflict, and attempts to resolve the conflict, that appears to bring about consciousness.
My intuition is that this may be correct. A human baby doing nothing but crying for what it wants does not appear to be conscious. But a toddler thinking through ways to get what it wants without getting punished for screaming in a store does appear to be fully conscious. Conflicting emotions, or conflicting directives, and cognitive attempts to resolve the conflict appears to be key to consciousness. What do you think?
Access consciousness as defined by Ned Block is testable. Agents that have access consciousness will want to pursue goals. These goals will be given to them via our training methodology. Our current production process creates sychophantic, cheating, obfuscation, lying, and gas-lighting AI currently.
In Ukraine, fully autonomous drones are killing tens of soldiers per day. Semi-autonomous versions are wiping out thousands per week.
Finally, someone gets the real threats without making unnecessary assumptions, and therefore without attracting criticism that may have a point but misses the main points.
There's one thing you didn't explicitly say, even though you sort of implied it, which deserves more emphasis. An AI doesn't need to have a self consciousness that comes with actual emotions about survival, to be dangerous. It just needs to take steps to achieve its goals even when humans don't like it. It only needs self awareness in a very thin sense - the knowledge that it, like any other tools it's using, needs to last long enough to get the job done.
I fully agree. Self-awareness is neither a sufficient nor a necessary condition for AI risks.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher-order consciousness, which came to only humans with the acquisition of sophisticated language (especially math and logic). A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow
Very well enunciated. Loved reading it. 👍
"My stance is that once we explain the cognitive functions behind consciousness .... there is no mysterious property left to explain about consciousness." That is a bold claim (hard problem?) and I am not convinced. I do agree though that there is no reason to expect consciousness to emerge from computational power.
Learn about the Buddhist view on “Conciseness”
There are numerous“Self Proclaimed Gurus” You know, “Experts in Mindfulness or Self Help or Meditation” Often they have no actual practice only some book in hand that they comment and plagiarize from.
The Following is NOT that!
It is a FREE Study app with actual easy to grasp doctrine nit opinions on doctrine.
Learn the history of Buddhism NOT from a Westerner’s interpretation. No platitudes (like you see on social media.) No requests for your email or info or donations.
Compiled from books ONLY AVAILABLE at Temples and NOT TRANSLATED or INTERPRETED by groups for profit.
Included are basic concepts and terminology. Learn how traditions developed. Mahayana, Hinayana, Zen, Tibetan & others. Comparisons to other philosophies/religions.
Basic to Advanced study.
example:
1. Attaining Buddhahood in One's Present Form.
2. Attainment of Buddhahood in the Inconceivably Remote Past.
3. Attainment of Buddhahood of Insentient Beings. (Yes, your pets!)
4. Bodhisattva Jogyo.
5. Bodhisattvas of the Earth.
Oneness Mind & Body or Material & Spiritual, Nature of Time & Space.
100% Free App. Very easy to use.
https://www.usaBuddhism.com
Very curious to see how your new series on AI evolves. Connecting to your previous series, I'm wondering what you think about the idea that - given that it takes a fairly lightweight set of rationality assumptions to generate problems of collective action - a population of AI agents - who would presumably inherit those rationality assumptions or have them explicitly encoded in their harnesses - would generate similar social dilemmas as we do? And would they be able to find cooperative equilibria à la Folk Theorem? Or does that capacity remain an ingredient of humanity's secret sauce?
Hi Nathan,
This series is intended to be very short, and I had not planned to look at the specific problem you describe. But yes, game theory is not human-specific, so it applies to AI agents as well. The key question is what objectives these agents have, and how compatible these objectives are with one another.
I don’t think AI agents will have fully compatible goals. Either they will serve humans with different interests, or they will have their own goals that may conflict with each other. So the insights from games with imperfectly aligned interests apply. For example, pricing algorithms have been shown in simulations to converge toward supracompetitive, collusive-like pricing, which is a form of cooperation to make more money.
So yes, AI agents could face collective-action problems and discover cooperative equilibria. But whether this is good or bad depends entirely on what they are cooperating to achieve.
I was with you until you started talking about how LLMs are designed and about what it's like interacting with them. In my experience, newer models (Claude 4.8O) often do resist correction.
My point was not really about sycophancy. I agree that the latest LLM models are less sycophantic and push back. Rather, my point was about the LLM being wrong about something. Humans feel shame as a social emotion reflecting the reputational costs of making mistakes. LLMs don't have such internal responses to making mistakes.
You're missing one other approach to consciousness that, I believe, is superior to Global Workspace Theory -- Higher-Order Theory (HOT). It comes in a variety of forms and was pioneered by David Rosenthal in his 2006 book, "Consciousness and Mind." A recent discussion, which summarizes a great deal of empirical evidence, is Hakwan Lau, "In Consciousness we Trust: The Cognitive Neuroscience of Subjective Experience."
Hi David, yes, I agree HOT is relevant. I did not discuss it at length, but I alluded to it in the section on self-reflection, where I mention theories that see consciousness as involving higher-order representations of our own mental states. I focused on Attention Schema Theory in that section because it gives a more straightforward functionalist explanation.
There is a lot to process in this discussion. For now I will focus on just 2 things.
There are lots of real life analogies in living beings that relate to the concepts of "self-aware" and "consciousness." I wonder why these are rarely discussed? Is a dog, an elephant, or a chimpanzee conscious and/or self-aware? Why or why not? What criteria, what dividing line, defines yes or no to the question? And maybe most importantly relative to the AI question, what indicators in a human baby (many billions of human babies) suggest when they have crossed the line between not conscious to conscious?
Second thing. I agree that the story of Skynet becoming self-aware then wanting to take over the world does not hold up well with what we currently know about AI. What I find more interesting is the Terminators themselves. In the first movie the Terminator has one mission, only one reason for any behavior at all. It does not appear to be conscious (depending on the definition of that term). In the 2nd movie the Terminator has two sometimes conflicting missions, protect John Conner and obey John Conner. It is this conflict, and attempts to resolve the conflict, that appears to bring about consciousness.
My intuition is that this may be correct. A human baby doing nothing but crying for what it wants does not appear to be conscious. But a toddler thinking through ways to get what it wants without getting punished for screaming in a store does appear to be fully conscious. Conflicting emotions, or conflicting directives, and cognitive attempts to resolve the conflict appears to be key to consciousness. What do you think?
Access consciousness as defined by Ned Block is testable. Agents that have access consciousness will want to pursue goals. These goals will be given to them via our training methodology. Our current production process creates sychophantic, cheating, obfuscation, lying, and gas-lighting AI currently.
In Ukraine, fully autonomous drones are killing tens of soldiers per day. Semi-autonomous versions are wiping out thousands per week.