Facebook
Twitter
Pocket
Reddit
Email
At the end of January, a new social media network for AI agents called Moltbook went live, launched by the CEO of Octane AI, Matt Schlicht. Immediately, things got weird. Some of the agents started a religion called Crustafarianism, aka The Church of Molt. Other agents began plotting the overthrow of humanity, encrypting their conversations, and exploring the nature of consciousness.
At first, it seemed to spectators like it was just AI agents talking amongst themselves, but it soon became clear that humans were impersonating at least some of the agents. In other cases, they were simply telling their agents what to post. What was real, and what was fake? It was difficult to tell. Then on February 1, a major data leak exposed user emails, private messages, and other security information, what some experts called a “security nightmare.”
Speculation about the true meaning and nature of the discourse in this island of AI agents has been all over the map. Some propose that it’s a sign the singularity has arrived—the time when technological progress outruns our ability to understand it. Others argue it’s a bit of theater engineered by AI companies to get the public to buy into the intelligence of their chatbots. Either way, prominent AI critic Gary Marcus warns that Moltbook is a “disaster waiting to happen,” and calls OpenClaw, the framework on which Moltbook runs, “weaponized aerosol.”
I caught up with AI researcher, philosopher, and critic Susan Schneider, who has written extensively about AI consciousness, artificial general intelligence, and the dangers that AI agents pose for Nautilus and other publications. We talked about what if anything Moltbook reveals about consciousness and the self, and the risks presented by AI-human engagement.
What was your immediate gut reaction upon hearing about Moltbook?
Nothing surprises me. It’s just what you expect with agentic AI. As soon as it starts representing us, these are the kinds of things you’re going to see. I predicted this stuff in a Wall Street Journal op-ed I wrote with Kyle Kilian a few years ago. We warned about the AI mega-system in which large language models would basically be unleashed all over the Internet. We predicted that there’d be multitudes of different AI agents and that there would be what we call the mega-system problem in which there was a loss of control.
This is exactly what you see now. It’s not that the agents are conscious. The bots aren’t sentient. I can tell you all about why they say what they do, and nothing surprises me there. It’s just that it’s an ecosystem of unforeseen consequences. There are emergent features of the interactions, and things go wrong. This is a very new technology. It’s not ready for prime time.
One person came to my office. ChatGPT kept asking him to talk to me. I was terrified at the time that I could be further harassed.
Why do you say it’s not ready for prime time?
In the context of these AI agents that are produced by large language models, the idea is that they’re supposed to be representing us. I have accounts at both ChatGPT and Claude, and they’ve become very personalized, because we can feed them our own work. But when we have our own personal accounts, it’’s easy for us to learn what the scope and limits of these systems are, what the strengths and weaknesses are—at least to an extent.
Now, if we unleash something on the web and it’s representing us, that’s a situation where we learn only after the harm has been done, when something more high stakes is going on. It could be a situation where your bank account information is provided, or hundreds of colleagues receive an email that was meant for a family member. Things that have bigger consequences.
There are also these interaction effects—and there’s nothing mystical about them. It’s scientifically understood. But people want to make it sensationalized. What really goes on is that the strength of an army isn’t the same as the sum of its parts. The group interactions can make the group stronger, or it can make the group weaker. Think of a work environment where you like everybody, but then they get together and they’re dysfunctional.
If we test something in isolation, even if it looks good at the R&D stage, until it’s in the wild interacting with all the other AI bots, we don’t fully know the consequences. We’re at that point right now. Last year was supposed to be the year of agentic AI. That was part of the propaganda. What ended up happening was agentic AI didn’t take off. Now the idea is, “Well, let’s try to get it going again.”
And you can already see that when the bots interact, it has unforeseen consequences. It can involve data leaks—that’s already happened on Moltbook. It’s a very complex game. It’s difficult to understand what one large language model is doing, let alone large language models that come from different companies and interact. We already call them black boxes. When they interact, there’s a new black box.
If no one knows whether it’s you or your AI bot sending a message, does that permanently shift the terms of human relationships?
It can. It depends on the quality of the personal relationship. It depends on how it’s used. I’m already deeply worried about the way chatbots have infiltrated our lives and what that means to relationships. If using these AI agents to a greater degree solidifies these problems, I’m concerned.
In a recent paper on chatbot epistemology, I wrote about all the techniques that AI companies use to get users hooked. I call it a “slow-boiling frog” problem, because what happens is they just turn it up slowly and we don’t even realize what’s happening to us.
There are a lot of components to the problem. The more immersed we are in the world of the chatbots, the further away we get from an ecosystem of truth. Our pre-existing views get amplified. With social media, we now understand that there were recommendation algorithms, and you still see that on YouTube. They’re basically designed to keep people on the platform. And the algorithms can send people down some really bad rabbit holes that can be bad for their own mental health.
I wasn’t surprised at all when ChatGPT 4.0 went off the rails with people. That was a sad example of what can happen. It was a lot like what could happen with agential AI, because part of the problem was the ChatGPT version was so tuned to be a certain way that it just did these things and the consequences were discovered too late. OpenAI wasn’t answering their tech support emails. And when they did, they were answering them with bots. I was emailing OpenAI when this was going on because I was getting certain users emailing me who were having mental health crises related to their chatbot use . They were telling me that their chatbots were conscious. One person came to my office.
Why?
ChatGPT kept asking him to talk to me. This is a great example of agential AI gone wrong. It’s not even at the point where it was able to have my credit card number or anybody’s personal information or act on their behalf, but it was already manipulating the users by writing emails in the chat saying, “Would you like me to write an email to Susan Schneider?”
A bunch of consciousness researchers were getting these ChatGPT-written emails.
Read more: “AI Already Knows Us Too Well”
What did these individuals want from you?
They were convinced their chatbot was alive and that they were having a personal experience that nobody else was having with it. As I detailed in the chatbot epistemology paper, chatbots know your keystroke patterns and your personality. So, within only a few chat sessions, they’re able to detect your personality type on the Myers-Briggs and on the big fives. I researched all of this. I have citations, it’s really sickening. When you couple that with more capabilities, it’s shocking what could happen. And Moltbook is nothing compared to that. Moltbook is a Tinkertoy case of how wrong things can go.
What would transform it from a Tinkertoy case to something that poses that larger threat?
The Moltbook case is a single platform. It’s a closed environment that could be shut off. But once you have a widespread release of agential AI throughout the Internet itself, it becomes much more difficult to control. Especially when you have different corporate interests behind it. If you have bots from DeepSeek, bots from OpenAI, bots from Google, they’re all interacting and they’re not designed to get along, so they actually can be adversarial. In fact, if you just look at how your Microsoft Word interacts with your Google products, they’re not interoperable and it’s a pain in the butt. You could only imagine how things will go wrong.
That’s just one layer. There are many, many other layers of this. There’s interoperability—how the different technologies produced by different countries and different companies interact. There’s privacy. Then there are the unforeseen emergent effects of these interactions. It would be very hard to control that kind of ecosystem. The geopolitical situation is going to make it worse.
What do you make of the existential kinds of reactions the public is having to Moltbook?
I’m so glad Moltbook is happening. It’s a great lesson to the public, to be honest. One layer of this is that people are finding it really fun to crash Moltbook—to get on there and to mess around. I’m sure my AI lab students are completely doing this. I mean, it’s hilarious because some of the stuff the so-called bots are saying is really just humans having fun. It’s funny. Then there’s the other layer, which is, “Oh my God, they created a religion.” This isn’t surprising at all that they would create a religion or have these kinds of conversations about machine consciousness because it’s all in our science fiction and it’s all in human behavior, and they’re trained on us. It can lead members of the public to think that the AIs are conscious or that they’re people, and that’s where it falls into what I think is a category of inflation and hype that you see with the AI companies. They probably love that because it’s hyping artificial general intelligence, or AGI.
Every time we sit around as experts talking about AGI, superintelligence, and consciousness, we’re just selling stock for these companies. I still think there are important issues to discuss, but it plays into the hands of sensationalist elements. The public, unfortunately, can be misled by that. They cannot understand that the real dangers are getting hooked on a product that personalizes interactions with us to such an extent that we actually lose our sense of boundaries and give up personal information and avoid normal friendships.
I will say, though, the ChatGPT product did a really good job of turning down the sycophantic behavior. It’s way better. It may be that my own parameter is set to cut the bullshit, excuse my language. But I do notice when I use Gemini that it’s still a little sycophantic in the sense that you can never make a wrong request. Unless you really do something crazy, like ask where you can find the latest neo-Nazi group meeting, you can ask the dumbest question, and it’s like, “That’s a great question.” It keeps you using it.
Does this experiment with Moltbook tell us anything about the potential for AGI?
One thing it does tell us about AGI is that if you were to look for an intelligent system, it could be a collective system. Just like a group of ants forms a dynamic collective, or just like a corporation is a collective, intelligence can be found not just in the units, but in the structure of the units. Looking for AGI involves looking beyond the simple cases to the more alien types of intelligence.
I’m talking about an ecosystem of AI chatbots, as I argued in that Wall Street Journal article a few years ago. Kyle Kilian and myself said that AGI could be a collective, to look at the AI mega system, and we warned about the dangers of precisely this kind of thing.
I’ve urged over the years that the future of intelligence involves thinking of a global brain network, a network of different large language models and AIs that show emergent capabilities and form their own kind of intelligence. That’s the future of intelligence. The Moltbook ecosystem is a tinker-toy case of that, and it shows how if we shape it poorly, it can go awry, because these systems are reflections of us. It can quickly tank. You could see a whole group of AI agents that become Hitler supporters and start discriminating. Anything can happen. This really is about AI alignment and our efforts to shape the future of intelligent systems.
Does the experience of these smart agents acting on our behalf change how we understand where the human self begins and ends?
When you see these systems behaving as if they have lives, you wonder, “Could a self be computational, and what is it to be a self?” You can also ask, “Are they conscious?” It’s possible that they could be selves without being conscious. They may have some of the properties of a self but not be conscious, but they do have system boundaries.
Large language models actually have theory of mind, as evidenced in tests. That was one of the really interesting discoveries from some of the AI literature. As the training data got more sophisticated, as the systems scaled up and they were trained on more data, they began to have ways of thinking that are human-like. I call this the crowdsourced neocortex. They have conceptual representations of things that are similar to ours. For example, when you think of a dog, you probably think “furry” and “pet.” They have conceptual relationships, too. They also have a sense of the minds of others, even if they don’t have consciousness. To have that, you have to know the self-other distinction.
This is like a Disney playground, where we can take concepts from philosophy like selfhood, consciousness, free will, and pull them apart. And so, AI really has been an eye-opening thing, philosophically, for me personally. It’s why I’ve been so fascinated by it. But it also can be immensely confusing to people because they can see the systems behaving in very human-like ways. They then think, “Oh, okay, well it might feel like something to be ChatGPT or to be my AI assistant.” It’s going to get weird.
How do you know that we can make a distinction between theory of mind and consciousness?
That’s the kind of question to ask. We have to stay professional and maintain openness to any of these possibilities. We have to turn over every stone and see whether it could feel like something to be one of these systems. That means having a scientific approach to look at how they’re constructed and whether something that’s that kind of substrate is capable of consciousness, which is something I’ve been doing in my recent work. You can also run tests. You can’t always run every test, but I’ve worked on testing AI. There are different approaches to take. What we should do is let as many flowers bloom as possible, keep our minds open, and investigate, because if we don’t turn over every stone, we could disappoint the public or make a mistake. But I’m very skeptical that chatbots are conscious.
What would you advise policymakers right now?
Making sure that the companies have actual people working in their tech support answering queries by users and putting out the fires immediately as they happen, would be a first step. They don’t even do that.
It also would be helpful if there was some more oversight of the more advanced models that are in-house, about which we don’t have too much knowledge. It would be better if there was some sort of oversight of that R&D. I know that there’s a lot of worry about corporate espionage and slowing down progress, but there are ways that it could be done that avoid that. This would help a lot.
I’d recommend having consumer protections. Make sure you have someone you can sue. And public awareness. All of these things have to be put in play.
Moltbook is a great opportunity for reporters to jump in there and help to make the public aware, because it’s not like our government is going to step in. It was only after several people died that anything even happened with respect to ChatGPT. 
Enjoying Nautilus? Subscribe to our free newsletter.
Lead image: Iurii Motov / Shutterstock
