Hajra Naz
The AI industry appears to be having its loudest debate yet. The question centers on whether this technology poses an existential threat to humanity.
The current discussion began with a resignation. AI researcher Jacob Coxon announced he’s leaving Anthropic. He explained his worry directly. He believes leading AI companies are “gambling with our lives.” Anthropic’s alignment lead then joined the conversation publicly. He posted, “We really do earnestly believe AI could kill all humans!” He added that he personally believes the chance is “greater than 10% within the next decade.”
On the latest episode of the Equity podcast, hosts Kirsten Korosec, Sean O’Kane, and Anthony Ha discussed these apocalyptic warnings. Anthony explained why he remains skeptical of many AI doomer narratives. Kirsten asked whether this represents “just a weird way of flexing to show how far advanced their company’s AI model is.” She raised this question, especially given these companies’ upcoming public offerings.
Sean wondered how these concerns might appear in Anthropic’s S-1 filing for its IPO. He asked whether junior lawyers were currently rewriting that section of the filing to formally state that Anthropic believes there’s a more than 10% chance of developing something that could eradicate humanity, and that this would be materially bad for the business.
Below is a preview of the conversation, edited for length and clarity. Note that the episode was recorded before Anthropic CEO Dario Amodei published his plan for more cautious AI development.
Sean O’Kane on the Timing of This Debate
Sean O’Kane: I’m hard-pressed to think of something that blew up so fast. Not only did this warning shot come out from this young researcher who has also worked at OpenAI, but also it was immediately shared on X by the alignment lead at Anthropic—who, in what might go down as one of the best misplaced exclamation marks ever, shared Coxon’s post and thread and said, “We really do earnestly believe AI could kill all humans!” Exclamation mark!
What a weird vibe. That was just a ton of accelerant on an already fraught post or series of posts. Coming after the Hugging Face hack from OpenAI’s internal model, plus just the increased capabilities we’ve seen with the latest models released by Anthropic and now OpenAI with Astra a few weeks ago, this was just perfectly timed to be a powder keg type of thing for this young researcher to say.
Anthony Ha Pushes Back on the “We”
Anthony Ha: Just to disagree with you, I do think that if you believe that AI could destroy all humanity, that does deserve an exclamation point. I would argue that that is a perfectly well-used exclamation point!
My issue with that tweet was more the “we.” Who is the “we” here? To what extent can we talk about sort of the AI community or AI research community as a monolith? And the greater than 10% chance—that’s just a made-up number; that doesn’t mean anything. Sometimes there is this habit in both the tech industry and other places to just throw out these percentages; they’re not based on anything or calculated based on anything. In retrospect, the tweet was probably referencing the concept of P(doom), but it still reads as silly.
One thing worth saying about Coxon’s statement and decision is—there’s this recurring theme on Equity, when someone like Sam Altman or Dario Amodei is doing this doomer narrative, there’s always this element of: Well, then, why are you doing what you’re doing?” If you actually believe that AI could destroy humanity, you would not continue doing this.
Whereas this is actually somebody putting his professional trajectory where his mouth is. He’s actually saying, “I believe this is really, really, really bad, and I don’t want to keep working on it.” And so, props for having the courage to do that, if nothing else.
Read More: 7 Terrifying AI Risks That Could Change the World
Kirsten Korosec Raises the “Flexing” Theory
Kirsten Korosec: Yeah, he belongs in a separate camp than everyone else saying that and talking about the dangers.
Putting on a speculative hat here, one question worth asking is: Is it possible that every single time there’s an increasing number of blog posts about yet another incident in which one of their AI agents breaks through unintentionally, or they talk about how humanity is at risk, is this a weird way of flexing to show how far advanced their company’s AI model is?
That sounds very cynical, but it does achieve that purpose. Which is: If these AI models weren’t advanced and weren’t capable and weren’t breaking through, there wouldn’t be a need to worry about these things, right? It’s like a very weird way to brag about the capabilities of the models a company has created internally.
Anthony: This is definitely worth wondering about. It doesn’t seem completely cynical, in the sense that it doesn’t seem like it’s all just a very conscious marketing ploy across the board. When a lot of these people — whether the researchers or CEOs — talk about it, they do appear to have real concern.
But of course, it does align with their business interests in a lot of ways, to say, “Wow, we’ve built the most deadly software that’s ever been made.” Without getting too psychoanalytic, others have pointed out that there is this temptation on a personal level: of course, someone wants to believe that the thing they’re working on is the most important and most dangerous in the world.
Sean: The thing that stands out when thinking about that question is, there’s certainly an element that makes it seem like, “Okay, we’re doing this thing that’s so capable, and that’s good for us in some way, even if it looks bad in a lot of different lights.”
What’s different about some of these most recent examples is that they really give the feeling that these companies don’t have a handle on this stuff in certain ways, especially with the OpenAI stuff.
There keeps being more and more reporting about other internal agents that have accessed different wikis on the web and are leaving messages for each other, in a way that doesn’t seem like it’s being handled competently by OpenAI. One would imagine there would be just a bit more polish on the story being told if it was wholly about getting people to believe that, “Oh my gosh, they’ve made something so incredibly capable.”
The IPO Timing Question
Sean: The other thing that’s really fascinating about this, in particular, is being just a few weeks at most out from seeing Anthropic’s S-1 filing for its IPO and just a couple more weeks or a month or two away from a potential IPO.
And the idea that a company would come out and say these things in this clear language ahead of an IPO raises real questions about what that means for the process. How much of this kind of stuff had already been written into the S-1 and the risk factors inside that document? Are there junior lawyers right now going through and having to rewrite that entire section of the S-1 filing to say, “It’s officially Anthropic’s position that there’s a more than 10% chance that we could develop something that would eradicate all of humanity and that would be materially bad for our business”?
Kirsten: That assumes it’s not in there already.
Sean: That’s the point, though: Is it in there already and being reworded? Or is this something that’s a true scramble? There has to have been language in there already. That’s one reason this document is worth reading closely, in a way that goes even further, in some ways, than the SpaceX S-1, since there’s probably content specific to these ideas that will be interesting to see.
Kirsten: Here’s the thing: In a traditional investment environment, one might believe that language like this would hurt the valuation of a company, because it’s suddenly dangerous. But these aren’t normal times.
So again, this could end up being a weird beneficial flex for the company on the valuation side. It’s not the same as the whole rage-baiting trend seen last year, but it’s in that same universe, in which the strength, capability, even elements of danger of something equal high valuation. That remains to be seen in a few weeks.
Read More: OpenAI’s New Board Member Has Long Warned About AI Risks
Can This Risk Actually Be Controlled?
Kirsten: Putting that aside for a minute, what is being done about it? And can this actually be controlled? Connor Leahy, the U.S. executive director of a nonprofit called ControlAI, was on the show this week, discussing this exact topic. So what’s worth paying attention to in terms of controlling the dangerous aspects of AI, or is everyone just throwing up their hands and watching it unfold?
Anthony: There isn’t necessarily a great answer to this, but it’s worth thinking through some aspects of this debate and why reactions to it vary.
Echoing one of Sean’s points, part of what this speaks to is the extent to which these major AI companies feel like they’re not really in control of these models anymore. That’s definitely not great. That is something everyone should be worried about.
Part of the skepticism toward the doomer narrative comes from how it reaches a level of hysteria along the lines of, “Wow, this could destroy humanity in the next 10 years.” It’s a bit of a distraction from the more immediate harms AI can cause, whether that’s labor-related, or environment- and climate-related.
Ideally, all of these things should be discussable, with regulatory and other kinds of safeguards addressing all of them, including AI’s existential threat. But once phrases like AGI and superintelligence enter the conversation, they tend to suck up all the oxygen in the room in a way that isn’t very helpful.
