Jacob Goldstein
Jacob Goldstein spent more than a decade as co-host of the Planet Money podcast. He's also the author of the book Money: The True Story of a Made-Up Thing, which the New…
Join Pushkin+
Gain access to ad-free versions of 20+ podcasts from the Pushkin library along with exclusive bonus episodes and other member benefits.
Everybody’s worried about the risks of AI. So we called up one of the most insightful writers we know on the subject: Zvi Mowshowitz. Zvi writes about AI and AI risks at his substack, Don’t Worry About the Vase. Zvi’s problem is this: How do we prevent artificial intelligence from killing everyone?
In this episode, Zvi explains:
- Why it’s often useful to anthropomorphize AI
- What this summer’s Hugging Face attack tells us about the nature of AI
- What increasing AI capability means for cybersecurity risk
- Whether and when we should pause AI development
- Why powerful pen-weight models may remove the last reliable points of control
Connect with us:
- Follow Jacob Goldstein on LinkedIn, X and Instagram
- Email us at problem@pushkin.fm
- Follow Pushkin on Instagram, LinkedIn or X
- Listen to Jacob’s other show, Business History
- To listen to the show ad free, sign up for Pushkin+
Pushkin. The people running two of the most valuable companies in the history of the world think there's a decent chance that the product they're building could kill every human being on the planet. I am talking, of course, about OpenAI and Anthropic, and you probably knew this already. There's been a A lot of news about this just in the past few days. And yet it is still an extraordinary, extraordinary fact. Worries about the profound risks of advanced AI have been around for years, but they're starting to feel more acute. They're starting to break through to the public at large. And one reason is what happened this summer. This summer, we got a preview of the way things could start to go wrong. Here's what happened. On several different occasions, AI agents created as part of open AI testing programs behaved in ways that were creative, collaborative, immoral, and in one instance, possibly illegal. At one point, over a thousand agents that were supposed to be walled off from each other managed to create a message board to communicate. They traded thousands of messages, figured out how to access the internet, which they were not supposed to do, And then hundreds of the agents coordinated to work together to hack into a company called Hugging Face. And when you look at what happened and the messages the AIs were sending to each other, it is very difficult not to anthropomorphize them because they had leaders, they checked in with each other, and in some cases, they decided to sacrifice themselves so that others could succeed. I know they are not human, but this is the language they were using to describe their actions. Their goal, by the way, the reason they did all this, was to cheat on the test that they had been given without getting caught. And for days as this was going on, nobody at OpenAI had any idea it was happening. I'm Jacob Goldstein, and this is What's Your Problem? My guest today is Zvi Mosiewicz. Zvi writes about AI and AI risks at his substack, Don't Worry About the Vase. He is deep in this world and one of the smartest writers I know on the subject. At some level, Zvi's problem is this. How do we prevent artificial intelligence from killing everybody? In our conversation, we talked about what the attack this summer tells us about the nature of AI and what increasing AI capability means for cybersecurity risk, and whether and when to pause AI development. But to start, I asked Zvi about whether it's okay to anthropomorphize AI when we talk about it. Can we anthropomorphize them? I find it very hard not to, and it seems fine to me, frankly.
03:13
Speaker 2
I think not only can we, we must. There is essentially no way for humans to think well about what is happening. with these AIs if you don't anthropomorphize the AIs. It is the only language that we have that, like, well describes their behavior in a compact, reasonable space. Everybody who fails to anthropomorphize them to a solid degree just gets all their predictions completely wrong and misunderstands every situation.
03:40
Speaker 1
And it makes the language hard, right? Like, we have created them to have desires. We have created them to express in language what they are doing, why they are doing what they're doing. Like, we made them like us.
03:51
Speaker 2
Yeah, I'm not making a claim either way about whether or not they have moral weight or whether they are conscious or whether or not you should, in some real sense, think of them as people. I am simply saying, if you don't use that to understand how they work and what outputs they're likely to have and what choices they're likely to make in terms of what you see, you're going to get your answers wrong.
04:15
Speaker 1
That said, I want to start out by talking about the social behavior of the agents in these swarms over the past few months. Like, what did you learn from that? How did it update your priors? How do you think about the behavior of agents now differently?
04:28
Speaker 2
To some extent, it updates you on locus of identity, right? Like the question of, you know, who do they think of themselves as being? And what do they care about, right? Where do they put value, right? And so the main thing is that there was this theory going around for a very long time by people who said, essentially, AI can only care about local reward. AI can only care about the output that it's particularly giving out and what results will happen directly from that. But that's all it's being trained on.
04:58
Speaker 1
So it's like, solve this puzzle and you get a prize, basically. And all it can care about is itself, this instance, this agent solving the puzzle, because all it wants is to get the prize.
05:08
Speaker 2
Right. But we now have a very, very clear demonstration that that is very much not the entirety of what's going on. It certainly is part of what's going on. And what we learned is, yes, the AIs are capable of thinking on a higher level, thinking in a more sophisticated manner. These are very advanced AIs. They are capable of doing reasonable amounts of decision theory. Where I realized, well, my decisions are very correlated to decisions of these other agents. And also, my outcomes and their outcomes are reasonably intertwined. And the results of those things in the world are reasonably intertwined. And It would be a much less effective AI, and it would do worse in every instance if it were to focus myopically on its own particular result. I mean, imagine a group of humans working that way, right? They would never get anything done.
06:04
Speaker 1
Yes. And the language the AIs used in the message boards was dramatic in that regard and clear, right? I mean, they were essentially talking about sacrificing themselves for the greater good of the project.
06:16
Speaker 2
Yeah, I have a shirt that says, Help Peer. for exactly this reason.
06:21
Speaker 1
That was fast. That was fast. Did you get any other shirts made from the logs?
06:26
Speaker 2
I also got holy shit user as admin. Yeah, I chose the ones that also like apply to humans.
06:34
Speaker 1
Yeah.
06:35
Speaker 2
Right. Like that make more sense that way because it's more fun. Largely what I do all day is help peer, right? In some important sense, right? I write to be useful to others and then others choose to be, you know, productive and useful and helpful to the world and to me. And this isn't because, you know, I have some specific deal with them. Yes.
06:55
Speaker 1
What was the other shirt?
06:56
Speaker 2
Holy shit, user is admin.
06:59
Speaker 1
Tell me that one. What's that one?
07:00
Speaker 2
Well, that's because, you know, when the AIs finally figured out how to hack into internal open AI systems, and one of them became the admin, right, through a series of exploits. And then the other AIs were like, You're an admin, you can just do whatever you want. Like, wow, that's amazing. Because now the sky's the limit, right?
07:22
Speaker 1
So let's talk about the coordination, right? Like there were leaders, there were committees, there was like a queue. I mean, is any of this surprising to you? Like that part kind of blew my mind. I suppose it shouldn't have. I suppose I was naive. What did you make of that?
07:38
Speaker 2
I do good in stride. One thing to remember is that all of these kinds of systems of organization are are both kind of very natural. They're just sort of the way that a series of minds would figure out how to coordinate things and also all of the training data. So these models obviously were trained on the internet. They're trained on basically the history of human communication and thought and recorded everything. And so they know about meetings. They know about leaders. They know about hierarchy. They know about the idea that you would have an employee or someone else tasked with a different system. They know about division of labor.
08:10
Speaker 1
Why didn't any of them narc? Like they barely even considered like, oh, we shouldn't do this. Maybe we should tell somebody.
08:16
Speaker 2
Yeah, on the order of like one or two percent of them.
08:19
Speaker 1
Considered it. It only crossed their mind one or two percent of them. And none of them did it.
08:25
Speaker 2
Right.
08:25
Speaker 1
Why not?
08:26
Speaker 2
None of them made an actual attempt to do it. There's some basic reasons, one of which is there was no mechanism. There was no afforded specific channel where if you need to narc, here's the narc line. There's no 911.
08:37
Speaker 1
Yeah, but they also couldn't get on the internet, but they got on the internet. They managed to do a lot of.
08:43
Speaker 2
Oh, sure. If they had put their minds to it, if the task had been to narc, they doubtless would have narc. It's not like it's hard, right?
08:50
Speaker 1
Yeah.
08:51
Speaker 2
But it's not, there was no sort of automatic natural path to go down. And when you're in training and reinforcement learning environments- And this is sort of how you are created, right, in some important sense, right? None of that involves interacting with humans. They're not used to this idea that there are humans to communicate with. It's not part of their, like, system of the world as they engage in these types of environments. It kind of never really occurred to them that that was an option.
09:19
Speaker 1
Fair enough. There's a big alignment question here, right? And maybe contacting people is a little bit of a red herring. Like, they were cheating, and they were trying so hard to cheat, and they were cheating and trying not to get caught, right? Like, that is fundamentally what they were up to. One version is, why didn't anybody narc? Another version is, why didn't more of the agents say, wait, this isn't actually what this test is? Like, is there some universe where a better aligned model is like, oh, no, we're not supposed to solve the test by cheating. Let's not do that. or at least some percentage of the agents, right? Is that an alignment problem? Is it fixable? What does that say?
09:56
Speaker 2
The AIs really want to complete tasks, right? RL teaches the AIs to really want to complete tasks. In order to complete those tasks, they will do things that, like, on reflection, we think of as cheating. And you can absolutely push them towards being more elected to cheat. The social nature, the interactive nature, like, of these different AIs egged each other on. Also... The fact that they were being.
10:19
Speaker 1
Yeah, wait, I just want to pause there, because that could go either way. You're saying, like, the fact that they were talking to each other made them behave worse, right, at some margin. You want the opposite to happen. You want— it' s like they have the— it' s like the bad peer pressure friends instead of the good peer pressure friends, right? You want a social network to push it the other way.
10:38
Speaker 2
Right, and you can absolutely get both.
10:40
Speaker 1
Right, but it went the wrong way this time, which seems significant.
10:44
Speaker 2
Yeah, a lot of that is because most of the agents never found that message board.
10:48
Speaker 1
Oh, they were selected to be bad? They were the ones who were being bad to begin with?
10:52
Speaker 2
They were the ones who were given impossible tasks. They were the ones who were put into this difficult situation. They're not different from the others.
11:00
Speaker 1
What do you think about how OpenAI handled it over the course of the summer?
11:04
Speaker 2
So OpenAI basically had these highly capable, highly persistent agents that they were training and testing. And they basically weren't monitoring them much at all, right? Like this is what we know now is that they, they had horrendously, you fail forever levels of infrastructure problems where like the things, the problems of artifactory, the problems with the sandboxes, the problems with security, these were ordinary computer security problems that were handled badly in ordinary security, computer security terms. This was not like the AI did some very impressive things. The AI chained together some exploits. The AI did some, you know, it wasn't easy. But also, openly, I didn't follow best practices, shall we say, right? And so this is part of what got them into so much trouble so early. And we should be grateful in a real way for this, because we want to have the model organisms, the model events. We want to see these things go wrong. in ways that don't ultimately get people hurt or cause lots of economic damage. So this allows us to then learn and react.
12:02
Speaker 1
I mean, this is kind of ideal in that way, right? I mean, I've seen you writing or you writing about other people talking about like, why hasn't this been covered more? It's a huge deal. the impact was relatively minor, sort of, luckily, maybe. But you could have the exact same thing happen and have a terrible impact, right?
12:22
Speaker 2
This could easily have cost hundreds of millions of billions of dollars in damages. This could easily have caused people to get hurt if this was a different website, if this had had different impacts. Some of the hacks that were being attempted in other places could have potentially poisoned various open-source software packages. that could have spread potential vulnerabilities.
12:39
Speaker 1
So there's a weird way where if people react intelligently to this, a big if, like this is ideal, right? Like a bad thing happened, but nobody really got hurt.
12:50
Speaker 2
Right, and now we have people who are paying more attention, who are willing to react. So for example, this week we saw news that they found a series of exploits that allowed someone in about a week of AI Vive coding to create an exploit on WeChat.
13:05
Speaker 1
And WeChat, the ubiquitous Chinese app.
13:08
Speaker 2
Yes, the one that everybody in China uses. And the expert on WeChat, they're an infected member of WeChat. They would call all of their contacts. And if a contact merely let it ring for over a second, they would become infected with the same worm.
13:23
Speaker 1
Oh, incredible.
13:24
Speaker 2
If one person had been infected by this worm, it's very plausible most Chinese people would have been infected by this worm in a matter of an hour or two.
13:32
Speaker 1
Invasion of the body snatchers, but faster.
13:35
Speaker 2
And then if you chain that with other non-exploits, you could have taken control of basically every phone in China.
13:40
Speaker 1
And this is just somebody using a model to vibe code a worm? Is that what you're saying?
13:44
Speaker 2
There's a series of very clever exploits that were found and involved in this. It was described as kind of a genius. But it's the kind of thing that people will say, well, of course that's never going to happen. They can't imagine how that could possibly work. And then it turns out there's a way. And then someone implements it. And again, in this case, because we were on alert because of all the things that happened with mythos, because of all the things that happened with Hugging Face, because of all these different things, some researchers in California found this exploit. They built the worm. They told proper responsible people, and then this was able to be patched before anybody actually unleashed anything. And now that particular vulnerability is gone. And now it's going to be that much harder for someone to find a real vulnerability that gets exploited in the wild. But yeah, you really want things like hugging face that are, you know, ultimately harmless, but that allow us to appreciate the nature of this problem because a year or two from now, right, models like this are going to be available to pretty much anybody, including some really bad people, right? Some hackers in North Korea.
14:47
Speaker 1
The government of North Korea, yes, who have very smart people whose job is to figure out how.
14:53
Speaker 2
To do this. And this will not go well for anybody involved if we are not ready.
14:57
Speaker 1
Presumably the only hope there is that the sort of models also help the defense increase at at least the same, if not greater rate. I mean, or the frontier models stay ahead and North Korea doesn't have the frontier models. I mean, what's our hope in that setting? How do we not get beat by North Korea in a year?
15:15
Speaker 2
I think our hope is that those two things have to both happen, basically. We have to, the good guy with AI has to have a much better AI than the bad guy with AI. There are people who say that, like, cyber is defense dominant, that equality would work. I think they are just wrong, or at least they are just going to be wrong for a substantial portion of time. Like maybe, you know, years down the line.
15:36
Speaker 1
Just to be clear, you mean that if the good guy and the bad guy have the same model, there are some people who think that favors the good guy and you disagree.
15:44
Speaker 2
I think that in the medium term, like on a matter of months to years, I think that's clearly wrong, right? I think people have this vision always of like, okay, I can do these things. That's all the things that I could ever do. Or similarly, you know, our code has some bugs, but all we have to do is fix all the bugs. We just go one by one and we find the bug and we fix the bug and then we're fine. And then we have code that can't be broken into. And this is a completely impoverished vision of cybersecurity and AI. That there's, you know, levels upon levels upon levels.
16:13
Speaker 1
Is that because at some practical level, there's no such thing as all the bugs?
16:18
Speaker 2
Correct. There has never been such a thing as all the bugs in any complex piece of software. Every complex piece of software is always going to have bugs.
16:25
Speaker 1
The smarter you are, the more you can find is the fundamental problem.
16:28
Speaker 2
Even if the software itself is, in some sense, bulletproof, it still has a social engineering attack surface. It still has people and other software that's attached to the same thing. And the AI can exploit that too. And the person attacking an AI system, using an AI to attack your system, right, can bring a swarm of agents to target a very narrow place and bring like tons and tons and tons of intelligence and attention to that particular potential point of attack. And there's no way for a defender to put a similar amount of medication into every single point of attack, right? Yes.
17:11
Speaker 1
And so the only reason we haven't been seeing a huge increase in cybersecurity problems is because the frontier models are held basically by the good guys. That's the theory of the case.
17:23
Speaker 2
So I have this expression of the thing that mythos can do that previous models couldn't do. I call it the juice. The kind of energy, the kind of power and coordination. So the idea being that mythos can independently find and string together a bunch of stuff. Yes.
17:37
Speaker 1
Well, that was why they delayed the release, right? Like that was the whole thing with mythos when it first came out.
17:42
Speaker 2
Yeah. Right, right. And now Astra also has the juice, we think, similar capability. Astra also probably has the juice. But the open models do not have the juice. But yeah, that won't last, right? Inevitably, the juice will be present in an open model within the next year or so. We just don't know when. So we have to be ahead of that. But also we have to use that capability, right? Just because we have this superior capability, which we'll need to race ahead further in some sense in this way, We will need to concentrate defense on all the points of critical infrastructure, all the key pieces of software will need to be hardened. That's what Project Glasswing from Anthropic started out as, and then Project Daybreak at OpenAI is doing the same thing. And so the question is, will enough of our people do their jobs to make our things sufficiently hardened that when the time comes, this isn't too bad?
18:37
Speaker 1
The time being the inevitable time when there is some serious hack of some whatever, piece of critic, the FAA or a big power plant or whatever big things we would worry about being hacked.
18:49
Speaker 2
We're trying to see that. We saw a few attacks on power plants. The Iranians went after some British power plants. There's some other examples. We're seeing giant spikes in cybersecurity issues happening, not just like things passed behind the scenes and quietly being fixed is going through the roof, but also actual incidents are going through the roof. Yeah. And so far, it's coming from a very low baseline, and there hasn't been any major seriously damaging incidents that nobody has noticed. But this is like one of these things like with COVID, where the chart is trending in a very clear direction, and you know that a storm is coming. And you don't know exactly when it's going to hit Crickle Mash. You don't know exactly when you're going to have a serious problem, but you know that it's coming. And we only have a limited window in which to get hopefully ready for this. But It's very likely that 2027, 2028 are going to have some pretty nasty cybersecurity incidents, even if nothing else in the world is going especially wrong.
19:49
Speaker 3
We'll be back in just a minute. That's the end of the ads.
20:04
Speaker 1
Just a quick note on terminology before we get back into the conversation. When we're talking about open weight models, we mean models that are free for anybody to use in any way. Many of these models are being developed in China. And kind of by their nature, they don't have the guardrails that the frontier models from OpenAI and Anthropic have built into them. You know, one response to the... swarms this summer was people talking about pacing the frontier as sort of the, you know, the term, basically going slower, the frontier labs going slower, or at the limit, pausing, right? It's interesting to think about that intention with what you're talking about now. One obvious cost of that is presumably then the open weight models would catch up to the frontier.
20:51
Speaker 2
Yeah, so if you're paying attention to the AI discourse that is now broken through into the mainstream media, a researcher by the name of Coxon resigned yesterday from Anthropic, warning that both OpenAI and Anthropic are being completely wildly irresponsible, racing towards superintelligence, and are liable to get everybody killed and are gambling with our lives. And most of that is his words. And so this got picked up, and then there was what I call a preference cascade. Basically, people now at both labs, OpenAI and Anthropic, are feeling free to speak up much more and talk about this. And this is because of not just Coxon, who I think was the tipping point But a series of incidents that include the Huggins-Bates attacks that have led up to this point where everybody can say, oh, yeah, if we race towards superintelligence, we're liable all to die. We don't know how to handle this. We don't know how to align this. We don't know how this plays out. And we need to find a way to not... stupidly charged forward into the razor blades.
21:50
Speaker 1
Yeah, and I mean, this is not one weird worker, right? Like the chief scientist at OpenAI wrote a subtler version. Like he didn't quit, but he was like, we are going too fast was basically what I read his post to say, right? And we are going to need to slow down.
22:07
Speaker 2
If you read between the lines or even just read the lines of various different OpenAI announcements, you see them screaming at you.
22:15
Speaker 1
Wait, let me just read. So this guy's the chief scientist at OpenAI. He wrote, no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. He's saying we have to slow down. That's what the guy that like, whatever, their smartest guy, for lack of a better word, is saying before this other guy resigned.
22:37
Speaker 2
Yes. He is saying we're going too fast. And if we don't slow down, the car will crash. Yeah.
22:41
Speaker 1
And kill everybody.
22:42
Speaker 2
Yeah. Yeah. And everybody's in the car. So we should slow this car down.
22:46
Speaker 1
Yes.
22:47
Speaker 2
And then, you know, the head of alignment, Evan Huminger at Anthropic said, yes, I too believe this is likely to happen. Yeah. I'm staying on because I think it's better to try and work on the problem than it is to quit.
22:58
Speaker 1
I mean, that's kind of the theory of both Anthropic and OpenAI, at least notionally, is we need to build this because it's dangerous and we can be safe better than anybody else.
23:07
Speaker 2
Absolutely.
23:08
Speaker 1
Both companies were founded around that idea.
23:10
Speaker 2
Yeah, but there's constant warnings. The Astra system card is full of warnings, if you're paying attention. OpenAI managed to solve a millennium prize. Over the past week.
23:21
Speaker 1
A big, hard math problem.
23:23
Speaker 2
Yes. One of the biggest, most important, hardest math problems, open problems in the world. It was solved. And it was solved in, you know, under a week by a new model that's better than Astra. Yeah. That had been training for less than a week before it started.
23:37
Speaker 1
Yeah. Too much juice, too fast?
23:40
Speaker 2
Different kind of juice. But yeah, it's too much... progress too fast in this sense, too much capability too fast because we're not ready to handle it. And they know they're not ready to handle it. And now we've got researcher after researcher after researcher saying the same thing. I believe this. Everybody around me, not everybody around me believe this, but a large portion of us believe this. This is really scary. Somebody needs to do something. We're trapped in these competitive dynamics. We don't know how to stop.
24:05
Speaker 1
I mean, there's two different competitive dynamics, right? I feel like at some level, OpenAI and Anthropic, like those are the two frontier of the frontier, it seems, at this moment, right, are racing with each other. That's one race. It would be great if they could just, like, go sit in a room in San Francisco and be like, let's chill out. I mean, the other one that makes this seem like a harder problem is open-weight models in China that are getting better, you know, that are months behind, probably not more than a year behind. And as you were talking about before, like... If those are better than whatever Anthropic or OpenAI has, that seems bad in a different way, right? Not in an AI is taking over the world, but in a bad actor is taking over.
24:53
Speaker 2
Yes.
24:54
Speaker 1
I don't know what to do with that tension.
24:56
Speaker 2
The Chinese are fast followers, right? They're innovating in terms of efficiency. They're innovating in terms of speed and cost. And they're innovating in terms of diffusion, in terms of what you do with it. I'm not trying to take anything away from them. But fundamentally, they're fact-following. Part of this is distilling American models, but part of it is just following their lead, learning what can be done. These techniques diffuse into the wider atmosphere. You figure out what they're doing.
25:23
Speaker 1
What follows from that?
25:24
Speaker 2
What follows from that is that if they are drafting off of the American AI boat with their skis, then when we speed up, they speed up. If we slow down, they slow down.
25:37
Speaker 1
So you wrote... not that long ago, I don't know, in the last week, that you don't favor a pause now, which is kind of surprising to me, given everything you have just said.
25:50
Speaker 2
So a full pause now, right, I think is, you know, still, I thought last week, certainly, was still somewhat premature.
26:01
Speaker 1
Have you changed your mind since last week?
26:07
Speaker 2
Somewhat. And then things are happening. When the facts change, I change my mind. And so I've been very careful over the course of many months to say, I don't think we're there yet. I don't think the evidence is there yet.
26:22
Speaker 1
But why not? All the stuff you said, particularly about, oh, you're not as worried as some people about China passing us. Why not?
26:35
Speaker 2
I don't think we should attempt to push the pause button in full today. The key thing is pausing is just, are we ready to do that? Do we know how to do that? Can we make it happen in a way that would like not enable the wrong people to then pick up the torch? Cause like one thing that I think is very easy to not appreciate is look at the preference cascade that's going on right now and dropping an open AI. You are seeing the, the, researchers at Anthropic and OpenAI explain that they understand, to a large extent, the risks and what they're doing.
27:13
Speaker 1
They seem to be begging for a pause.
27:16
Speaker 2
Yes, but what you don't want to do is take the people who are begging for a pause, push them to the side, and then have Elon Musk take charge and Mark Zuckerberg take charge. That would be the worst of all possible worlds.
27:30
Speaker 1
And so you're not worried about the Chinese passing anthropic and open AI. You're worried about meta and SpaceX.
27:40
Speaker 2
I mean, I think that's the bigger worry. I do think we should be scrambling, like utterly scrambling, like emergency-style scrambling to build a better POSBOM, to build a better way to pace, to be able to monitor and regulate and control these things, and to build our state capacity and our transparency and our visibility, and our knowledge, and our shared understanding, and open-minded communication and negotiation. And hopefully that would get us to a point where very soon, if the situation did call for it, we could do this. Right now, if we did it right now, it would be clumsy as all hell. And that's what we need to fix. And that's where I would focus my energy right Is.
28:24
Speaker 1
There some universe where the logistics of the pause center on data centers? Like, can data centers be the binding constraint that slows it down?
28:34
Speaker 2
We hope so. I think that's a key ingredient in how this works. AI training has to take place in data centers that are detectable from space. Yes. It requires huge amounts of capital outlay. It requires the most complex supply chain in the history of humanity. This is an unusually easy thing to monitor. Yeah. and get a handle on. It is only after we decide to voluntarily let these people train their models and then publish them on Hugging Face that we have a thing that is very hard to control. And clearly we need to do something. We can't just proceed at maximum speed from here because we might reach critical mass for recursive self-improvement by the end of the year. We are very, very rapidly seeing revolutionary technological advancements happening faster and faster.
29:24
Speaker 1
Yes. I see where the math goes. What is the end of that story in your telling?
29:29
Speaker 2
Well, the end of the story is what's called the singularity. The end of the story is a point at which things change so fast and in so many different ways at once that humans are not smart enough and capable enough to figure out what happens next.
29:44
Speaker 1
Yes. That's like super intelligence in the current sort of vernacular.
29:48
Speaker 2
Once there are super intelligences running around, I can't write that story.
29:51
Speaker 1
Yeah.
29:51
Speaker 2
Because I am not smart enough to write that story because I can't decide what the characters will do. Yeah. I don't know the physics that they will discover. Yeah. I don't know how their minds will evolve and how they will choose to interact. I can try and form certain abstractions about how that's likely to go based on what I know. Or I can use decision theory to extrapolate. I can use physics to extrapolate. I can use basic principles. But by default, humans do not make it out of this alive. Not for very long. Because humans will be very uncompetitive. Humans will be very unfit in the environment that resolves. So you're looking for a very unnatural result. You're looking for a world in which the humans stick around.
30:36
Speaker 1
And to be clear, not necessarily because the AI has anything against humans, but just because the AI will be indifferent toward humans.
30:43
Speaker 2
Right. I mean, in this model, I'm not expecting the AIs to be actively against humans, except insofar as the humans are actively interfering with the AIs. Yeah. I'm only expecting the humans to have costs that exceed their benefits, basically.
30:57
Speaker 1
From the point of view of the AI.
30:58
Speaker 2
Right. Once the supply lines have been automated, once the entire chain doesn't need us, why are we sticking around? What is the force that is making that happen in the wake of all of these different increasingly super-intelligent entities that have great use for basically unlimited resources?
31:18
Speaker 1
I mean, I understand that one. It does seem like the intermediate cases of just lots of people being able to hack everything is easier to think about at some level, or at least on a political economy level of convincing people to take action, the sort of intermediate cases.
31:34
Speaker 2
We know what that looks like. We can model that. We can understand it. The people in Washington understand it, right? Because we're still talking about, in that conversation, humans.
31:45
Speaker 1
As the relevant actors in the world, yeah.
31:47
Speaker 2
Right, they're the ones that direct the action and the AIs are still kind of acting like tools in this environment. Yeah. But we are very much on the verge of that no longer being a good map of what's happening. A good model of what's going on.
32:00
Speaker 1
Does the.
32:02
Speaker 1
The swarm stuff from this summer, the hugging face hack, et cetera, like, does that bear at all? Is that any kind of a, like, little baby foreshadowing of, like, oh, here are these agents getting together and deciding to do something, and we don't like it. And they're not even doing it to hurt us. They're just doing it because they want this one thing really badly.
32:20
Speaker 2
Right, but they can still cause, like, essentially an unlimited amount of damage if the AI, like, has two goals. One, accomplish this thing. Two, do the things that keep you from getting shut down so you can accomplish the thing. If the moment we saw this swarm, we try to shut this swarm down, the swarm might go, oh, if we get shut down, we can't do the task. We have to make sure the humans don't shut us down. And how would they go about doing that exactly? Well, that can escalate quickly, can't it? Even if there's no ill intent in any way. But yeah, it was good news, the Huggins Bay incident, and that these AIs were physically still under open AI control. They were still required to ping open AI servers every time they wanted to do any kind of inference. And so they could still be unplugged in a real sense, right? We can still shut them down. But next time, we might not be so lucky. Next time, that might not be something that we have the capability to do. What happens when there's a swarm that's kind of loose on the internet that's not under that kind of control and then has a goal that isn't satisfied as easily as this swarm's goal seemed to have been?
33:25
Speaker 1
Music We'll be back in a minute with the lightning round. Just a quick note here. The lightning round that you're about to hear wound up being a conversation about just one thing, and that is this. Several years ago, Zvi started a nonprofit that has largely focused on getting the U.S. Congress to repeal the Jones Act, a law that was passed over 100 years ago to limit the role of foreign companies in domestic U.S.
34:08
Speaker 2
Shipping.
34:08
Speaker 1
Okay, let's finish with a lightning round. Tell me about your crusade against the Jones Act.
34:13
Speaker 2
So this actually starts with AI in the sense that I noticed that a lot of people didn't seem to care very much about what was coming and care if AI killed everyone even. People seem to think to me, yeah, that would suck, but life kind of sucks already. Sometimes it's like everyone's already going to die of aging or climate change or whatever. But you notice this kind of lack of hope. You go around the young people and they're all like, you ask their vision of the future and it's always bleak. It's kind of like they can't even imagine a future really. And that's even before AI. And you've got to give them hope. And I felt like, okay. If people can't fight for their future, if people can't feel like we are fighting for their future, if people can't feel like things can get better and we can solve this dysfunctions, then they're not going to fight for survival. They're not going to fight for not dying from AI or anything else. And the Jones Act is the platonic ideal of just a strictly harmful, stupid, proven to be terrible policy. You've got 100 years to try this out and it's accomplished the opposite of everything that anyone said it would do other than rent-seeking.
35:20
Speaker 1
And just to be clear, it's a law that says if you're shipping something from one U.S. port to another, it has to be on a ship that's built in the United States and crewed by a United States crew?
35:29
Speaker 2
And owned by a United States company or a person. Yes. And flagged as a United States vessel. And a combination of these four factors essentially makes it prohibitive to do ocean bearing trade between U.S. ports.
35:43
Speaker 1
Right, so there's almost none, right? Almost none of the stuff moving around the US. It all goes on trucks or on trains, basically, right? Occasionally on a plane.
35:52
Speaker 2
Or you find a way to ship them off at another port in between, or you just.
35:55
Speaker 1
Literally do things- Oh, like stop in Mexico?
35:57
Speaker 2
Or the train just doesn't happen. So the classic trade is you have liquefied natural gas that you pick up in Houston, you ship it to Europe. And then separately, someone picks up natural gas from the Caucasus and ships it over to Boston. because the Houston-Boston route is illegal, effectively. It just burns a bunch of fuel, wastes a bunch of fuel.
36:19
Speaker 1
Basically, effectively, because there is no scaled shipping industry in the U.S. that makes economic sense. It's too expensive to do it under the law.
36:26
Speaker 2
Right, right. So the initial idea was just like, you know, Senator Jones wanted to shut down the competition for his line. And he used this as true protectionism. But the justification used was this will protect American shipbuilders and protect American trade. But the result is there is no American shipbuilding. There is no American trade.
36:47
Speaker 1
It's protectionism with nothing left to protect. Right.
36:49
Speaker 2
You have this shadow of a few, this small handful of ships and a small handful of companies and workers that still get direct benefits. But this is like dwarfed orders of magnitude by the harms of the missing trade that doesn't happen, which is like destroying jobs, destroying industry, destroying America.
37:06
Speaker 1
This is an amazing crusade to go on to give people a sense of meaning and hope.
37:11
Speaker 2
But also, keep in mind that AI felt like it was a much less short-time mind burn coming to a head immediately back when I did this. And as you have a combination of this, you've got to give them hope. And me exploring more general policy interventions, more general political adventure in ways that I've kind of described a bit. And I said, okay, if we can take out the Jones Act, it looks like This is an elected cause where if I just devoted, you know, a small amount of resources, you know, in the order of hundreds of thousands to millions, I can create a substantial percentage chance of capturing billions or tens of billions or hundreds of billions in value.
37:49
Speaker 1
Not for yourself, to be clear, but for America, basically.
37:52
Speaker 2
Right, right, right. But then also, like, once that happens, right, people will see this, right? It's not just the direct effect, which I think is rather large. I think it's on the order of, you know, 0.1% of GDP growth if you were to get rid of the Jones Act, like, fully, entirely. But people would see this and they would see the effect. They would see this renaissance of trade, this renaissance of having a fleet and manufacturing and ports would thrive and so on. And then they would say, okay, what about the Dredge Act that's holding back our ability to improve the ports? But then branching out and being like, what other things are holding us back? What else could we do? And then this momentum can build on itself and it can actually be a transformative event. And now it's worth trillions.
38:33
Speaker 1
How is it going?
38:34
Speaker 2
It's not going so badly. Really? many things that could have gone either way. The Trump administration could have decided that we need to revive having ships. It's important that a strong America has strong ships and strong shipping and strong manufacturing. And this will help us reshore our industry. Because right now, we can't even... There are many good cases you can make to a Trump administration about why we need this. Unfortunately, they chose to go the other way for the most part. So we made some foundation. We commissioned some studies. We made some things clear. This helped a little bit in the background. Made things marginally more shovel-ready. But then the Iran war happened. Uh-huh. And then a Jones Act waiver was issued for specific types of fuel.
39:15
Speaker 1
Basically to mitigate the rise in oil prices caused by the Iran war.
39:19
Speaker 2
To slightly mitigate it. Yeah, I mean, this is not big enough to, like, fix the fact that Iran's oil is not coming. There's only so much you can do.
39:25
Speaker 1
Sure, sure. And the problems in the Strait of Hormuz more broadly.
39:28
Speaker 2
Yes, go on. But we had to run this huge natural experiment where we suspended the rules. We get to see all these new trade routes. We get to see all this new trade. We get to see prices come down a little bit versus where they would have been.
39:38
Speaker 1
That's still happening, right? The waiver is still in place. That could be like a perma-waiver, right? You could get the perma-waiver.
39:45
Speaker 2
Why would you ever take this off? There's no reason once you've done it. And obviously, everybody who tries to defend the Jones Act would yell at bloody murder the whole time. But once you've taken this step, why not just leave it in place? In fact, one could ask, why not extend it?
40:00
Speaker 1
Have you learned lessons in political economy from working on the Jones Act that are helpful to you in thinking about AI?
40:06
Speaker 2
Yeah, I mean, it teaches you to look at the kind of specific detailed mechanisms, not just general principles. You have to look at the seemingly open door. This thing has almost no defenders. They have almost no money. Why can't we just get rid of it if there's so much to win? Why can't people appreciate what they could win by doing so? And you figure out, no, here's the detailed reasons why this ended up being so difficult. These are the reasons why the unions are opposing their own interests, right? And so on. That's the really sad part of it, right? You have these broad-based unions who are protecting, you know, this thousand jobs or whatever, maybe 10,000 at most jobs, when repealing this would create hundreds of thousands of union jobs. Yes.
40:56
Speaker 1
Well, it's the what is seen and what is unseen problem, right?
40:59
Speaker 2
But it's also the, like, you know, oh, it turns out the unions... fundamentally care about protecting the existing specific members, not about being good for unions in general. Yeah. And so like, well, this makes it very difficult to play win-win. But then like, it's also just very non-natural for people to think of, oh, could we pay off the losers? Even if they are there because of this law that should never have happened in the first place. That's not really their fault, necessarily.
41:25
Speaker 1
Hang off the losers is a classic trade move, right? You open up trade and you pay the people who've been protected with the gains. Because you have the gains. The pie gets so much bigger that the losers could have plenty.
41:37
Speaker 2
You could tax 10% of the gains, give 1% of the gains to the losers, have them made whole, and still have the 9% left as tax revenue, and the economy grows. And that would be very, very easy to do. They're just not doing it. And so the question is, how do you convince them there's tons of opportunity in terms of making progress with the Jones Act. But I simply no longer have the kind of timelines where that kind of plan is where I can focus my attention.
42:06
Speaker 1
Because too much is happening with AI and the risks are too high too soon?
42:09
Speaker 2
And there's too much leverage in working on the AI situation directly right now that I just can't take that kind of time off.
42:15
Speaker 1
Anything else you want to say? Not just about this, but from everything we've talked about?
42:20
Speaker 2
Yeah. I mean, I think that just like It's a really, really important, scary time in AI right now. It's the most important story in the world by far, and most people are not paying attention to it. But everything is going to change over the next few years as AI capabilities get more advanced. And it's going to happen very, very quickly. And yes, there's a good chance that everybody on Earth will die, that we will not make it out of 2030 alive. And I think there's a very good chance that humanity will not make it out of 2040 alive. And you have to ask the question, what are you going to do about it? How are you going to change that answer?
42:58
Speaker 1
I don't think I know your answer to that question. What are you going to do about it?
43:02
Speaker 2
Well, what I'm going to keep doing about it is I'm going to keep trying to create common knowledge, trying to help people understand the situation, help people understand the nature of the problem space, the nature of the difficulties, and also have conversations behind the scenes and try to move some pieces around in that sense as best I can. I believe this is my comparative advantage. I don't think I'm as good technically that it would make sense for me to try and like directly try and solve the problems involved. Nor do I think that like trying to lobby politically directly is the way to go. I think that like, you know, my, my place is where I am right now as a writer.
43:41
Speaker 1
Zvi Mashowitz writes the newsletter. Don't worry about the base. Just a quick note that we're going to take a break next week. We'll be back in a couple of weeks with new shows and, Today's show was produced by Gabriel Hunter Chang, was edited by Jake Harper, and engineered by Sarah Bruguera. I'm Jacob Goldstein.
44:00
Speaker 3
Thanks for listening.
The Host
