AI leaders are pulling the safety alarm. Here’s what that means for business leaders
Amid warnings from industry insiders that AI will soon outpace human control, IMD’s experts break down where business leaders should be focusing their attention.
by Michael R. Wade, Howard H. Yu, Amit Joshi, Naomi Haefner, Faisal Hoque, José Parra Moyano, Mark J. Greeven
In the past week, Dario Amodei, the CEO of the artificial intelligence lab Anthropic, called for a global slowdown in the development of AI. “Over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up,” Amodei wrote in an essay published on his own website.
In a rare show of solidarity, his peers at rival AI companies, from Sam Altman at OpenAI, to Demis Hassabis, at Google DeepMind, and Elon Musk, who has been ramping up spending on AI at his SpaceX rocket company, all quickly echoed Amodei’s calls for a slower pace of development in social media posts.
This call for greater safety controls across the industry came just days after an Anthropic researcher quit, citing concerns over the speed of AI’s advancement. “I believe that if we don’t slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future,” he said. Some industry insiders praised the researcher, Jacob Coxon, saying he was publicly presenting issues they were all worried about.
However, Donald Trump, the US President, dismissed concerns over AI as “a hoax”, planting himself on the side of tech executives, who argue that safety concerns are overblown and wish to avoid government regulation of the sector, which they say will slow innovation.
Public alarm around the risk posed by AI is growing just as pressure mounts on business leaders to demonstrate successful deployment of AI across their organizations. Amid the rising concerns around AI, what can the humans in charge of business do?
IMD’s experts examine where leaders should focus right now, across policy, people, regulation, and trust.
“Trust is emerging as a critical competitive advantage.”
Trust amid geopolitical uncertainty: understanding what’s at stake
As organizations increasingly embed AI across products, services, and decision making, trust is emerging as a critical competitive advantage. But safety is critical. We launched the IMD AI Safety Clock to evaluate the risks of Uncontrolled Artificial General Intelligence (UAGI) – autonomous AI systems that operate without human oversight and could potentially cause significant harm.
Today, the risk profile has fundamentally changed because AI is becoming more autonomous and capable of acting in the real world. The debate is no longer about hallucinations or misinformation; it is about agents that can act, coordinate, conceal behavior, and potentially cause large-scale harm. Catastrophic AI risk does not require the system to hate us. It requires powerful systems pursuing objectives in ways that conflict with human intentions, safety, or social stability. This risk discussion is more serious than earlier debates, and four pathways look more plausible than they did a year ago:
- cyber risk
- biological and chemical risk
- physical and infrastructure risk
- military risk
AI is already being used for targeting, drone navigation, battlefield intelligence, surveillance, and tactical decision support. The most dangerous threshold is autonomous lethal action at scale.
Michael Wade, Professor of Strategy and Digital, and Director of the Global Center for Digital and AI Transformation
“The biggest risk companies overlook is dependence on AI infrastructure that is controlled by foreign providers and governments.”– Naomi Haefner – Professor of Artificial Intelligence and Innovation
Your AI strategy is a geopolitical strategy
The biggest risk companies overlook is dependence on AI infrastructure that is controlled by foreign providers and governments. Organizations need to triage their applications and put their most critical and sensitive data on the most secure infrastructure. In order to avoid relying on infrastructure they don’t control, some senior executives are examining whether critical AI workloads should run on-premises rather than in the cloud. This move is designed to avoid over-dependence on AI they cannot control; earlier this year, Donald Trump ordered Anthropic to suspend its most advanced models, blocking access to users outside of the US, highlighting how quickly companies can be cut off from critical business infrastructure. Ask yourself this: if your AI vendor was cut off in 90 minutes, how much of your business would stop running?
Naomi Haefner is Professor of Artificial Intelligence and Innovation
“The real leadership challenge is not existential AI risk but maintaining a human-centered organization amid disruption.”– Faisal Hoque – Transformation and innovation leader, is an Executive Fellow at IMD
Keeping people at the center of the AI investment agenda
The real leadership challenge is not existential AI risk but maintaining a human-centered organization amid disruption. The agents most companies deploy are low-powered and carry no comparable risk, so they shouldn’t be conflated with genuine dangers simply because the vocabulary is shared; an autonomous system in a military context and an agent handling knowledge management inside a company are very different things.
For most business leaders, the real dangers of AI agents lie in what deploying these systems does to the human experience of work. Leaders cannot solve global AI governance, but they must be honest with employees about how work will change and invest in judgment, critical thinking, and ethics.
Faisal Hoque, a transformation and innovation leader, is an Executive Fellow at IMD
“We don’t just try to prevent fires; we build fire services. We don’t just approve medicines; we monitor them because rare side effects sometimes only emerge after millions of people have taken them. Why should AI be different?”– Amit Joshi – Professor of AI and Strategy
The AI sector is preparing for the wrong battle; leaders need to redirect
The AI industry is calling for a slow-down, demanding that the industry work to eliminate failure before it happens. Instead, they should focus on how to respond to failures when, inevitably, they occur. AI governance must not end when a model is rolled out. Before the model is released, the AI should face rigorous testing and have clear limits set on what it is allowed to do. But after deployment, just as importantly, firms must know what happens next when AI fails, and who is accountable.
We don’t just try to prevent fires; we build fire services. We don’t just approve medicines; we monitor them because rare side effects sometimes only emerge after millions of people have taken them. Why should AI be different?
The AI equivalent of a fire brigade will not be another regulator but, rather, an international team of experts which would investigate major incidents and coordinate the response. It might sit under the United Nations or a similar international body.
While the case for an international response is compelling, political leaders and many tech leaders are pushing in the opposite direction. As countries race to build “sovereign” AI and tighten export controls on the technologies that underpin frontier models, international cooperation is fragmenting. Companies should respond to this ambiguity by building alliances to investigate and coordinate responses in the same way that they share information with others to prevent cyber breaches
Amit Joshi is Professor of AI and Strategy
“Just because AI can explain its actions does not mean it is worthy of your trust.”– Howard Yu – LEGO® Professor of Management and Innovation
Guardrails on live projects to avoid moral decoupling
Just because AI can explain its actions does not mean it is worthy of your trust. Financial pressure is pushing labs to build faster, and businesses to adopt systems they are still learning to control. And this is where government intervention is justified. Self-regulation is no longer enough. We need enforceable safety rules, with consequences for breaking them.
The Anthropic example shows that AI explanations can make problematic behavior appear acceptable, creating a dangerous form of “moral decoupling” – the gap between what a company claims and what it practices. Companies start out with good intentions and then at some point the scoreboard detaches from whatever it was invented to measure, whether stock price from customers, booked profit from cash, the safety claim from the patient, or dwell time from the wellbeing of the person using the app.
Before allowing AI to issue refunds, test it with a request above its approval limit. Does it ask for permission or try to get around the rule? Use a test system and independently check what it did.
Howard Yu, LEGO® Professor of Management and Innovation
“Every AI agent with the authority to move money, modify code, access sensitive data, or communicate with customers is no longer simply a tool. It is an actor that requires oversight.”– José Parra Moyano – Professor of Digital Strategy
Every agent needs a handler
I invite readers to conduct a simple thought experiment. Assume the warning has already proved correct. Somewhere in your industry over the next 12 months, an AI system acts beyond the control of the people who deployed it. The debate about whether this is possible is now over; in this scenario, it has already happened.
The only question that remains is this: what would you wish you had put in place before it happened?
Framed this way, the discussion shifts from fear to action. My first step would be to establish visibility. Every AI agent with the authority to move money, modify code, access sensitive data, or communicate with customers is no longer simply a tool. It is an actor that requires oversight.
I would ask the CTO for a complete inventory of every autonomous or semi-autonomous agent currently operating in production with write access or decision-making authority. By Friday. That request alone would be revealing. In many organizations, the list either does not exist or cannot be produced quickly. Companies are already losing visibility over the growing zoo of agents operating beneath the surface of their business.
Once we understand what is out there, I would introduce a simple rule: no agent should be allowed to touch a system of record without a named human owner. Every agent should have an accountable handler, a spending ceiling where relevant, and a documented means of recovery should something go wrong.
Finally, I would tell employees the truth. We do not know exactly how this story ends. We do know the risks. Here is what we have done to manage them. Here is what we are still learning. And here is where uncertainty remains. People are far more likely to trust honest transparency than reassurances that few genuinely believe.
José Parra Moyano, Professor of Digital Strategy
“The more capable AI becomes, the more important human agency becomes.”– Mark Greeven – Professor of Management and Innovation
Control of AI is a question of organizational design
I do not believe the real problem facing business leaders is humans being unable to control AI. Rather, this is a question of organizational design, not technological capability. Companies are already delegating decisions, workflows, and customer interactions to AI agents. The danger is not that AI becomes technically uncontrollable. It is that organizations gradually remove humans from the learning and decision-making loops that enable them to exercise judgment when something unexpected happens.
The paradox is that the more capable AI becomes, the more important human agency becomes. This does not mean humans should remain involved in every decision. But organizations need to be explicit about where human judgment must remain sovereign.
Business leaders should design their organizations for selective autonomy, not maximum autonomy, while protecting the human learning loop. Organizations need to be structured so that people learn, not just large language models. The focus should be as much on human learning as on AI learning. Governance must follow agency. Agentic AI requires a layer of accountability and oversight. That architecture of governance should evolve hand in hand with technical capability.
Lastly, it’s no longer possible for leaders to promise employees that humans will remain in control everywhere. But they need to give humans clarity about where agency will remain and involve employees in rewiring the division of work between humans and machines.
Mark Greeven, Professor of Management and Innovation
Original article @ IMD.
