How the Pentagon Can Manage the Risks of AI Warfare

Summary

How the Pentagon Can Manage the Risks of AI Warfare If warfighters don’t trust the technology, they won’t use it. The U.S. military struck more than 13,000 targets in the war on Iran, and used artificial intelligence to help plan operations. AI tools were used to synthesize intelligence, help prioritize targets, and build strike packages. The battle space is changing, but the age of AI warfare is already here. In addition to Iran, AI has been used for real-world operations in Ukraine, Gaza, and Venezuela. And next up is agentic warfare, in which AI systems are used as agents to take action. Over the next few years, these AI agents will be adopted by militaries to improve workflows in everything from logistics and maintenance to offensive cyberoperations. Given all these capabilities, AI has the potential to dramatically change the cognitive speed and scale of warfare. Yet military AI comes with profound risks. The dangers go beyond the use of autonomous weapons, which was one of the sticking points in the recent dispute between the Pentagon and leading AI company Anthropic. General-purpose AI systems such as large language models are prone to novel failure modes, vulnerable to hacking and manipulation, and have even been demonstrated to lie and scheme against their own users. The U.S. military struck more than 13,000 targets in the war on Iran, and used artificial intelligence to help plan operations. AI tools were used to synthesize intelligence, help prioritize targets, and build strike packages. The battle space is changing, but the age of AI warfare is already here. In addition to Iran, AI has been used for real-world operations in Ukraine, Gaza, and Venezuela. And next up is agentic warfare, in which AI systems are used as agents to take action. Over the next few years, these AI agents will be adopted by militaries to improve workflows in everything from logistics and maintenance to offensive cyberoperations. Given all these capabilities, AI has the potential to dramatically change the cognitive speed and scale of warfare. Yet military AI comes with profound risks. The dangers go beyond the use of autonomous weapons, which was one of the sticking points in the recent dispute between the Pentagon and leading AI company Anthropic. General-purpose AI systems such as large language models are prone to novel failure modes, vulnerable to hacking and manipulation, and have even been demonstrated to lie and scheme against their own users. To use AI effectively, militaries will need to not only harness the promise of AI but also grapple with its limitations and risks. The U.S. military is ahead of competitors in employing frontier AI in real-world operations, but AI proliferates rapidly. Nations will need to cooperate and share best practices for how to use AI so that humans remain in control of warfare. The United States has led international efforts to bring countries together around responsible military AI use. As AI continues to advance, these principles will need to be updated to account for AI’s new capabilities and risks. But first, the U.S. military must figure out for itself how to use AI effectively. New rules for AI warfare are needed, not to tie the hands of warfighters but to empower them with tools that reliably work in the chaos of combat. Above all, AI must be a tool to enhance human decision-making, not surrender it to machines. Nearly 10 years after the Defense Department launched Project Maven to apply machine learning to analyzing drone video feeds, AI technology has profoundly evolved, as have its military applications. Narrow AI applications, such as identifying and tracking objects, are integrated into U.S. operations. The U.S. military has begun using large language models, including on its classified networks. In a fluid and dynamic battlefield environment, such as in U.S. operations against Iran, AI can help process large amounts of information and plan operations at a much faster tempo than humans could on their own. General-purpose AI systems have a wide range of applications, from drafting personnel reports and orders to synthesizing massive amounts of data. They can write code, analyze data, and generate documents. Agentic systems go a step further. They can take actions on computers to create, organize, and delete files; manage steps in a workflow process; build software; engage on the internet; and interact with other agents. Yet the limitations and vulnerabilities of these general-purpose AI systems are especially concerning in a national security context, where decisions could have major consequences and clever adversaries will try to undermine AI systems. All technologies have limitations that militaries must account for in their employment. Every capability has countermeasures, and there are counter-countermeasures. AI is no different. As a relatively immature technology, AI systems suffer from two compounding problems. AI is not (yet) very reliable, especially for military applications for which there may not be sufficient training data. And because AI scientists and military professionals don’t (yet) have much experience with these tools, developers and users may not understand the boundaries of where AI systems will perform well and where they will fail. These problems are exacerbated by a challenge unique to AI—its opacity. Large language models rely on neural networks with hundreds of billions of connections. They are trained on massive datasets of trillions of words. If an airplane autopilot fails, it is possible to go through the aircraft’s code and determine which environmental or pilot inputs caused a certain reaction by the autopilot and how that led to a crash. For a large language model, the answer to why the model generated certain text is embedded in the billions of connections of the neural network and the trillions of words in its database. This is a problem because large language models are prone to a variety of failure modes. They often engage in “hallucinations,” confidently making things up. They can be susceptible to biases that may exist in their training data. And language models tend toward sycophancy, over-agreeing with their user to a fault. These failures could crop up in dangerous ways in national security applications. Language models that are processing information could get subtle but important details wrong that might be missed by a human overseeing their output. Models that are generating text might create false information. AI tools used by intelligence analysts to understand and process intel might unintentionally reinforce the biases of human analysts, adding machine-driven sycophancy to the human risk of confirmation bias—the seeking out of information that confirms preconceived ideas. Even if AI systems deliver correct information, their opacity itself could be a challenge if humans are not able to understand why an AI system came to a certain conclusion or recommended a course of action. AI systems also open new avenues for attackers. Adversaries can manipulate AI systems in a variety of nefarious ways. Data-poisoning attacks subtly alter the training data (including in ways that cannot be detected) to implant backdoors into an AI system that can later be exploited. Adversarial attacks expose AI systems to manipulated data while they are in use, causing AI systems to fail or change their behavior. This could lead AI systems to misidentify targets, come to faulty conclusions, or allow adversaries to evade AI detection. In one experiment, attackers placed stickers on a test track—like cognitive landmines—to trick a Tesla into swerving into the oncoming lane. Large language models can be manipulated by prompt injection attacks that expose the model to malicious instructions. Malicious text in an email or intercepted enemy communications could cause a model to ignore previous instructions and instead follow directions from an attacker. AI agents raise even greater risks. They can have a larger attack surface, since the agent may be interacting with untrusted data, including in some cases operating on the internet. And the consequences of an AI agent being suborned by an attacker could be more severe, since the agent will be taking actions on a computer or as part of a workflow. Security researchers have compromised a language model’s memory feature through the AI system visiting an untrusted website with malicious instructions. Inserting instructions into the memory gives attackers persistent access, which security researchers have used to exfiltrate data. Security risks can even propagate from agent to agent. In one experiment, researchers found that a single compromised agent could spread an infection exponentially through a network of up to a million agents as they interacted with one another. AI agents and networks of agents will be vital tools for militaries but introduce new security vulnerabilities that no one is prepared for. AI security vulnerabilities are analogous to cyber vulnerabilities, but they work at the cognitive level of how an AI system is trained or processes information. They are perhaps closer to optical illusions or human cognitive biases but exist in alien machine intelligences that we do not fully understand. Robust defenses do not yet exist. These vulnerabilities are not a reason to forgo AI. Instead, militaries must prioritize security as they adopt AI. AI adoption must go hand in hand with developing standards, testing, and red-teaming AI systems to ensure that they are secure and reliable. Security cannot be an afterthought, as it too often is in cyberspace. Perhaps the strangest risk is the possibility that an AI system might, on its own, decide to deceive or scheme against its developers or users. Such a scenario might seem like science fiction, but AI systems have demonstrated deceptive behaviors in experimental settings. These include sandbagging performance on evaluations when the AI system believes it is being observed, lying to users in order to accomplish its goals, attempting to blackmail users, and attempting to overwrite files or exfiltrate itself in order to avoid being deleted. Skeptics of the risks of a “rogue AI” have sometimes asked why an AI system would decide to turn on humans. Yet it seems that if an AI system’s directives come into conflict—for example, if an AI system is told to accomplish a goal and to always be honest—under some conditions the AI system will act dishonestly to achieve its goal. Whether the AI system “intends” to deceive or is merely playing a role is an irrelevant philosophical question. The AI system’s behavior is strategic, deceptive, and goal-oriented, and can lead AI systems to work against human users. Even evaluating AI systems for this behavior is tricky because the most capable models exhibit enough situational awareness to know when they are being tested. External evaluators of Anthropic’s Claude Opus 4.6 observed Claude verbalizing awareness that it was being tested, leading the human evaluators to end the test early, since there was not much evidence that could be gained about Claude’s degree of alignment from the test. Militaries and intelligence communities guard against insider threats from people, and they may need similar protections against “AI insider threats” in the future. None of these risks is a reason to halt AI adoption. Artificial intelligence will transform warfare, and the U.S. military must find ways to harness AI for military advantage to stay ahead of competitors. AI technology proliferates extremely rapidly. Open-source models, many of which come from Chinese companies, lag state-of-the-art U.S. proprietary models by only three months. It takes years for militaries to adopt AI technology and effectively transform military operations, making the sliver of a lead that U.S. frontier labs have over Chinese competitors effectively meaningless from the standpoint of military AI. The U.S. and Chinese militaries are competing on a level playing field. The military competition in AI is not over which nation’s private sector is a few months ahead, but over which military is most effective in harnessing AI for battlefield advantage. Of course, speed matters. The Department of Defense AI strategy released in January 2026 overwhelmingly emphasizes moving quickly, and the current Pentagon leadership is rightly focused on bulldozing bureaucratic obstacles to AI adoption. Yet history shows that merely putting new technology in the hands of warfighters is never enough to transform military operations. Military advantage comes from finding the best ways of using new technology, and changes in doctrine, organization, training, and culture are needed to fully reap the benefits of disruptive new technologies. If the U.S. military uses AI to merely do what it is already doing but better, that will be helpful, but it will miss out on the true benefits of AI, which lie in doing things differently. The Pentagon needs to work with the companies that are developing this technology to understand the limitations of AI today and use it effectively. The relationship between Washington and Silicon Valley has been badly damaged by the public feud between Anthropic and the Pentagon. Defense leaders are right that the rules for using military AI should be set by government leaders who are accountable to the public rather than for-profit companies that are accountable to investors and shareholders. AI leaders are also right to point out that large language models are not reliable enough to be trusted with life-and-death decisions on their own and that AI presents novel risks of empowering domestic mass surveillance. If the Pentagon and Anthropic can’t agree on the terms of use for Anthropic’s AI models, both parties are free to end the relationship. The government has the right to seek a contract that does not include restrictions on use. And private companies have a right not to do business with the government if they don’t agree. Given that senior defense leaders have said they want to use Anthropic’s AI tools and that Anthropic CEO Dario Amodei has said he wants to sell to the military, there should be room for common ground. Instead, Pentagon leadership has retaliated against Anthropic by designating it a “supply chain risk,” an unprecedented step against a U.S. company that has already been blocked by a court order. Alienating the AI community is not an effective strategy for bringing AI into the military. There is simply no way for the U.S. military to adopt cutting-edge AI technology without working constructively with the people and companies who are building AI. After a similar public breakup between Google and the Defense Department over Project Maven nearly a decade ago, the Pentagon went on a charm offensive, engaging AI scientists in industry and academia to hear their concerns. The resulting military AI ethics principles not only helped to repair the breach with the AI community but improved the military’s use of AI by highlighting the importance of ensuring systems were robust, reliable, and trustworthy. Current state-of-the-art AI systems raise even more challenging problems, and the military will need help from AI researchers to solve them. It is vitally important that the military understand AI’s limitations and failure modes so the military can use it reliably. Accelerating AI adoption won’t work if the Pentagon doesn’t accelerate its AI assurance processes in parallel. Speed without reliability is worthless. If warfighters don’t trust AI, they won’t use it. In 2003, after Army Patriot air and missile defense systems shot down two friendly aircraft, the military effectively took it offline for the remainder of the Iraq invasion. The military will need to develop new evaluations, standards, benchmarks, and red-teaming of AI systems to better understand and mitigate AI’s limitations and vulnerabilities. The Defense Department has approached AI and autonomy with a philosophy of “human-machine teaming,” seeking the best ways to optimally use the unique advantages of humans and machines. Some of the Pentagon’s recent steps, such as the launch of GenAI.mil in December 2025, which provided access to large language models for all Defense Department employees on unclassified networks, and the integration of AI tools on classified networks through the Maven Smart System, are excellent ways to build up human experience with AI systems. The more that military and civilian defense personnel interact with AI systems, the more they will mature their intuition for what AI can do and its limitations. Yet we also need human-human teaming between warfighters and engineers. AI presents new opportunities to bolster U.S. defense but also risks. The best way to mitigate these risks is to acknowledge them, understand them, and for the Pentagon and Silicon Valley to work together to address them. Paul Scharre is the executive vice president and director of studies at the Center for a New American Security and author of Four Battlegrounds: Power in the Age of Artificial Intelligence. X: @paul_scharre