A plea to the Pentagon: Don’t sacrifice resilience on the altar of innovation

Americans love to invent, and this culture of innovation has driven US military acquisition. From Samuel Colt’s revolving pistol to Defense Advanced Research Projects Agency’s funding of Internet 1.0 to the Joint Improvised Explosive Device Defeat Organization’s rapidly fielded IED jammers, we excel at pushing the state of the art.

This mythos informs a narrative that what is valuable is The New—the upgrade to something bigger, badder, and sexier. Innovation has been such an effective prescription for economic growth and military dominance that we view it as the answer to whatever disease ails our national-security institutions. What the United States needs to reinvigorate its defense base, compete with China, and win the global economy must be more innovation.

Except the United States does not suffer from a lack of innovation; it suffers from a lack of resilience. All the defense, critical infrastructure, and commercial systems that have been compromised in the last ten years—and that are still being compromised, surveilled, ransomed, and taken offline—are not easy targets due to lack of innovation. They are vulnerable because they are brittle and unmaintained, and thus not resilient. Military systems and infrastructure, financial institutions, energy grids, healthcare providers, and public safety capabilities—the systems that keep modern society from descending into chaos—are fragile because they are not built to recover from attack.  

Resilience is what allows systems to take a hit and keep going. Resilient systems are well-maintained, fixed before they break with predictive maintenance, and progressively strengthened at weak points. They have the spare parts needed to keep flying or driving. They are continuously tested and red-teamed as part of a culture in which finding cracks is a badge of honor, not a threat to the organization.

The current incentives within the Department of Defense (DoD) do not reward resilience and in many cases undermine it. An acquisition culture that favors paying for hours of labor instead of performance benchmarks creates a perverse incentive to build systems that maximize labor to operate. Federal contractors’ revenue-maximizing strategy is to win bids on low-cost stand-ups that require lots of labor (the acquisition term of art is “a self-licking ice-cream cone”). The financial incentive is to attempt to replicate commercial capabilities (build versus buy), modify commercial capabilities so that they become labor-intensive to maintain, and avoid the automation necessary for security and effective maintenance. Program managers are afraid to update software because poorly engineered “spaghetti code” will break; when systems are patched together with short-term fixes that were never engineered to perform at scale, even security updates can cause problems that are difficult to diagnose and solve.

System owners in the public and private sectors need to think about building fast-healing, adaptable, durable systems, especially the “boring” ones that people only tend to notice when they break. Military e-mail is a good example. When e-mail is not reliable, people find workarounds. Whether the workarounds are less secure than badly maintained enterprise systems is an open question.

Stellar maintenance and logistics do not get people promoted, and thus are not prioritized among program managers who are more focused on delivering novel, “sexy” capabilities. Ambitious program managers and contractors rely on early-phase prototypes—or “minimum viable products” (MVPs), to borrow a term from Silicon Valley—to get over the line and justify more money to “scale” a capability. “Scale” is interpreted as marketing a capability to drive adoption by stakeholders and end-users who will get program money. Promises are made about how this effort is The Future and is going to be awesome. Software developers are running the show. Everyone is stoked. There is an institutional assumption that someone else will go back and harden the MVP, which isn’t a product and was never engineered to perform at scale. There is a naïve faith that someone more conscientious and less charismatic, perhaps, will go back and redesign the technology, processes, and procedures to make the sizzle into steak. Except it doesn’t happen, because by the time a qualified systems engineer views the horror show of taped-together, unmaintainable code, it’s too complex to fix and too politically enshrined to critique. When development displaces engineering, you get systems that require lots of bodies to maintain with custom wrenches, like a sports car that chronically breaks down. Such systems are unreliable and insecure—built to fail.

The more buzzword-laden technology that DoD acquires this way—artificial intelligence, machine learning, virtual and augmented reality, remote-controllable green grids, autonomous vehicles—the more the defense community’s attack surface expands. It’s possible that China is doing to the United States what the United States did to the Soviet Union in the 1980s: trumpeting huge strategic investments that an adversary drives itself into the ground trying to match. This time, the United States is losing the initiative by allowing an adversary to define US strategic objectives as the pursuit of innovation, which a nation-state cyber adversary can steal and compromise, rather than a ruthless focus on resilience, which makes theft and compromise more difficult.

DoD should make two major shifts in strategy to stop playing this losing game. First, it must measure, manage, and build resilience, and treat it as the first-order capability that it is. This will require changes in engineering methods, unprecedented transparency, the use of predictive maintenance, and advanced techniques for security and fail-overs. Resilience also requires adaptability, which makes it easier for programs to innovate at speed and scale than it would be if innovation per se were the primary objective. On the policy level, DoD should require software bills of materials and active maintenance (security service-level agreements with response times measured in days) as table stakes for acquisition. Such approaches are increasingly common in other industries. Leading institutions like the Mayo Clinic already have contractual terms and conditions to require active maintenance of all components for any software delivered by a vendor; so should the Pentagon.

Second, the US government must shift investment to entities that are incentivized to be resilient and subject to downside risk if they are not. The DoD needs multiple security solutions and stakeholders to check each other’s work—to keep “blessed” processes from becoming single points of failure. Intramural rivalries among the military services can drive resilience: They should red-team each other’s systems. Imagine the consequences for security posture the first time the Air Force gets owned by the Marines. These services should compete for end-users—and so should the Defense Information Systems Agency. If a program wants to migrate to a software factory with better service and reliability, that should be feasible in weeks.

Military acquisition programs love to standardize, but DoD needs to think about designing systems to accommodate the kind of heterogeneity that enables resilience. DoD should pay private-sector enterprises to deliver commercial capabilities with open architecture, in which one supplier’s product or service can be rapidly replaced by another’s with minimal lag time in contracting or integration. Yes, we need more than one cloud—this is critical infrastructure, not Highlander. DoD’s Joint Enterprise Defense Infrastructure award to Microsoft kept military cloud infrastructure from turning into a monocrop.

Defense acquisitions should be less like the Macy’s Thanksgiving Day Parade and more like NASCAR, where tire-changing speed and pit-crew coordination are recognized as key performance metrics. If the United States does not train and acquire with recovery in mind, it risks having core systems taken down without a shot fired.

JC Herz is co-founder and COO of Ion Channel and a fellow at the National Security Institute at George Mason University.

Further reading

Image: The Pentagon building is seen in Arlington, Virginia, on October 9, 2020. Photo via Carlos Barria/Reuters.