May 26, 2020
The 5×5—Is it a game or is it real? Simulations and wargaming in cyber
In the summer of 2002, the United States Armed Forces conducted the Millennium Challenge (MC02), a three-week exercise simulating a military confrontation between “Blue” forces, the United States, and “Red” forces, a fictitious formidable Persian Gulf state. Consisting of computer simulations and live exercises, the joint, 13,000-person, $250 million wargame’s objective was to put the United States military’s “transformation”––a shift toward more network-centric operations––to the test. Aside from its size and cost, the wargame was notable in how it revealed organizational and tactical deficiencies of US forces. Retired Lieutenant General Paul Van Riper, having played the part of a “Red” commander, told the Guardian afterward that US forces were ill-prepared for battle. Riper’s forces were so successful in fact, that they forced a reset of the game under new and more Blue force friendly rules.
While not all come close to the scale of MC02, wargames are a mechanism for states to understand how their adversaries might operate, and––perhaps equally as important––how they themselves coordinate, organize, and respond in crisis. Cyber simulations can be helpful, not just militarily, but also in performing tests that could be devastating if not impossible to replicate in the real world. Take the Defense Advanced Research Projects Agency’s “black start” exercise on Plum Island, New York for example, where engineers worked to stave off attacks on a makeshift electric utility, without damaging the actual electrical grid.
In the private sector, wargaming can help leaders understand the kinds of damage that could be inflicted on their organizations in the event of a cyberattack, and lead them to channel resources and investment towards more effective mitigations and capabilities. Greater insight into risk and response allow public and private sector organizations to better prepare for crises before they happen and rerun history to stave off defeat in future. Wargames can be complex live events or low-cost simulations. They can even be the basis for major reforms to policy and doctrine, giving us much to understand about them. Shall we play a game?
Cyber Statecraft Initiative experts go 5×5 to discuss the keys to effective wargaming.
#1 What areas or sectors appear to be doing cyber wargaming effectively? What lessons can be learned from them?
Maria-Kristina Hayden, global head of cyber wargames & awareness, The Bank of New York Mellon: “The original wargamers in the United States are the military branches, which conduct large-scale, realistic games simulating wartime conflicts. In recent years, military experts have spread the wargaming concept to other industries as well; in particular, critical infrastructure industries such as energy and finance. Lessons learned are innumerable, but the most important is: practicing response to realistic scenarios is the single best way to increase resiliency in the face of rapidly changing threats.”
Andreas Haggman, cyber security skills policy lead, UK Department for Digital, Culture, Media, and Sport: “From (unclassified) information I came across during my PhD research, the defense and national security sectors seem to be near the forefront of cyber wargaming. This is both unsurprising and frustrating. Unsurprising because those sectors have historically been competent wargame users, but frustrating because a lot of it is done behind closed doors and therefore limits how much the rest of us can learn from them.”
Nina Kollars, nonresident fellow, Cyber Statecraft Initiative; associate professor of the Strategic and Operational Research Department and core faculty member, Cyber and Innovation Policy Institute, Naval War College: “Everyone is feeling around in the dark when it comes to cyber wargaming effects. I like the folks who are doing work on critical infrastructure like Dragos. They’ve taught us that the problem isn’t so much about super elite zero days and military classified data. Instead, the problem is a lack of understanding of dependencies and the effects of those dependencies. Computers, communications systems, and computer networks are linked together, but not equally so, and not in ways we understand. If I toppled this network over here, will it affect all your military assets? One or two systems? Systems I hadn’t anticipated? Dependencies models are hard to build but they are very important for cyber.”
Jacquelyn Schneider, Hoover fellow, Hoover Institution, Stanford University; nonresident fellow, Cyber and Innovation Policy Institute, Naval War College: “Perhaps I’m biased, but I think some of the best work being done to use wargames to understand cyber problems is at the Naval War College’s Cyber and Innovation Policy Institute. We led a series of games on cyber and critical infrastructure over the past few years that were really important to thinking about how DoD should support the private sector (or not). Those games informed the DoD cyber strategy and then helped hone some of the concepts that were introduced in the strategy.
“I’m also seeing a fantastic resurgence of wargaming within academia. My team at the Naval War College and now at Hoover has been running large-N, multiple iteration experimental games on cyber operations and nuclear stability. We just completed the live series with about 480 participants and we are about to run the first virtual game on May 27th. There is also great research involving games and cyber coming from Brandon Valeriano and Ben Jensen at the Marine Corps University.”
John Watts, senior fellow, Scowcroft Center for Strategy and Security: “Cyber wargaming seems to be split between two groups at opposite ends of the spectrum: those doing hands-on, tactical exercises, TTXs, and wargames—cyber capture the flag, etc.—and high-level cyber policy games. Finding a way to game the middle ground is difficult on any topic, but cyber is particularly hard due to the level of technicality. There may be some people doing this effectively, but I haven’t seen much. Despite the difficulty, it’s a really important goal and one that more effort needs to be put into trying to achieve.”
#2 What is the greatest misconception about wargaming?
Hayden: “There are three: that wargames are only applicable to the military, that they must be highly technical, and that they must simulate large-scale war. The concept of wargaming is also extremely effective in “tabletop” and corporate environments—allowing various cross sections of corporate teams the opportunity to learn about current threats, solve realistic problems, and strengthen muscle memory of response.”
Haggman: “That you need to be a gamer to do wargaming.”
Kollars: “That wargame outcomes can ‘prove’ anything. Wargames provide insights that point us to new gaps in our thinking about how a situation might play out. It terrifies me when people say “we learned x or y from the outcome of the wargame.” Wargames are useful in stepping through the dynamics of human decision making––whether that is intended to teach a leader, or to put a spotlight on blind spots or gaps in our planning that deserve more analysis.”
Schneider: “One of the greatest misconceptions about wargaming is that they are about war. While historically “wargames” have been about military campaigns, the use of games to understand human behaviors and future outcomes goes far beyond war. They are used extensively to think about disaster preparedness, crisis management, and even business strategies.”
Watts: “I think people get hung up on the name—that it is about war/conflict and that it is just a game. I’ve seen many, many people over the years try to re-brand it in different ways (Analytical Games, Peace-Games, etc.), but very few have done so effectively. That may be too obvious an answer though. The biggest misconception in executing a game comes from those designing it who worry it won’t generate the outcomes they want, or that it won’t be substantive enough. It’s always important to consider and analyze such things in design, but people need to trust the process too. The whole point is you don’t know what people will do or what the outcomes are. Trust those you’ve selected to play the role and see what happens. Even if it’s a “failure”—it can usually tell you something new.”
#3 How real is too real in wargaming? What level of realism produces good outcomes?
Hayden: “The right level of realism is key to ensuring an engaged audience and a meaningful event with lasting takeaways. Too real is when a wargame element is perceived as a real-world threat and causes panic or disruption outside of the game.”
Haggman: “If the level of realism raises game complexity to the point that the game becomes inaccessible. To paraphrase Phil Sabin: a simple game that is played will always be more instructive than a complex game that is not played.”
Kollars: “From an active learning perspective, the more it aligns to reality the deeper the impression it will make on the players. There’s a range of different kinds of games out there. For some reason everyone tries to elevate the analytical/discovery elements of wargaming as being the most valuable. I couldn’t disagree more. I need warfighting leadership to practice working together, under time constraints with imperfect data, making unpleasant choices.”
Schneider: “The balance between abstraction and reality is one of the big challenges of gaming. The key to managing that balance is a clear research question and deliberate design going into the game. A strategic level game—for instance the cyber and critical infrastructure games we ran at the Naval War College—doesn’t need a lot of detailed technical knowledge about how cyber exploits occur but it does need realistic strategic considerations. In contrast, a more tactical or operational game that is designed to game through the efficacy of cyber operations on networks (like some of the GRIDEX kind of games) needs less realism about strategic policy and greater attention on technical realism.”
Watts: “I don’t believe in “too real.” Every war-game has an element of abstraction, otherwise it would just be real life. The appropriate level of abstraction depends on your specific goals and objectives, and that varies for every game. Every war-game is also inherently a series of trade-offs. People should fret about that, they should accept it, be intellectually honest about it and then seek to design the game that best achieves their objectives.”
#4 What is the future of wargaming? Is there a next big thing that will change how organizations wargame?
Hayden: “Wargaming will only increase in popularity as public and private groups continue to see the value in these simulations. Cyber wargaming in particular, in which a cyber disruption is part of the story, will continue to lead the way.”
Haggman: “I can see mixed-technology games becoming a thing—especially given that opportunities to gather people around a table may be limited for some time yet! An organization that can realize the social benefits of manual gaming with digital solutions will be on to something.”
Kollars: “Rethinking how we get effective collaboration in wargames to include classified games under conditions of social distancing. We have got to figure out how to ensure games can move forward without bunching bodies together and without sacrificing on the quality of the interaction. To be clear, even though we are being forced into this, making these changes could also be a tremendous asset revealing new insights in combined and joint games. Depending on the objective of the wargame, playing asynchronously in a distributed manner, could be more realistic.”
Schneider: “I’m really excited about the proliferation of wargames beyond the DoD, particular research games like the Signal game, led by the Project on Nuclear Gaming at Berkeley. Signal has been very innovative about their use of iteration and virtual gaming interfaces. They are able to be experimental in design, sample, and medium because they are not beholden to “sponsors” who are looking for particular outcomes (a difficulty in DoD games). That’s why I’m also really excited about the emerging consortium of researchers interested in marrying social science research methods with wargaming; there is great wargaming methodology work that is coming out from Erik Lin-Greenberg, Reid Pauly, Andrew Reddie, Ellie Bartels, and Stacie Pettyjohn among others.”
Watts: “There has been a reticence among a lot of traditional ‘wargamers’ to embrace software applications. Many believe you have to have experience designing physical games for them to be valuable. There is definitely a benefit to that, but that view also misunderstands and under values the process of designing a software-based game. The flipside is that a software-based game isn’t inherently valuable just because it’s the new hotness or has virtual reality or some other gimmick. The process of problem identification and game design is still as important as ever. Finding the middle point between those two views, and calibrating approaches so that they embrace software-based applications but retain the fundamental characteristics and design principles of traditional wargaming is the next big frontier. A large benefit of wargaming is its experiential insight. Software based games amplify that. So, we’re inevitably heading towards it, but I’m guessing there will be a lot of missteps before we see its true value.”
#5 How vulnerable are wargames to our own biases and preconceived notions? How can these be overcome?
Hayden: “Wargame designers and players alike can be guilty of bias and failures of imagination. To overcome this, 1) wargames should be designed with a flexible story path that includes a variety of potential outcomes (across a wide plausibility/likelihood scale), and 2) facilitators should foster a culture of openness and creativity in which players feel comfortable testing the boundaries of their preconceived notions. If not during a game, when?”
Haggman: “It might be argued that these are not vulnerabilities to overcome, but that wargames should purposefully expose and challenge our biases and preconceived notions. In my personal experience, a player who comes out of a game saying “I can’t believe how wrong I was about X” is a successful outcome.”
Kollars: “In all honesty, I’m more concerned about wargames being misused because people don’t understand the limitations of the method. If I have a pet war concept that I want to validate, I can throw together a scenario, call it a wargame, and go through the motions. Again, that’s not what they are for, but that won’t stop people from doing it. It comes down to transparency about the objective, the process, and the final report. Full release and complete transparency of all the data captured during the game can help as long as there is a practitioner community that can review it and provide peer review…but with wargames that are complex multi-day affairs with 100+ players, not to mention levels of classification? Good luck with that.”
Schneider: “Wargames are extraordinarily salient––that’s why they are great learning tools. However, because they are so influential in shaping how we think, a poorly designed game (or one that is designed to get a certain outcome), can be extremely dangerous for proliferating bias. This is why lessons from social science—about generalizability, substantive bias, valid treatments, etc.—can be so helpful when designing games.”
Watts: “Very. Every game is an abstraction, and the way the game designed abstracts the real-world is a manifestation of the way they understand and view the world. There are ways to mitigate individual bias, and any good war-gamer should strive to do that. But every war-game requires trade-offs and they’ll never be 100% objective. That’s why we need more diversity and different perspectives. And every game-designer needs to own those biases and be intellectually honest about the effect on the outcomes. That’s the best you can do in my opinion, and as long as you don’t attempt to portray the results as anything other than that, I am comfortable with it.”
Simon Handler is the assistant director of the Atlantic Council’s Cyber Statecraft Initiative under the Scowcroft Center for Strategy and Security, focused on the nexus of geopolitics and national security with cyberspace. He is a former special assistant in the United States Senate. Follow him on Twitter @SimonPHandler.
The Atlantic Council’s Cyber Statecraft Initiative, under the Digital Forensic Research Lab (DFRLab), works at the nexus of geopolitics and cybersecurity to craft strategies to help shape the conduct of statecraft and to better inform and secure users of technology.
Further reading
Image: Banner Image: "Malware" - Visualizing malware as a landmine with an unsuspecting user about to step on it in their bare feet, symbolizing the danger of moving through cyberspace unprotected. Source: Mai Arollado/CyberVisuals