May 12, 2022
The 5×5—Reflections on trusting trust: Securing software supply chains
Nearly every bit of technology and infrastructure that enables modern society to function runs on software. Lines of code—in some cases millions of them—underpin systems ranging from smart electronic kettles to fifth-generation fighter jets. A significant portion of software is cobbled together with and dependent on many pieces of open-source, non-proprietary code that, by and large, is built and maintained by volunteer software engineers for whom security may not be a top priority. As such, vulnerabilities abound in widely used systems, and securing the entirety of the increasingly complex software supply chain is no easy feat.
Notable examples like the Sunburst/SolarWinds cyber-espionage campaign shed light on how adversaries are increasingly exploiting vulnerabilities in the software supply chain to compromise critically important public- and private-sector systems, and yet software supply chain security remains an underdeveloped aspect of public policy with meaningful gains starting to be made only more recently. On May 12, 2021, US President Joe Biden signed Executive Order (EO) 14028 on Improving the Nation’s Cybersecurity, a portion of which addressed the need to enhance software supply chain security. On May 5, 2022, in accordance with the EO, the National Institute of Standards and Technology (NIST) released its updated Cybersecurity Guidance for Supply Chain Risk Management.
Between 2010 and 2021, according to the Atlantic Council’s Breaking Trust dataset, at least forty-two attacks or vulnerability disclosures involved open-source projects and repositories. Public policy still has significant room to address how both government and software developers improve the security of software supply chains, especially the wider health of the open-source ecosystem.
We brought together five experts with a range of perspectives to discuss the implications of insecure software supply chains and realistic paths to securing them.
#1 How does the security of software supply chains impact national security?
Jennifer Fernick SVP & global head of research, NCC Group; member of the Governing Board, Open Source Security Foundation (OpenSSF):
“Software is the core infrastructure of almost every aspect of contemporary life, and a security vulnerability in any aspect of a system or network. Its effect on government, intelligence, defense operations, financial transaction networks, energy companies, telecommunications, food and pharmaceutical supply chains, and other public- or private-sector critical infrastructure can have devastating consequences in the physical world. The now-popular notion of securing the ‘software supply chain’ ultimately reflects a growing apprehension that the security risks of computing systems are many layers deeper, more invisible, and more interdependent and impactful than they initially seemed.”
Amélie Koran, senior fellow, Cyber Statecraft Initiative; director of external technology partnerships, Electronic Arts, Inc.:
“Security does not always mean reliability. It is an outcome in most cases, such as resiliency is, but for most, the typical CIA triad of confidentiality, integrity, and availability drives most assessments of basic system and software security. While we can develop a very secure national security system—from defense to utility and other infrastructure—the additional security components sometimes hurt reliability or resiliency, as they must perform a set of steps or lack inclusion of typical convenience features that non-critical or national security systems tend to avoid, as they may just be bad security choices.
In this case, often the selection of components for ensuring a secure system, including the construction, building and testing, as well as operation of those utilized for national security purposes become very oblique and even harder to manage. For many years those building these systems eschewed open-source software due to the fear of unexpected or unpredictable inclusion in the code base would make systems less secure. Over time, however, they have realized that, in most cases, the transparency of such software sources results in more rapid fixes, and thus it closes the windows of opportunity for potential attackers versus closed-source or even more bespoke systems that do not go under as much scrutiny. In other words, some of the most trusted national security solutions may actually be the most insecure from iterative tests and test of rigor by others. A test is only as good as the test developer, and once you cut those creators and testers down, less issues tend to get caught and resolved.”
John Speed Meyers, security data scientist, Chainguard:
“It is not too much of a stretch to say that the functioning of most digital systems—including those of western militaries, governments, and societies—have become deeply reliant on a hard-to-understand and hard-to-secure software supply chain. It is like we built a digital Manhattan on a foundation of quicksand and swamps.”
Wendy Nather, senior fellow, Cyber Statecraft Initiative; head of advisory CISOs, Cisco:
“No matter how they are compromised or by whom, software supply chains have an outsized network effect on the security and stability of everything from utilities to emergency response, transportation, healthcare, aviation, and public safety. As everything digitally transforms, the attack surface grows in subtle, remotely accessible ways, and it potentially affects even those populations without access to technology.”
Stewart Scott, assistant director, Cyber Statecraft Initiative:
“Software is eating the world, or so I am told, so securing any system or application relevant to national security is critical—everything from computer systems on fighter jets to government-adjacent email accounts. Securing one’s own systems is challenging enough, but modern software services are a patchwork of in-house programs, purchased products, imported libraries, cloud applications, open-source components, and even copy-and-pasted code. Security for software supply chains extends the challenge to using others’ code securely, identifying what products are developed and maintained securely, and even figuring out what dependencies exist. It is incredibly complicated and difficult for security everywhere, and national security especially, as incidents like the Sunburst campaign and log4j illustrate.”
#2 What are the challenges to building more secure software supply chains? Do developers, intermediaries, consumers know what they need to do?
Fernick: “Vulnerabilities are cheap to create but expensive to find—even the best programmers in the world regularly write code with security vulnerabilities, and even the best security tools on earth will fail to find all of these vulnerabilities without the time-intensive intervention of human experts. Yet, even theoretically perfectly secure code would more likely than not depend on another piece of software that is full of exploitable vulnerabilities, will run on an insecure operating system on top of flawed hardware, and be deployed through a build pipeline that can be compromised by attackers. Security is very hard, and yet I feel like as an industry we push too much of this responsibility downstream to other developers and users who are not equipped to face it. Instead, we need to ‘shift left’ and assume that vulnerabilities are present, but find ways of reducing them at scale or detecting and remediating them early, through improved programming languages and frameworks, scalable vulnerability-finding tools, and other systematic investments in improving the ecosystem as a whole.”
Koran: “Complexity is by far one of the greatest challenges to securing software and digital supply chains. Most tools and systems that are available to organizations provide a patchwork of awareness of the overall risk, and require a high level of competence in the minutia of the software or system in order to generate a plan of action, short of a basic “update this code with something deemed more secure.” One of the major issues in all of this also is consumer based. In most cases, it takes extra effort to track back the health of code or source components that may be utilized to build systems. These may be stitched together, but could be, once together, increasing the insecurity or risk of operations in certain configurations.
Put it this way, while the suggestion of a software bill of materials (SBOM) may tell you what is in the box, it does not tell you if the ingredients are good for you. It is only one portion of a chain of decision-support processes that are necessary to build safer and more secure software supply chains. Imagine standing in the cereal aisle at a grocery store, and while you can look at the ingredients between a toasted whole wheat cereal and “sugar bombs,” what appetite are you satisfying chasing one over the other? Technically, the base components of the grains and such within in them may be very close to one another, but if there is more of one bad item (e.g., preservatives or sugar), while it solves the same task of giving you a breakfast, the satisfaction on consumption may be different. The same goes for software development. Pick a quick and dirty “all-in-one” suite that solves problems quickly, but may be opaque as to what is in it and how it was built, or make an artisan selection of bespoke code—you will have a lot of potential work ahead from choosing a turn-key opportunity provided by the former over the latter.”
Meyers: “There are many. If you are reading this article on a computer of some sort, ask yourself why you trust the chain of software that allows you to read this article. If you have no answer, try to find a little solace in the knowledge that most experts do not have a great answer either. And nobody really knows what they need to do, although software supply chain integrity frameworks like SLSA (Supply chain Levels for Software Artifacts, pronounced “salsa”) are a good start.”
Nather: “Software is organic and dynamic; it changes faster than humans can follow, as it is the result of human contributions on a worldwide scale. Software challenges cannot be compared directly to a hardware supply chain or a manufacturing line because of these additional complexities. Only with specific expertise can developers and intermediaries know what they need to do, and many developers do not come from the traditional educational pipelines any more. The answer is NOT to rely on consumers to have this expertise to make their own market-driven choices; it is to ensure that security is baked into the software standards, protocols, tools, and automation at scale.”
Scott: “Writ large, the challenge is identifying and managing an incredible number of rapidly changing relationships that range everywhere from import lines to massive government acquisition contracts. There is a huge range of understanding about and capability to address software supply chain security. If developers, maintainers, vendors, CIOs, and consumers aren’t all on the same page about how to improve supply chain security—let alone provided with sufficient resources to get things moving—there’s going to be progress in some places and awkward lapses elsewhere. For example, GitHub is moving towards universal multi-factor authentication, which will help secure massive amounts of open-source components, but many entities will not even know the degree to which they are relying on and/or contributing to that code, especially at higher organizational levels.”
#3 How would you describe the state of public-private sector collaboration on securing software supply chains? Compared to where it could be?
Fernick: “I am optimistic and encouraged by the proactive and collaborative engagement between senior US government officials and the Open-Source Security Foundation (OpenSSF), a cross-industry effort to improve the security of the open-source ecosystem, which I helped to establish with colleagues across the industry in early 2020. The January 2022 White House meeting on Software Security brought together a powerful alliance of public- and private-sector organizations, including OpenSSF, to discuss initiatives to: (1) prevent open-source software security vulnerabilities, (2) improve coordinated vulnerability disclosure, and (3) reduce vulnerability remediation times. In mid-May, we will be returning to Washington, DC with a bold mobilization plan for exactly that; several carefully-defined initiatives that will together help radically improve the security of the world’s most critical open-source software.”
Koran: “As much as somebody would want to rehash “I’m from the government, I’m here to help” as an opening line, developers and other technicians tend not to like bureaucracy meddling in their creations and orbits, especially if it means more under-resourced work for them to perform. While compliance is also not the best carrot to achieve results either, a hybrid model of standards that can be modeled and accepted as a method to align, much like NIST’s Special Publication series, which both public and private sectors use to get to a reasonable level of assurance. Possibly a similar Commerce Department/NIST-driven guidance that understands scale and complexity, but also operating methods, such as making recommendations support DevOps that can be better integrated. Ensuring that input for this guidance incorporates not only broad industry input and support, but also addresses needs from small- and medium-sized businesses, as well as enterprises. Most regulations and guidance are selectively chosen due to the lack of timely, affordable and manageable actions to comply. In short, “keep it simple, stupid” (KISS) should be the name of the game to have better than average uptake, and note that all of this should also be iterated upon.”
Meyers: “Nascent, especially when it comes to the security of the open-source software supply chain. Anecdotally, when I worked at In-Q-Tel, a strategic investor for the US intelligence community, many intelligence community staff used to look at me cross-eyed when my colleagues and I would suggest that they should devote their time and resources to open-source software supply chain security. Log4j and the recent White House meeting on open-source software security suggests, however, that the times are changing.”
Nather: “There have been excellent ad hoc responses to specific events, but we need to create a more repeatable process that includes not just the ‘biggest and loudest’ private sector companies, but also the ones below the Security Poverty Line, which are equally likely to be victimized by software supply chain attacks. For example, the Blackbaud ransomware incident at last count affected over a thousand organizations, many of them nonprofits who provide critical services. Beyond response, we need to create a way for every organization to understand its supply chain risk. SBOMs are a start in this direction, but it is an after-the-fact report at this point, not a demonstration of secure software development practices. Make no mistake: this is not a chain; it is a vast web in which we are all somewhere in the middle.”
Scott: “There is building momentum in the federal government around software supply chains, and fora like the Cybersecurity and Infrastructure Security Agency’s (CISA) Joint Cyber Defense Collaborative are a good start at bringing industry to the table. Parts of the private sector, too, have recently started pouring a lot of resources into the issue, especially open source, but it is patchwork. Some companies are piling millions of dollars on to the issue, and some are not ready to take seriously how much it affects their security. It is also unfortunate that a lot of that momentum seems a response to recent incidents—I would love to see more proactive security collaboration capitalize on a well-intentioned reaction to compromise. In that vein, expanding the scope of existing public-private partnerships to include industry consumers outside of the usual IT vendors and, regarding open source, the nonprofits, maintainers, repositories, and package managers responsible for a lot of the actual code in question. Continued formalization of those ventures would be great too—supply chain and open-source security need to be ongoing discussions within among all stakeholders in the cyber policy world.”
More from the Cyber Statecraft Initiative:
#4 How can the United States and European Union most effectively contribute to the security of open-source software?
Fernick: “Coordination among major stakeholders is key, as open-source software is a vast, complex global ecosystem. Our success at securing the supply chain hinges upon having a singular place where representatives from government, the private sector, and open-source software projects alike come together to work on and make impact-prioritized, coordinated investments in things like security audits of critical software, improving vulnerability disclosure and remediation, and coordinating vendor-neutral emergency response teams to support open-source software maintainers in times of security crisis. Piecemeal initiatives and investments cannot comprehensively solve a lifecycle problem like securing the software supply chain—attackers will simply exploit the weakest remaining link.”
Koran: “Coordinate. There is a number of international organizations that have attempted to bite off various pieces of the open-source security, and software and digital product security, pie. Organizations may be looking at anything from vulnerability disclosure treatments to secure coding practices, but many of them attempt to reinvent their own wheel for their influence areas. Because of that, there is no single or coordinated voice or guidance as to what to do. The community, private sector, and individuals either choose to strike out on their own or pick and choose those pieces of guidance, frameworks, or regulations that may be best suited to them, or, in the worst case, may be the minimum bar in order for their work to comply. While asked how to “contribute”, it does not always mean having to directly provide technical contributions, like code, infrastructure or other bits and bobs; it can mean doing what governments do best, which is to get the right people talking to one another to share information at various levels. That is where they need to start—get on the same page and the right heads sharing knowledge and experiences.”
Meyers: “While I welcome transatlantic cooperation on open-source software security, each government probably needs to examine its own software supply chain security practices before there can be an open-source software Atlantic Charter.”
Nather: “Ensure alignment in the standards, practices, regulations, etc. that are being generated. Like everything else in cyber, open-source software development is cross-border, and world-wide coordination is required to ensure security is effective and aligned with the motivations of the open-source project contributors and maintainers. Tracking the dependencies in open-source software and identifying those ‘linchpin’ components that cross a certain threshold of impact would be a good start, as would a coordinated effort to fill resource gaps for software projects that are under-resourced or abandoned.”
Scott: “Governments can start by recognizing open source as infrastructure—it is everywhere, comprising 70 to 90 percent of many codebases, and it supports a huge part of the innovation and functionality in the digital economy. Ideally, that kind of framing will lead to regular, proactive investment from government alongside industry and help keep a transatlantic approach from getting bogged down by different approaches towards licensing and privacy. Open-source security is much more about how responsibly consumers are using (and tracking their use of) components, contributing to them, and supporting the ecosystem. An infrastructure approach should also help move away from the simple narrative that its maintainers are not getting paid enough—sometimes that is true, and sometimes maintainers have immense corporate support or even work for premiere IT companies. Usage, tooling, and self-knowledge are all hugely important and point to a much broader solution set than throwing money towards developers and hoping they can ‘fix everything.’
#5 What single proposal or idea to better secure software supply chains would you like to see Congress pass in the next year?
Fernick: “In the legislation that it passes, I would like to see Congress work to incentivize companies to work collaboratively with good-faith security researchers who choose to responsibly report security vulnerabilities that they find in software products to the affected vendors, without fear of retribution or legal risk on behalf of the researcher. Many companies still lack the maturity to see good-faith security vulnerability reports for what they are—a free gift, and an actionable opportunity to improve their own products to help protect their customers, and the world, from threat actors.”
Koran: “There needs to be focus. Most of the directives that have come since the recent change in administration have been merely addressing the federal government rather than the larger sphere. While that is an admirable focus, it is very top down and scoped very small. If Congress is to engage, it needs to be a wider and more comprehensive action where, possibly driven by some federal stewardship, the onus really lives within the community—among developers, private organizations, and individuals. This also should not be a “thou shalt” type of direction, but a way to structure guidance, oversight and engagement. This could possibly be either addressing which government agency or commission has the lead for certain areas, or by establishing something new to subsume a number of critical roles. This is not going to just to address a technical compliance issue, but also governance and sustainment by possibly providing grant-making capabilities, interfaces to the private sector, academia, and international partners. It will also need to be funded. Good ideas without a budget are just good ideas, not actions that can be relied upon for an outcome.”
Meyers: “The creation of an open-source software security center within the federal government, perhaps within the Department of Homeland Security, seems like a promising first step. This center could help assess and improve the security of the open-source software that the federal government and critical infrastructure relies upon and even contribute to the security of the overall open-source software ecosystem.”
Nather: “Securing the software supply chain goes beyond just creating secure software, as the Executive Order points out in calling for basic controls such as multi-factor authentication, monitoring and alerting to secure the development, distribution and production environments as well. The underlying problem is still assessing risk and impact, and prompting those conversations among suppliers and consumers. As this is all currently being piloted by the Biden administration, the next logical step would be for Congress to put it in a statutory framework to ensure effective oversight.”
Scott: “I would love to see Congress stand up offices in government dedicated to open-source security and sustainability. An official office in CISA, even a small one, would provide a clear place for industry and maintainers to turn to and interface with, and the Critical Technology Security Centers would be great outputs to channel grantmaking. Having that formal infrastructure in place would make it much easier to involve developers and maintainers in open-source policy discussions in which they have not been included much yet. It would also help put to rest any lingering misconceptions that open source is inherently less secure than proprietary code or that open source is something that should—or even could—be avoided. Digital infrastructure everywhere depends in large part on open source, so the challenge is not securing or fixing that community or ecosystem—it is figuring out proactive, leveraged investments that government and industry can regularly make in its security.”
Simon Handler is a fellow at the Atlantic Council’s Cyber Statecraft Initiative within the Scowcroft Center for Strategy and Security. He is also the editor-in-chief of The 5×5, a series on trends and themes in cyber policy. Follow him on Twitter @SimonPHandler.
The Atlantic Council’s Cyber Statecraft Initiative, under the Digital Forensic Research Lab (DFRLab), works at the nexus of geopolitics and cybersecurity to craft strategies to help shape the conduct of statecraft and to better inform and secure users of technology.
Image: Image by Elchinator from Pixabay