The Library Problem
What do the British Airways hack of 2018, the Twilio hacks of 2020 and 2021 and probably the worst of all, the SolarWinds breach of 2020 have in common? In all cases, bad actors inserted malicious code into production systems in order to access user data or compromise systems in other ways.
This type of attack is called a supply chain attack (also known as a third-party attack, value-chain attack or backdoor breach) and it is a growing trend as a lucrative attack vector. According to a recent report by Argon Security, an Israeli cybersecurity firm specializing in protecting the software supply chain, attacks grew by over 300 percent in 2021 versus 2020. What makes it a huge problem is the growing complexity of modern software and the enormous reliance placed on third party libraries to provide core functionality, or at least, supporting elements of it. Many applications make use of tens or even hundreds of third party libraries, and it’s clearly not possible for engineers to check every single one for malicious code. Remember, the libraries themselves are often constantly being updated and new versions released.
To clarify, we are talking about two distinct problems. First, there are vulnerabilities that could be exploited by a knowledgeable bad actor, which makes up the largest component. But the smaller, potentially more effective (and growing) component amounts to deliberate code changes made into libraries to be exploited.
So how do we manage this?
The good news is that there exists a central clearing house of known vulnerabilities called Common Vulnerabilities and Exposures (CVEs). This service catalogues hundreds of thousands of software vulnerabilities; although, by nature, it cannot be all-encompassing and other vulnerabilities may exist. It’s also worth noting that, in the case where a problem is found, for myriad reasons, many packages are not fixed in a timely manner.
A good example is a flaw that was found in the gnu c library in 2015. A domain name lookup function contained a buffer overflow vulnerability that could have caused a system crash or allowed attackers to execute malicious code (CVE-2015–7547). This vulnerability was introduced in 2008 and not fixed until Feb 2016! This is a flaw that was inherent in Linux, which powers an enormous number of cloud and enterprise servers and a vast majority of internet “plumbing”.
CVE launched as an open source effort at the end of the nineties. It was joined in the mid-noughties by a US govt. effort called NVD, CVE feeds data to NVD, providing additional detail around the problem that could prove useful in some situations. But, even with these resources, it is clearly not possible for any software engineering team to keep track of all the notifications, library updates etc. And remember, this is all supposed to happen in a world of agile development and frequent — sometimes multiple — daily build and release cycles.
An alternative solution is the OWASP Dependency-Check. It is a free, mature project, regularly updated and an excellent tool for gathering and reporting a list of dependencies with CVEs. Currently, all the major languages are covered (Ruby, C++, Python and JavaScript) and it can be run from the command line but it can be made even more useful using a key technology that enables frequent build, test and release: Continues Integration/Continuous Deployment (CI/CD).
It’s essential you are automating your build and test process with some form of CI/CD — not just on demand, but regularly - to truly take advantage of this in your delivery process. Here’s why: Assume your engineers are working on an important feature with the deadline fast approaching. The check identifies a vulnerability, but updating the vulnerable library has a risk of introducing a regression, compatibility issue and/or possibly a slew of other bugs. The only way to mitigate this is an automated test and daily build process, which gives the team time to react appropriately. One point to bear in mind, though, is that vulnerabilities can have different severity levels and impact. Some may have no impact for a project, so it’s worth noting that the dependency checker can report false-positives.
Common build tools, such as Maven and Jenkins, can be integrated with plugins. But if, like most modern tech companies, your infrastructure exists in the cloud, it is also possible to use the OWASP service as part of your build process, demonstrated for GCP nicely here or in a more complex reference example for AWS here. There are other options if you use a source control platform such as GitHub, which has a dedicated security page regarding supply chain attacks. It also provides tools such as Dependabot, which can scan your codebase and provide notifications through a number of channels, including Slack, directly to your Engineering team.
The decision to implement such processes needs to be measured against the actual risk of attack. It’s possible that some of your products or services have no access to sensitive customer data (e.g. credit card information). In these cases, it may not make sense, as the value has to be perceived to be high for a determined hacker to invest the time needed to be successful. However, it’s worth including this as part of any risk assessment that you should be already doing for your company.
The take-away from this is that you cannot just hire a security company to do a penetration test once a year and assume all is well. You need to embed security in your engineering culture and take advantage of state-of-the-art tooling to cover as many attack vectors as possible. Never rest on your laurels — the landscape is always changing, and remember, hubris always wins.
