Digital demands have risen in the last few years, forcing software development teams to increase the frequency of their releases. Some companies, like Amazon and Netflix, report deploying code thousands of times per day. Continuous integration and deployment (CI/CD) often enables this rapid cadence.
CI/CD pipelines typically combine an assortment of technologies to help ship code from a local development environment into production. Yet, the CI/CD process itself hasn’t historically been treated with the same security forethought as typical application codes. As such, new CI/CD vulnerabilities have emerged, as seen in the recent Jenkins server attack, CircleCI breach, and other exploits tied to CI/CD-related dependencies.
Below, we’ll analyze the state of modern CI/CD security and review the most common risks. We’ll consider how to take a DevSecOps approach to mitigate CI/CD vulnerabilities and provide some general suggestions on safeguarding your continual release systems.
Understanding CI/CD and Its Vulnerabilities
Modern CI/CD now incorporates a diverse tech stack with countless integrations and vendor tools. These services work in tandem to help automate the delivery and deployment of new code.
A CI/CD pipeline usually includes certain stages such as compiling code and running unit tests, producing an artifact from the code and performing acceptance tests, and, sometimes, automatically deploying the app into production. But along the way, numerous blindspots could compromise the integrity of the final release.
“Cyber threats need to be on the radar for all CI/CD pipelines as risks continue to increase,” said George Symons, Vice President of Strategy for Cloud, Infrastructure, and Security, Persistent Systems. “Some of the most common vulnerabilities in CI/CD pipelines come from vulnerabilities in third-party library dependencies, insecure coding practices, and insufficient access controls.”
Which companies are the most important vendors in cybersecurity? Check out
the Acceleration Economy Cybersecurity
Top 10 Shortlist.
Overview of the OWASP CI/CD Top 10
So, what are these specific CI/CD vulnerabilities, and how can we spot them?
When assessing cybersecurity risk, it’s often helpful to turn to the security research group, Open Web Application Security Project (OWASP). Its OWASP Top 10 CI/CD Security Risks, which I summarize and explain below, gives an overview of some of the top CI/CD vulnerabilities:
1. Insufficient Flow Control Mechanisms: Refers to a lack of the necessary security vetting, which could allow a hacker to gain control over a system within the CI/CD process.
2. Inadequate Identity and Access Management: Overly-permissive states, stale accounts, external identities, and other access management concerns that may arise from challenges managing identity across many CI/CD tools.
3. Dependency Chain Abuse: Flaws in how software development systems work with dependencies, such as accidentally fetching a malicious package.
4. Poisoned Pipeline Execution (PPE): Compromised source control systems might allow attackers to issue malicious commands into the software build process.
5. Insufficient PBAC (Pipeline-Based Access Controls): Without the proper access controls for pipeline execution, attackers could abuse permissions and move laterally to affect other systems.
6. Insufficient Credential Hygiene: Insecure secret management could expose CI/CD credentials and give attackers the ability to interact with systems.
7. Insecure System Configuration: Poor system configurations or default settings may leave “low-hanging fruit” open for attackers to leverage.
8. Ungoverned Usage of 3rd Party Services: The ease of granting third-party services access to internal assets could result in risky shadow IT.
9. Improper Artifact Integrity Validation: Insufficient artifact validation could enable hackers to push compromised code to steps further down in the build process.
10. Insufficient Logging and Visibility: Without quality logging and visibility, security teams may have difficulty finding and resolving incidents or conducting post-mortems.
Methods to Mitigate CI/CD Risks
As you can see, CI/CD can be an intricate, interconnected process with many areas to exploit. It’s also a new concept, and implementors may not understand the inherent risks. Thus, to reduce the risks above, it’s important to first map your surface area, specifically your CI/CD environment and the tools in use, as well as centralize logging and monitoring to increase awareness
Next, be careful to review all dependencies utilized in the CI/CD pipeline — vet the introduction of new vendor tools, and carefully review the provenance of open-source packages to avoid name confusion and typo-squatting. “CI/CD pipelines often rely on third-party libraries and dependencies, which can contain vulnerabilities that could be exploited by attackers,” said Symons. To mitigate this risk, it’s essential to keep all dependencies up-to-date and use automated tools to scan for vulnerabilities in third-party code.”
OWASP recommends mapping identities across services and ensuring the rule of least privilege is followed for pipeline execution. This can help avoid unauthorized usage and broken access control. “It’s essential to implement access controls at every stage of the pipeline, such as enforcing least privilege access and using multi-factor authentication,” added Symons.
Lastly, leaders will want to encourage secure coding practices around CI/CD and consider tools that provide automated security testing. Some other tips for securing CI/CD include:
- Protect credentials used for CI/CD tools and do not expose them in public repositories.
- Include external verification or validation within the CI/CD process.
- Protect secrets carefully, time limit them, and continuously map CI/CD permissions.
- Protect CI configuration files and isolate unreviewed code whenever possible.
- Additional strategies include code signing, artifact verification, and configuration drift detection.
Securing CI/CD: Final Thoughts
CI/CD is now crucial for many organizations, furthering DevOps and helping maintain a rapid pace for software updates and feature releases. A quick release process streamlines the developer’s conversation with their system, helping improve developer productivity. Looking to the future, CI/CD will likely grow in sophistication and incorporate more automation and artificial intelligence (AI) into the software release process. And as the reliance on CI/CD accelerates, it will be necessary to audit the security of these powerful new automations and the privileges they grant.
“Overall, an organization needs to start with a security-first mindset,” said Symons. “The entire team should approach the development process with that mindset. This means thinking about security from the beginning of the development process and continuously assessing and mitigating risks throughout the lifecycle.”
By following the above ideas, organizations should have a better footing to begin addressing CI/CD security risks. For further insights, consider diving into each of the OWASP Top 10 risks linked above for additional case studies and risk mitigations.
Want more cybersecurity insights? Visit the Cybersecurity channel:
