What Is Poisoned Pipeline Execution (PPE)?
Poisoned pipeline execution (PPE), an OWASP CI/CD Security Risk, is an attack vector that abuses access permissions to a source code management (SCM) system with the intent of causing a CI pipeline to execute malicious commands. While PPE attackers don’t have access to the build environment, they have gained access to the SCM, which enables them to inject malicious code into the build pipeline configuration to manipulate the build process.
CICD-SEC-4: Poisoned Pipeline Execution Explained
Poisoned pipeline execution (PPE), listed as CICD-SEC-4 on the OWASP Top 10 CI/CD Security Risks, represents a sophisticated attack strategy targeting continuous integration and continuous deployment (CI/CD) systems.
Customers may have a variety of options available to them:
In the PPE strategy, attackers execute malicious code within the CI portion of the CI/CD pipeline, bypassing the need for direct access to the CI/CD system. The method involves manipulating permissions against a source code management (SCM) repository. By altering CI configuration files or other files on which the CI pipeline job depends, attackers inject malicious commands, effectively poisoning the CI pipeline and enabling unauthorized code execution.
Successful PPE attacks can enable a broad range of operations, all executed within the context of the pipeline's identity. Malicious operations can include accessing secrets available to the CI job, gaining access to external assets the job node has permissions to, shipping seemingly legitimate code and artifacts down the pipeline, and accessing additional hosts and assets in the job node's network or environment.
Given its critical impact, low detectability, and the existence of multiple exploitation techniques, the PPE attack poses a widespread threat. For security teams, engineers, and red teamers, understanding PPE and its countermeasures is critical to CI/CD security
Pipeline Execution Defined
In the context of continuous integration (CI), pipeline execution flow refers to the sequence of operations defined by the CI configuration file hosted in the repository the pipeline builds. This file outlines the order of executed jobs, as well as detailing build environment settings and conditions that affect the flow. When triggered, the pipeline job pulls the code from the chosen source (e.g., commit/branch) and executes the commands specified in the CI configuration file against that code.
Commands within the pipeline are invoked either directly by the CI configuration file or indirectly by a script, code test, or linter residing in a separate file referenced from the CI configuration file. CI configuration files typically have consistent names and formats, such as Jenkinsfile (Jenkins), .gitlab-ci.yml (GitLab), .circleci/config.yml (CircleCI), and the GitHub Actions YAML files located under .github/workflows.
Attackers can exploit the ability to manipulate commands executed by the pipeline, either directly or indirectly, can be exploited by attackers to execute malicious code in the CI.
How Exploitation of CICD-SEC-4 Happens
For a PPE attack to be successful, several criteria must be met:
- The attacker must obtain permissions against an SCM repository. This could be through user credentials, access tokens, SSH keys, OAuth tokens, or other methods. In some cases, anonymous access to a public repository may suffice.
- Changes to the repository in question must trigger a CI pipeline without additional approvals or reviews. This could be through direct pushes to remote branches or through changes suggested via a pull request from a remote branch or fork.
- The permissions obtained by the attacker must allow triggering the events that cause the pipeline to be executed.
- The files that the attacker can change must define the commands that are executed (either directly or indirectly) by the pipeline.
- The pipeline node must have access to nonpublic resources, such as secrets, other nodes, or compute resources.
Pipelines that execute unreviewed code, such as those triggered off pull requests or commits to arbitrary repository branches, are more susceptible to PPE. Once an attacker can execute malicious code within the CI pipeline, they can conduct malicious operations within the context of the pipeline's identity.
Three Types of Poisoned Pipeline Execution
Poisoned pipeline execution manifests in three distinct forms: direct PPE (D-PPE), indirect PPE (I-PPE), and public PPE (3PE).
In a direct PPE scenario, attackers modify the CI configuration file in a repository they have access to, either by pushing the change directly to an unprotected remote branch on the repo or by submitting a pull request with the change from a branch or fork. The pipeline execution is triggered by the push or pull request events, as defined by commands in the modified CI configuration file, which results in the execution of the malicious commands in the build node once the build pipeline is triggered.
The D-PPE attack example illustrated in figure 1 transpires in the following succession of steps:
- An adversary initiates a new remote branch within the repository, altering the pipeline configuration file with harmful instructions to retrieve AWS credentials stored within the GitHub organization and transmit them to an external server under the attacker's control.
- The code push activates a pipeline that pulls the code, inclusive of the malicious pipeline configuration file, from the repository.
- The pipeline operates according to the configuration file, now tainted by the attacker. The malicious instructions command the AWS credentials, stored as repository secrets, to load into memory.
- Following the attacker's instructions, the pipeline carries out the task of transmitting the AWS credentials to a server under the attacker's control.
- In possession of the stolen credentials, the attacker gains the ability to infiltrate the production environment.
Indirect PPE occurs when the possibility of D-PPE isn’t available to an adversary with access to an SCM repository:
- If the pipeline is configured to pull the CI configuration file from a separate, protected branch in the same repository.
- If the CI configuration file is stored in a separate repository from the source code, without the option for a user to directly edit it.
- If the CI build is defined in the CI system itself — instead of in a file stored in the source code.
In these scenarios, the attacker can still poison the pipeline by injecting malicious code into files referenced by the pipeline configuration file, such as scripts referenced from within the pipeline configuration file, code tests, or automatic tools like linters and security scanners used in the CI. For example:
- The make utility executes commands defined in the Makefile file.
- Scripts referenced from within the pipeline configuration file, which are stored in the same repository as the source code itself (e.g., python myscript.py — where myscript.py would be manipulated by the attacker).
- Code tests: Testing frameworks running on application code within the build process rely on dedicated files, stored in the same repository as the source code. Attackers who can manipulate the code responsible for testing can then run malicious commands inside the build.
- Automatic tools: Linters and security scanners used in the CI commonly rely on a configuration file residing in the repository that typically loads and runs external code from a location defined inside the configuration file.
Rather than poisoning the pipeline via direct PPE, the attacker launching an indirect PPE attack injects malicious code into files referenced by the configuration file. The malicious code is ultimately executed on the pipeline node and runs the commands declared in the files.
In this I-PPE attack example, the chain of events unfolds as follows:
- An attacker creates a pull request in the repository, appending malicious commands to the Makefile file.
- Since the pipeline is configured to trigger on any PR against the repo, the Jenkins pipeline is triggered, fetching the code from the repository — including the malicious Makefile.
- The pipeline runs based on the configuration file stored in the main branch. It gets to the build stage and loads the AWS credentials into environment variables — as defined in the original Jenkinsfile. Then, it runs the make build command, which executes the malicious command that was added into Makefile.
- The malicious build function defined in the Makefile is executed, sending the AWS credentials to a server controlled by the attacker.
- The attacker can then use the stolen credentials to access the AWS production environment.
Public PPE is a type of PPE attack executed by anonymous attackers on the internet. Public repositories often allow any user to contribute, usually by creating pull requests. If the CI pipeline of a public repository runs unreviewed code suggested by anonymous users, it’s susceptible to a public PPE attack. The public PPE can also expose internal assets, such as secrets of private projects, in cases where the pipeline of the vulnerable public repository runs on the same CI instance as private ones.
Importance of Secure Pipeline Execution in CI/CD
Executing malicious unreviewed code in the CI via a successful PPE attack provides attackers with the same level of access and ability as the build job:
- Access to secrets available to the CI job, such as secrets injected as environment variables or additional secrets stored in the CI. Being responsible for building code and deploying artifacts, CI/CD systems typically contain dozens of high-value credentials and tokens — such as to a cloud provider, to artifact registries, and to the SCM itself.
- Access to external assets the job node has permissions to, such as files stored in the node’s file system, or credentials to a cloud environment accessible through the underlying host.
- Ability to ship code and artifacts further down the pipeline, in the guise of legitimate code built by the build process.
- Ability to access additional hosts and assets in the network/environment of the job node.
But organizations can safeguard their software products and infrastructure with a secure pipeline execution that ensures all code compiled, tested, and deployed is legitimate and untampered.
Risks Associated with Poisoned Pipeline Execution
The implications of PPE can be severe, ranging from unauthorized data access, compromised software integrity, system disruptions, to data breaches or even a total system takeover. These risks pose significant threats to both the business and its clients, underscoring the gravity of PPE.
In the eight-step supply chain compromise operation seen in figure 3, the attacker gains access to the CI pipeline and poisons components of the SaaS application. Via the poisoned component, the attacker builds backdoor functionality into the application and sends the poisoned plugins to downstream clients. Since the downstream organizations likely perceive the poisoned package as legitimate, they build it into their cloud or on-premises infrastructure.
From one poisoned CI pipeline, the attacker achieves exponential collateral damage, having created backdoor access to countless organizations. This was the case with the SolarWinds attack.
READ MORE: Anatomy of a CI/CD Pipeline Attack
Preventing Poisoned Pipeline Execution
Preventing and mitigating the PPE attack vector involves multiple measures spanning across both SCM and CI systems:
- Ensure that pipelines running unreviewed code are executed on isolated nodes versus exposed to secrets and sensitive environments.
- Evaluate the need for triggering pipelines on public repositories from external contributors. When possible, refrain from running pipelines originating from forks and consider adding controls such as requiring manual approval for pipeline execution.
- For sensitive pipelines, for example those that are exposed to secrets, ensure that each branch that is configured to trigger a pipeline in the CI system has a correlating branch protection rule in the SCM.
- To prevent the manipulation of the CI configuration file to run malicious code in the pipeline, each CI configuration file must be reviewed before the pipeline runs. Alternatively, the CI configuration file can be managed in a remote branch, separate from the branch containing the code being built in the pipeline. The remote branch should be configured as protected.
- Remove permissions granted on the SCM repository from users who don’t need them.
- Each pipeline should only have access to the credentials it needs to fulfill its purpose. The credentials should have the minimum required privileges.