What Is Insufficient Credential Hygiene?

5 min. read

Insufficient credential hygiene, listed as CICD-SEC-6 on the OWASP Top 10 CI/CD Security Risks, involves inadequate management and protection of credentials used in the CI/CD pipeline. Poor practices such as using hard-coded credentials, shared keys, and weak passwords give attackers easy access to CI/CD systems.

Factors that contribute to insufficient credential hygiene in CI/CD include the lack of a formal process for managing credentials, which often leads to uncontrolled proliferation of access keys and passwords. Breaches resulting from this CI/CD security risk typically cause significant damage to organizations, particularly those in highly regulated industries.

CICD-SEC-6: Insufficient Credential Hygiene Explained

Insufficient credential hygiene refers to the risks associated with an attacker's ability to obtain and use secrets and tokens spread throughout the CI/CD pipeline due to oversights in access controls, insecure secret management, and overly permissive credentials.

CI/CD environments consist of multiple systems that communicate and authenticate with each other. These interactions pose significant challenges in protecting credentials due to the diverse contexts in which credentials can exist:

  • Application credentials are used during runtime.
  • Pipelines use credentials to deploy infrastructure, artifacts, and apps to production.
  • Engineers use credentials in their testing environments and within their code and artifacts.

The variety of contexts, combined with numerous methods and techniques for storing and using credentials, leads to a high likelihood of insecure credential usage practices.

Oversights and Risk Factors

Code Containing Credentials Pushed to an SCM Repository Branch

Hard-coded credentials pushed to an SCM repository branch can occur by mistake or by intention, without understanding the risk. But once in the repository branch, the credentials are exposed to anyone with read access to it. Even if deleted from the branch, they continue to appear in the commit history, viewable by anyone with repository access.

Insecure Use of Credentials in Build and Deployment Processes

Credentials in build and deployment processes access code repositories, read from and write to artifact repositories, and deploy resources and artifacts to production environments. Given the number of pipelines and target systems they need access to, DevOps teams need to know:

  • The context and use method of each set of credentials in use
  • Whether each pipeline has access only to the credentials needed to fulfill its purpose
  • Whether unreviewed code flowing through the pipeline has access to credentials
  • How credentials are called and injected to the build
  • Whether credentials are accessible only in runtime and only from the contexts that requires them

Credentials in Container Image Layers

Credentials required only for building the image can remain in one of the image layers, available to anyone who can download the image.

Credentials Printed to Console Output

Pipelines often print credentials to the console output, either deliberately or inadvertently. Such actions can expose credentials in cleartext within logs, making them accessible to anyone with access to the build results. These logs can potentially flow into log management systems, broadening their exposure.

Unrotated Credentials

Credentials, spread throughout the engineering ecosystem, are exposed to numerous employees and contractors. Failure to rotate credentials leads to an ever-increasing population of people and artifacts possessing valid credentials. Pipeline credentials, such as deploy keys, exemplify this issue. Often managed under the "if it isn’t broken, don’t fix it" principle, these credentials can remain unrotated for years.

Credential Hygiene Defined

Playing a critical role in CI/CD security, credential hygiene refers to the practices associated with managing and protecting authentication mechanisms, such as passwords, tokens, keys, and certificates used to control access to systems and resources. It involves secure credential creation, storage, usage, rotation, and retirement.

In the context of CI/CD, maintaining superior credential hygiene helps prevent unauthorized access to development, testing, and production environments, ensuring the integrity and confidentiality of applications throughout their lifecycle.

Effective credential hygiene consists of several components:

  • Secure credential storage: Credentials should be stored using encryption and secure secret management solutions.
  • Credential rotation: Regularly changing credentials minimizes the window of opportunity for an attacker to use compromised ones.
  • Least-privileged access: Granting minimal necessary privileges for each user or process can limit potential damage if credentials are compromised.
  • Audit logging: Regular audits of credential use can help detect suspicious activities and ensure compliance with best practices.

How Exploitation of CICD-SEC-6 Happens

Attackers exploit insufficient credential hygiene by identifying exposed credentials and using them to infiltrate systems. For example, they might discover a container image with an exposed credential, either through manual inspection or automated scanning. Once found, they extract the credential, often using specialized tools that can retrieve information from container images.

After extraction, the attacker validates the credential to ensure it’s active, often from a compromised or disposable system to avoid detection. With a valid credential, the attacker gains access to the system or service. Depending on the permissions associated with the credential, they might access sensitive data, execute commands, or perform other malicious actions.

If the credential provides limited access, the attacker might use it as a steppingstone to escalate privileges or move laterally within the network. This could involve exploiting system vulnerabilities, launching phishing attacks against other users, or finding additional credentials within the accessed system.

Once the attacker has gained sufficient access, they begin exfiltrating customer data, intellectual property, or other sensitive information. They either store the data on a remote server controlled by the attacker or relocate it within the victim’s network for later retrieval. To maintain access to the system, the attacker might create new accounts, install backdoors, or periodically refresh the exploited credential to prevent it from expiring.

Importance of Credential Hygiene in CI/CD

Attackers covet credentials. Promising access to valuable resources or the ability to deploy malicious code and artifacts, credentials quite literally give bad actors the keys to the kingdom. And all too often, engineering environments unwittingly provide attackers with ample opportunities to acquire credentials.

Risks Associated with Insufficient Credential Hygiene

Insufficient credential hygiene exposes an enterprise to significant risks involving sensitive systems, data breaches, and disruption of development and deployment process. Poor credential hygiene can also result in reputational damage and regulatory fines.

Case Study 1: Exposed Link to Reset Credentials and a Misconfig with Role Assignments Gives Hacker Admin Privileges

In May 2019, Stack Overflow fell victim to a security breach. An unidentified user exploited a loophole to gain unauthorized moderator and developer access across all Stack Exchange Network sites. Using a sophisticated approach — initially probing the infrastructure and gradually escalating their privilege level — the attacker accessed Stack Overflow's source code. Credentials had been obtained through site settings, which led to the breach that exposed the personal data of 184 users. Stack Overflow's response team quickly intervened, revoking privileges, suspending the account, and initiating an audit. This incident led to a significant overhaul of Stack Overflow's credential management and security practices.

Case Study 2: Samsung's Misconfigured GitLab Instance Leads to Unintentional Source Code Exposure

In 2019, Samsung's development lab inadvertently leaked sensitive source code, credentials, and secret keys for several internal projects, including its SmartThings platform. The leak occurred due to the misconfiguration of a GitLab instance hosted on a Samsung-owned domain, Vandev Lab. The projects were set to "public" without proper password protection, allowing anyone to access and download the source code. The exposed data included credentials for an entire AWS account, private GitLab tokens, and private certificates for Samsung's SmartThings' iOS and Android apps.

Case Study 3: A Tale of Stolen Credentials and Concealed Cyberattacks

In 2017, Uber disclosed a significant data breach involving its Amazon Web Services account. Two hackers obtained login credentials to access data stored on Uber's Amazon Web Services, leading to the exposure of 57 million customers' and drivers' personal data. Uber's security team discovered that the attackers used stolen GitHub credentials to access Uber's private code repository. The company faced criticism for concealing the breach, which led to significant reputational damage.

Preventing Insufficiency in Credential Hygiene

A substantial potential for human error, paired with knowledge gaps pertaining to secure credentials management and the concern of breaking processes due to credential rotation, expose high-value resources to compromise. But DevSecOps teams can reverse risk profiles by replacing weak habits with strategic recommendations:

  • Establish procedures to continuously map credentials found across the different systems in the engineering ecosystem — from code to deployment. Ensure each set of credentials follows the principle of least privilege and has been granted the exact set of permissions needed by the service using it.
  • Avoid sharing the same set of credentials across multiple contexts. This increases the complexity of achieving the principle of least privilege as well as having a negative effect on accountability.
  • Prefer using temporary credentials over static credentials. In case static credentials need to be in use — establish a procedure to periodically rotate all static credentials and detect stale credentials.
  • Configure usage of credentials to predetermined conditions only (like scoping to a specific source IP or identity) to ensure that, even in case of compromise, exfiltrated credentials can’t be used outside your environment.
  • Detect secrets pushed to and stored on code repositories. Use controls such as an IDE plugin to identify secrets used in the local changes. Implement automatic scanning with each code push, along with periodic scans on the repository and its past commits.
  • Ensure secrets used in CI/CD systems are scoped in a manner that allows each pipeline and step access only to the secrets it requires.
  • Use built-in vendor options or third-party tools to prevent secrets from being printed to console outputs of future builds. Ensure all existing outputs do not contain secrets.
  • Verify that secrets are removed from any type of artifact — layers of container images, binaries, Helm charts.

Industry Standards for Credential Hygiene in CI/CD

Industry standards guide organizations in maintaining robust credential hygiene. The National Institute of Standards and Technology (NIST) recommends practices like frequent credential rotation, utilizing strong, complex passwords, and adopting multifactor authentication. ISO 27001 also provides guidance on access control, including the secure management of passwords.

Regularly Updating and Rotating Credentials

In CI/CD environments, it’s essential to regularly update and rotate credentials. Automated systems can help rotate credentials on a set schedule, reducing the risk of human error and ensuring consistent updates.

Implementing Multifactor Authentication (MFA)

MFA provides an additional layer of security by requiring more than one method of authentication to verify a user's identity. In a CI/CD pipeline, MFA can be implemented at various stages, such as during login or when making significant changes. Even if an attacker manages to compromise a password, without the second factor — like a fingerprint, a hardware token, or a time-based one-time password — they can’t gain access.

Auditing and Monitoring Access

Continuous auditing and monitoring of access logs can help detect unusual activity, potentially identifying a breach in progress. Automated systems can alert to suspicious behavior, such as multiple failed login attempts, logins from unusual locations, or unusual times of activity. These systems can also help ensure compliance with access policies and identify when accounts might have been compromised.

Avoiding Hardcoding of Credentials in Code

Hardcoding credentials in code is a common mistake that can lead to unintentional credential leaks. Instead, use secure methods of storing credentials, such as environment variables or secure vaults. Automated tools can scan code for hard-coded credentials, helping to prevent them from making it into production. This practice not only improves security but also makes it easier to rotate credentials, as code won’t need to be updated in the process.

Tools and Techniques to Strengthen Credential Hygiene

Modern tools can significantly enhance credential hygiene. Secrets management tools like HashiCorp Vault, CyberArk, and AWS Secrets Manager ensure secure storage and management of credentials. Additionally, credential management platforms provide features for automatic password rotation, auditing, and anomaly detection. It's also crucial to integrate these tools with the CI/CD pipeline to ensure secure deployment processes.

Establishing Credential Hygiene Policies

Organizations need to establish clear policies around credential hygiene, including the requirement for unique, complex passwords, regular password changes, and the avoidance of credential hardcoding. These policies should be regularly reviewed and updated in response to evolving threat landscapes.

Role of Automated Testing in Ensuring Sufficient Credential Hygiene

Automated testing is critical to maintaining adequate credential hygiene. Using static and dynamic analysis tools, organizations can automate the process of detecting weak or hard-coded credentials in the codebase.

Implementation of Secure Coding Practices

Developers need to adhere to secure coding practices to prevent credential-related security issues. This includes avoiding the hardcoding of credentials, utilizing environment variables or secure vaults for storing sensitive data, and regularly updating and rotating API keys and other credentials. Secure coding guidelines, paired with regular training, can help enforce these practices.

Insufficient Credential Hygiene FAQs

Hard-coded credentials refer to embedding authentication data directly into the source code. The practice includes placing usernames, passwords, tokens, or API keys within the code to authenticate to databases, services, or systems. Though seen as a convenience, hardcoding poses a significant security risk. If the code is exposed or becomes accessible, hard-coded credentials become publicly visible and can lead to unauthorized access, data breaches, and other security incidents. Secure coding guidelines strongly encourage hardcoding alternatives, such as using environment variables, secure vaults, or configuration files.
Cleartext refers to data transmitted or stored in an unencrypted form. Because it's readable and understandable to anyone with access to it, sensitive data should be encrypted, both in transit and at rest. Unfortunately, cleartext simplifies data processing and debugging, which can tempt developers to forgo encryption and risk the consequences.

Plaintext and cleartext are often used interchangeably, but their meanings differ slightly in terms of cryptography and data security.

Plaintext usually refers to raw, readable data in its pre-encrypted form, whereas cleartext is used to describe data transmitted or stored without encryption.

A key vault is a secure cryptographic key storage service that allows users to manage and control their cryptographic keys. It provides secure secret storage and tightly controls access to secrets, ensuring only authorized entities can access them. Key vaults are often used in cloud environments to store keys and secrets used by cloud services and applications, providing a centralized and secure solution for key management.
Scoping secrets refers to the practice of organizing and managing secrets within defined boundaries or "scopes" in a secrets management system. Each scope can contain multiple secrets, and access to these secrets is controlled at the scope level. This means that permissions are granted or denied to users or applications based on the specific secret scope they are trying to access. Scoping secrets is a critical practice in secure credential management as it allows for fine-grained access control, ensuring that users and applications only have access to the secrets they need, reducing the potential impact of a compromised account or secret.
A secret scope in Databricks is akin to a safety box for storing secrets. It organizes secrets and manages access to them based on the permissions configured by the scope admin. Secrets can include database credentials and connection strings, SSH keys, API keys, passwords, and private certificates. In essence, a secret scope serves as a centralized storage unit for storing secrets.
Databricks is a unified data analytics platform designed to simplify the process of building big data and AI solutions. It provides a collaborative workspace where data scientists, data engineers, and business analysts can work together. Databricks integrates with various data storage platforms and provides a managed environment for running data processing frameworks like Apache Spark. It also supports machine learning and AI workflows, making it a comprehensive solution for complex data analytics tasks.
In Databricks, a secret scope is a construct used to store secrets used by Databricks notebooks and jobs to access external resources securely. Databricks allows the creation of two types of secret scopes: Azure Key Vault-backed and Databricks-backed secret scopes. Azure Key Vault-backed secret scopes are stored in an Azure Key Vault, while Databricks-backed secret scopes are stored in (and managed by) Databricks. By using secret scopes, Databricks users can avoid hardcoding sensitive information in notebooks or jobs, enhancing the security of their data operations. Access to these secret scopes can be controlled, ensuring that only authorized users or applications can access the stored secrets.
Public key infrastructure (PKI) is a set of roles, policies, and procedures needed to create, manage, distribute, use, store, and revoke digital certificates. It's a framework for establishing trusted identities in a network. It uses asymmetric cryptography, involving a pair of keys — a public key, which is disseminated widely, and a private key, which is kept secret by the user. The PKI validates the identities of entities and securely exchanges data in an untrusted environment like the internet.
Secrets management refers to the tools and methods for managing digital authentication credentials like API keys, passwords, and tokens. Secrets management solutions provide mechanisms for storing these secrets in encrypted formats and controlling access to them, often through access policies and audit logs.
Identity federation is a system of trust between various software applications and systems across multiple enterprises that share access credentials. It allows for improved credential management across different systems.
A hardware security module (HSM) is a physical device that safeguards and manages digital keys for strong authentication. It provides a secure environment for encryption, decryption, and digital signing. Designed to be tamper-resistant, HSMs prevent unauthorized access or modification of the sensitive information they hold.
A digital certificate is an electronic document that uses a digital signature to bind a public key with an identity. The certificate includes information about the key, information about its owner's identity, and the digital signature of an entity that has verified the certificate's contents. Digital certificates are typically issued by a trusted third party, known as a certificate authority (CA), which guarantees the information contained within the certificate.