What Is Insecure System Configuration?

Insecure system configuration is an OWASP Top 10 CI/CD security risk. It arises when CI/CD systems deploy with suboptimal or default configurations. It can include unnecessary open ports, default credentials, unpatched systems, poorly segregated networks, or disabled security features. These vulnerabilities can expose the system to unauthorized access and increase the propagation of malware and the potential for malicious code injection into the deployment process, ultimately leading to data breaches and disruption to business operations. Insecure configurations can also lead to the misuse of legitimate CI/CD processes, enabling attackers to manipulate workflows and gain access to production environments.

CICD-SEC-7: Insecure System Configuration Explained

Insecure system configuration represents a significant security risk. It arises from deficiencies in the security settings, configuration, and hardening of various systems across the pipeline, such as source code management (SCM), CI systems, and artifact repositories. These vulnerabilities often serve as easy targets for attackers seeking to expand their reach within the environment.

Multiple systems from a variety of vendors make up CI/CD environments. To enhance CI/CD security, it's essential to concentrate not only on the code and artifacts flowing through the pipeline but also on the stance and resilience of each individual system.

Similar to data storage and processing systems, CI/CD systems involve numerous security settings and configurations at the application, network, and infrastructure levels. These settings play a significant role in determining the security posture of the CI/CD environments and their susceptibility to potential breaches.

Attackers hunt for CI/CD vulnerabilities and misconfigurations to exploit. Potential hardening flaws include:

• Systems running outdated versions
• Systems with overly permissive network access controls
• Self-hosted systems with administrative permissions on the underlying OS
• Poor credential hygiene

System Configuration Defined

System configuration refers to the process of setting up systems and services, defining how they interact, and establishing the rules governing their operation. This includes setting up hardware, installing and configuring software, and establishing network connections. Because the configuration process can significantly impact the functionality, performance, and security of a system, getting it right — and maintaining its optimal status — is vitally important.

Components of Secure System Configuration

A secure configuration involves correctly setting system parameters, managing access controls, and implementing security measures for the systems that underpin the CI/CD pipeline. Such configurations mitigate the risk of unauthorized access and prevent the exploitation of vulnerabilities in the systems that form the backbone of the development environment.

Complexity in the CI/CD environment stems from the CI/CD environment, in that system configuration extends beyond individual systems to the interconnections between tools, services, and platforms used in the pipeline. Not surprisingly, the primary component of effective and secure system configuration is strict configuration management.

How CICD-SEC-7 Happens

The root cause of insecure system configurations often points to human error, lack of proper procedures, or inadequate understanding of security requirements. It can result from something as simple as leaving default settings unchanged, permitting excessive permissions, or neglecting to update and patch systems.

A Hypothetical Situation

The attacker scans the target network, a tech company specializing in artificial intelligence, and discovers an exposed Jenkins server configured with default settings. Employing readily available tools and an API call, they proceed to mine the Jenkins server metadata for potential information about the underlying system. A goldmine of information floods their screen — data about plugins, jobs, system configurations, and more. Among this trove of details, one string of information stands out. AWS keys. They were being used by Jenkins for deploying applications on AWS and weren’t adequately secured. The keys are for an administrator account, granting potentially unrestricted access to the company's AWS environment.

Using the keys to infiltrate the company's AWS infrastructure, the attacker enters the heart of the organization's system. They locate an S3 bucket housing proprietary AI models, and with admin-level access from the stolen AWS keys, they swiftly download the models and exit without triggering an alarm.

The attacker then decides to exploit this system further. Aware that the Jenkins server has write permissions to the GitHub repositories, they plant a malicious code snippet into the main application source code that creates a backdoor into the application. In the next deployment cycle, the company unknowingly pushes the application into production. Now armed with a persistent backdoor, the attacker can pilfer data, manipulate system controls, and plant additional malware — all under the radar of the company's security systems.

Importance of Secure System Configuration in CI/CD

A misconfiguration at any juncture in the engineering environment could expose the entire pipeline to potential threats. An attacker leveraging the misconfiguration could obtain unauthorized access to the CI/CD system — or worse, compromise the system and access the underlying OS. The attacker might manipulate legitimate CI/CD flows, obtain sensitive tokens, and potentially access production environments. In some scenarios, configuration flaws may allow an attacker to move laterally within the environment and outside the context of CI/CD systems.

Risks Associated with Insecure System Configuration

DevOps teams with an understanding of the risks associated with insecure system configuration are equipped to design less vulnerable systems, take responsibility for the security of the systems they design, and mitigate risks when they arise.

Case Study 1: PHP Shifts to GitHub Following Security Incident and Potential User Database Leak

In April 2021, the PHP community faced a security incident involving git.php.net. Initially suspected as a server compromise, the investigation revealed that malicious commits were made using HTTPS and password-based authentication, bypassing the Gitolite infrastructure. The master.php.net user database may have leaked, prompting a system migration to main.php.net and a password reset for all php.net users. Git.php.net and svn.php.net were made read-only, and PHP's primary repository was moved to GitHub, enhancing security and streamlining the development workflow.

Case Study 2: Webmin Overhauls Security Measures Following Malicious Code Insertion Incident

In August 2019, Webmin, a web-based system configuration tool, suffered a security breach when malicious code was inserted into its source code. The breach, which was not an accidental bug, allowed remote command execution. The malicious code was introduced via a compromised development build server. Upon discovery, Webmin responded by updating the build process to use only checked-in code from GitHub, rotating all accessible secrets and auditing all GitHub commits over the past year for similar vulnerabilities.

Case Study 3: Nissan North America's Source Code Exposed Online Due to Misconfigured Git Server

In a significant security lapse, Nissan North America's source code for mobile apps and internal tools leaked online due to a misconfigured Git server. The server, left exposed with default username and password 'admin/admin', was discovered by Swiss-based software engineer Tillie Kottmann. The repository contained code for various Nissan apps, diagnostics tools, dealer portals, marketing tools, and more. Nissan confirmed the incident, secured the affected system and asserted that no personal data was accessible.

Case Study 4: New York State's IT Department Exposes Internal Code Repository Online

An internal code repository used by New York State's IT department was inadvertently exposed online, making it accessible to anyone. The GitLab server, discovered by cybersecurity company SpiderSilk, contained projects with secret keys and passwords for state government systems. The server was configured to allow anyone to create a user account and log in. The server was first detected online on March 18, and it was taken offline after the exposure was reported. The server was reportedly a test box set up by a vendor and has since been decommissioned.

Preventing Insecure System Configuration in CI/CD

Though misconfiguration can provide an entry point for attackers, leading to significant security breaches, secure system configuration remains overlooked in many development processes. Insider recommendations from the authors of the OWASP Top 10 CI/CD Security Risks list can put your systems in good standing:

• Keep an inventory of systems and versions in use, mapping each system to a designated owner. Regularly check these components for known vulnerabilities. When a security patch becomes available, update the vulnerable component. If no patch is available for the vulnerable component, consider removing the component or system. Alternatively, minimize the potential impact of exploiting the vulnerability by limiting the system's access or its ability to perform sensitive operations.
• Ensure network access to the systems aligns with the principle of least privilege access.
• Set up a process to periodically review all system configurations. Focus your review on settings that could affect the system's security posture. Ensure optimal settings.
• Grant permissions to the pipeline execution nodes based on the principle of least privilege. A common misconfiguration in this context involves granting debug permissions on execution nodes to engineers. Many organizations allow this, but it's crucial to consider that any user with access to the execution node in debug mode could expose all secrets while they're loaded into memory. They could also use the node's identity, effectively granting elevated permissions to any engineer with this permission.

Industry Standards for System Configuration Security

Several industry standards outline best practices for system configuration security. The Center for Internet Security (CIS) provides comprehensive benchmarks for secure configuration, while the National Institute of Standards and Technology (NIST) also publishes guidelines for configuring systems for security.

Encrypting Your Secrets

Secrets such as passwords, API keys, and database credentials should be encrypted at rest and in transit. Never store secrets in your code or configuration files. Use a secrets management tool like HashiCorp Vault or AWS Secrets Manager. These tools keep secrets encrypted and control access to them, helping to prevent your organizations’ credentials from landing in the wrong hands.

Logging and Monitoring Your Systems

A key part of maintaining secure system configuration involves establishing clear policies and routinely monitoring for compliance. It’s important to log all activity so you can detect suspicious activity and quickly respond to security incidents. You should also monitor your system for signs of attack, such as unusual traffic patterns or failed login attempts.

Patching Vulnerabilities

Ensure you have a comprehensive vulnerability identification and patching system in place. Systematically identify vulnerabilities and prioritize remediation. In instances when vulnerabilities can’t be patched, use alternative mitigations, such as removing admin rights. Remember, keeping your systems up to date means regularly applying patches and updates to your servers, applications, and CI/CD tools.

Eliminating Unnecessary Accounts and Privileges

Enforce least privilege by removing unnecessary accounts (such as orphaned accounts and unused accounts). This is one of the most powerful security practices for reducing your attack surface. Ensure that every component of your system — including users, processes, and services — has only the minimum privileges necessary to perform its function. Doing so will limit damage in the event of a compromised component.

Erecting Network Roadblocks

Dividing your network into smaller, isolated segments will limit lateral movement if an attacker gains access to your network. Use firewalls and access control lists (ACLs) to control traffic between segments. Encrypt traffic, block unused or unneeded open network ports, and disable or remove unnecessary protocols and services. Regularly audit your firewall rules.

Securing Your Build Servers

Your build servers are responsible for compiling and packaging your code, so they’re a prime target for attackers. Make sure that your build servers are properly hardened with up-to-date security patches and strong passwords. And remember that securing your build environment means isolating it from your production environment.

Auditing Your Existing Systems

Regular audits and reviews help ensure that system configurations remain secure over time. Carry out a comprehensive audit of your existing technology. Use penetration testing, vulnerability scanning, configuration management, and other security auditing tools to find flaws in the system and prioritize fixes. Conduct system assessments against resources using industry standards from NIST, Microsoft, CIS, DISA, etc.

Using Tools to Help Secure System Configuration

Many tools exist to help manage and secure system configuration. Configuration management tools such as Ansible, Chef, or Puppet allow for automated configuration and consistent application across environments. For cloud-based systems, cloud-native services like AWS Config, Azure Policy, and Google Cloud Security Command Center can aid in maintaining secure configuration.

Insecure System Configuration FAQs