What Is Insufficient Logging and Visibility?

5 min. read

Insufficient logging and visibility refers to a CI/CD security risk arising from inadequate data capture, storage, and analysis within continuous integration and continuous deployment processes. Listed as an OWASP Top 10 CI/CD Security Risk, this insufficiency creates blind spots that hinder the detection of anomalies and malicious activities within CI/CD systems. While inadequate logging prevents traceability of actions from code commits to deployment, limited visibility obstructs access to and interpretation of logged data.

CICD-SEC-10: Insufficient Logging and Visibility Explained

Insufficient logging and visibility involves a lack of comprehensive logging and monitoring in the CI/CD pipeline that allows adversaries to execute malicious activities undetected throughout the attack kill chain, a term coined by Lockheed Martin. In addition to potentially masking malicious activities and delaying response times, this security risk impedes the identification of a threat actor’s techniques, tactics, and procedures (TTPs) during post-incident investigations.

With myriad attack vectors targeting engineering ecosystems, CI/CD environments require in-built capabilities to promptly detect attacks. Addressing this challenge involves providing complete visibility on human and programmatic access.

The sophisticated nature of CI/CD attack vectors also demands system audit logs to help identify unauthorized access, privilege escalations, and policy violations. Equally important, it requires applicative logs to aid in the detection of malicious code injection, unauthorized changes, and vulnerabilities introduced during development, testing, or deployment phases.

Without these capacities, CICD-SEC-10, as it’s referred to by OWASP, leaves the door open to all possible fallout of unauthorized access to the CI/CD pipeline.

Logging and Visibility Defined

Visibility encompasses the ability to observe and understand the flow of code, artifacts, configurations, and associated metadata throughout the software delivery process. Logging, on the other hand, refers to the process of capturing and storing relevant events, activities, and data points within the CI/CD pipeline.

Several types of logs play a role in CI/CD security, each contributing to identifying potential threats, vulnerabilities, and issues in the pipeline.

Access Logs

Access logs record information about who accessed the CI/CD environment, when, and from where to help identify unauthorized access attempts and trace the origin of security breaches.

Authentication and Authorization Logs

Authentication and authorization logs track user authentication and authorization activities, such as successful and failed login attempts, password changes, and access privilege modifications. These logs help monitor and detect suspicious activities and potential insider threats.

Build Logs

Build logs capture information about the build process, such as which code changes were included in a build, who initiated the build, and any errors or warnings that occurred during the build.

Deployment Logs

Deployment logs record information about the deployment process, including which versions of the application were deployed, when, and by whom. AppSec teams use deployment logs to track changes to the production environment and ensure that only authorized and tested code is deployed.

Code Repository Logs

Code repository logs help identify unauthorized or malicious changes to the source code by tracking changes to the codebase — including who made the changes, when, and what the changes were.

Security Scanning Logs

Security scanning logs capture the results of security scans and vulnerability assessments performed on the code, infrastructure, and deployed applications.

Infrastructure Logs

Infrastructure logs record information about the underlying infrastructure used to support the CI/CD pipeline, such as server logs, network logs, and firewall logs.

Audit Logs

Audit logs track changes to the CI/CD pipeline configuration to confirm whether the pipeline aligns with organizational policies and best practices. Important changes recorded in this log include modifications to build or deployment scripts, changes to security settings, and updates to third-party tools and integrations.

Effective CI/CD security, in addition to the logs listed above, draws on the applicative logs generated by individual applications. These can involve performance logs, transaction logs, debug logs, event logs — all of which provide AppSec teams with insights into a given application's behavior, performance, and potential issues.

Components of Effective Logging and Visibility

Effective logging and visibility require careful consideration of several components. These include:

  • Selection of relevant log data
  • Implementation of proper log formats
  • Use of standardized event levels and severity levels
  • Integration of logging mechanisms into the pipeline workflow
  • Establishment of secure log storage and retention practices

Also multifaceted, comprehensive visibility entails tracking the movement and transformation of artifacts, managing dependencies, and capturing metadata associated with each stage of the CI/CD process.

Logging and Visibility in the CI/CD Context

Capturing all relevant pipeline events and benefitting from real-time awareness of the pipeline's security posture relies on the integration of logging and visibility tools and practices within the CI/CD workflow. Logging and visibility should be integrated throughout — from code commits, build processes, artifact storage, and testing to deployment and runtime. Each stage of the CI/CD pipeline presents valuable data and insights that contribute to overall security monitoring and incident detection.

By focusing on both audit and applicative logs, AppSec practitioners can correlate data across CI/CD stages, monitor the full pipeline, identify patterns and anomalies, and improve incident investigations.

How CICD-SEC-10 Happens

While maintaining a clear line of sight into system operations is paramount, full pipeline integration of logging and visibility doesn’t always happen. All too often, organizations grapple with the pitfalls of insufficient logging and monitoring, leaving them vulnerable to stealthy cyberthreats.

At its core, logging captures a digital trail of system activities, while monitoring interprets this data to detect anomalies. Blind spots result when these processes fall short, and attackers pounce, knowing that their malicious activities may go unnoticed.

Imagine a scenario where a threat actor infiltrates a CI/CD pipeline. Without detailed logs, tracing the origin of the breach becomes a daunting task. What’s more, without real-time monitoring, the breach could remain undetected for an extended period, giving the attacker an opportunity to compromise the deployment process.

The lack of comprehensive logging strategies and inadequate log capture mechanisms put organizations in harm’s way. Other inroads to exploitation of this risk involve incomplete or inconsistent log formats, insufficient event correlation, limited monitoring capabilities, and the absence of real-time visibility into pipeline activities.

Importance of Sufficient Logging and Visibility in CI/CD

With adversaries increasingly targeting engineering environments, timely and comprehensive data spells the difference between successful countermeasures and catastrophic outcomes.

Risks Sources and Outcomes

Insufficient logging and visibility pose significant risks to the security and integrity of the CI/CD pipeline. These risks include:

Inability to Detect and Respond to Security Incidents

Without sufficient logging and visibility, security incidents such as unauthorized access, code injection, or data breaches may go undetected. The lack of actionable log data hinders incident response teams' ability to promptly investigate and mitigate incidents. Persisting in the environment, attackers move laterally, further compromising systems and causing widespread damage.

Limited Forensic Analysis

A CI/CD breach without logs for forensic analysis compromises incident response and investigation. It becomes challenging to identify the breach, assess its scope, and perform root cause analysis, potentially leaving the environment exposed to further attacks. What’s more, tracing the attacker and attributing the attack to a specific threat actor could prove elusive without an IP address, user agent, or knowing the tools or tactics they used. The absence of logs also makes it difficult to determine the impact of the breach and can result in noncompliance with security regulations.

Ineffective Threat Hunting

Identifying potential threats and vulnerabilities within the CI/CD environment relies on vigorous logging and visibility functions, but insufficient data and limited visibility making it difficult to identify and address emerging risks.

Operational Challenges

Insufficient logging and visibility impact operational efficiency. Troubleshooting issues, diagnosing errors, and identifying performance bottlenecks become arduous without comprehensive log data and visibility into system activities.

Preventing Insufficiency in Logging and Visibility

Prioritize logging and visibility for your CI/CD pipelines with recommendations proven to potentiate security and enable effective incident management.

Map the Environment

To achieve strong visibility, you must intimately understand all systems vulnerable to potential threats. Any system involved in the CI/CD process — from SCM, CI, artifact repositories, package management software, container registries, and orchestration engines — could be a breach point.

Catalog every system your organization uses, including every instance of these systems, which is especially important with self-managed systems like Jenkins.

Enable Appropriate Log Sources

After identifying all systems, ensure activation of all relevant logs. Many systems don't enable these logs by default. Prioritize visibility for both human and programmatic access, emphasizing the identification of audit and applicative log sources.

Centralize Logs

Send logs to a centralized location, such as a security information and event management (SIEM) platform, to facilitate log aggregation and correlation across systems, enhancing detection and investigative capabilities.

Create Alerts

Set alerts to flag anomalies and potential threats. Monitor each system individually and watch for irregularities in the code shipping process, which spans multiple systems and demands a deep understanding of internal build and deployment processes.

Industry Standards for Logging and Visibility in CI/CD

Industry standards and best practices tailored for CI/CD pipelines help organizations stay ahead of attackers while preserving the agility of their deployment processes. In addition to defining logging requirements, using standardized log formats, and aligning with regulatory compliance requirements, ensure your organization has built a solid foundation for optimal logging and visibility.

Tools and Techniques to Improve Logging and Visibility

Organizations that integrate logging and visibility data with a security information and event management (SIEM) system benefit from the comprehensive analysis and correlation of security events. By merging logs from all stages of the CI/CD pipeline, they gain a complete view of their security landscape, enabling them to swiftly detect patterns or anomalies.

Knowledgeable of key indicators, DevSecOps teams can flag areas for enhancement in their logging and visibility practices. Such insights guide teams to refine their strategies, whether that involves optimizing log coverage, refining granularity, or extending log retention periods. Armed with the right logging and visibility solutions, teams can fortify the security posture of the CI/CD pipeline.

Real-Time Monitoring and Alerting

By actively monitoring the CI/CD pipeline, security teams can identify unauthorized access and vulnerabilities, allowing for timely intervention and mitigation of threats. Alerts for suspicious or anomalous activities ensure teams remain knowledgeable of critical security events, enabling rapid resolution and reduced risk of breach. Used together, real-time monitoring and alerting underpin the resilient CI/CD environment, safeguarding the integrity of the software development lifecycle.

Log Access Controls

Securing access to logs is vital to prevent unauthorized access and potential data tampering. Implementing role-based access controls ensures that only authorized users can view, modify, or delete logs. Regularly reviewing and updating permissions ensures that access remains restricted to relevant stakeholders.

Audits, Reviews, and Feedback Loops

Don’t neglect regular reviews of logging and visibility practices. Internal or external audits can identify gaps and weaknesses in logging configurations, alert thresholds, and log retention policies. With insights gained, AppSec teams can identify bottlenecks and enhance system performance. By continually iterating on logging practices from regular feedback, organizations can ensure that their pipelines remain agile and secure. Equally important, organizations can prevent insufficiencies that deny them the intel they need to achieve business outcomes.

Insufficient Logging and Visibility FAQs

Log aggregation involves collecting and centralizing log data from various sources into a single location. By consolidating logs from servers, applications, and network devices, DevOps teams can more efficiently monitor, analyze, and respond to events. Centralized logging aids in identifying patterns, troubleshooting issues, and detecting security incidents.

Metadata refers to the information associated with code, artifacts, and configurations in the software delivery process. It provides context and details about these components, making it easier to understand their purpose, relationships, and history.

Examples of metadata in the software delivery process include:

  • Version history of the code that details when a version was created, who created it, and if updates or fixes were implemented
  • Author information that identifies developers who contributed to the code — their names, roles, and contact details
  • Commit messages that explain what changes were made to the code and why
  • Build and deployment timestamps that record when a specific build or deployment occurred, enabling teams to track progress and identify potential issues
  • Dependency information that lists details about other software components, libraries, or frameworks the code relies on
  • Test results that provide the outcome of tests performed on the code, including pass/fail status and identified issues
  • Security scan results that detail security vulnerabilities or issues identified during security scans
A SIEM system consolidates and analyzes real-time data from across an organization's infrastructure, including servers, network devices, and applications. Its primary functions include event correlation, alerting, and reporting. By identifying patterns and anomalies, SIEM helps security teams detect, investigate, and respond to potential threats. Integration with threat intelligence feeds enhances its capability to recognize emerging threats.
While centralized logging provides a unified repository of logs, SIEM takes it a step further by actively analyzing and correlating those logs for security purposes.
Distributed tracing tracks and visualizes requests as they traverse through components of a microservices architecture. By capturing detailed information about operations and their interactions, it provides insights into performance bottlenecks, failures, and latencies. For DevOps and AppSec teams, distributed tracing tools like Jaeger or Zipkin are invaluable in diagnosing issues, optimizing performance, and ensuring seamless service interactions in complex, distributed systems.
A log analyzer processes and interprets log data to extract meaningful insights. By parsing, filtering, and visualizing logs, it aids in identifying system performance issues, security incidents, or operational anomalies. DevOps and AppSec experts leverage log analyzers to streamline troubleshooting, optimize system performance, and bolster security measures.

Logging level determines the granularity of information captured in logs. Ranging from detailed debug messages to high-level error notifications, logging levels allow developers and system administrators to fine-tune the verbosity of log outputs. Common logging levels include DEBUG, INFO, WARN, ERROR, and FATAL, each representing a different severity or importance.

In DevOps practices, dynamically configuring logging levels ensures efficient resource utilization that balances the capture of diagnostic details while avoiding log volume overload.

Audit logs track user activities, system events, and transaction histories, providing a clear trail of who did what and when. Applicative logs focus on the application's operational behavior, capturing events like system calls, errors, or status changes. By ensuring granularity in both types of logs, organizations can delve into any event, ensuring thorough investigations and informed decision-making.
An audit trail in CI/CD captures a chronological record of changes, operations, and events within the build and deployment processes. It documents who did what and when, providing a transparent history of code commits, build results, deployment statuses, and configuration changes. In security-sensitive environments, audit trails also play a pivotal role in detecting unauthorized or malicious activities.
Telemetry refers to the automated collection, transmission, and analysis of data from build and deployment processes. By gathering metrics, logs, and events from CI/CD tools and infrastructure, AppSec teams gain insights into system health, performance, and potential bottlenecks.
Observability in a software context is the ability to understand the internal state of a system through metrics, logs, traces, and other external outputs. Rather than merely monitoring CI/CD processes, DevOps and AppSec practitioners — via a holistic view of the software delivery process — are equipped to understand data and act on diagnostic insights.
Telemetry provides the raw data streams and metrics collected, transmitted, and received from systems, while observability is about the actionable insights derived from the data.

Coined by Lockheed Martin, an attack kill chain provides a framework for understanding the sequence of actions an attacker takes to penetrate and exploit a network. By dissecting attacks into phases, security teams can identify and counteract threats at each phase. The traditional kill chain includes seven stages:

  1. Reconnaissance
  2. Weaponization
  3. Delivery
  4. Exploitation
  5. Installation
  6. Command and control (C2)
  7. Actions on objectives

In the context of cloud security, understanding the kill chain allows AppSec teams to implement targeted defenses, detect intrusions earlier in the attack process, and effectively respond to curtail damage.

An advanced persistent threat (APT) is a prolonged, targeted cyberattack in which an intruder gains access to a network and remains undetected for an extended period. APT actors, often state-sponsored or well-funded groups, pursue specific agendas, such as espionage or data theft. DevSecOps teams combat APTs by employing continuous monitoring, threat intelligence, and proactive defense strategies.

TTP stands for tactics, techniques, and procedures, which, together, describe the patterns of activities or methods associated with a threat or groups of threats. Understanding TTPs is vital for identifying and countering advanced persistent threats (APTs).

  • Technique refers to the means used to carry out an objective, such as a specific type of malware or exploitation.
  • Tactic defines the broader strategic objective, like privilege escalation or data exfiltration.
  • Procedures detail the sequence of actions or steps taken by adversaries to accomplish their objectives.

By analyzing TTPs, security professionals can predict threat actor behavior, enhance detection mechanisms, and develop more effective defense strategies.