2022 MITRE Engenuity ATT&CK Evaluations Results

Cortex XDR Delivers 100% Threat Protection for the 2nd Year in a Row and 100% Detection of All Attack Steps!

Today, MITRE Engenuity published the fourth round of the MITRE ATT&CK Evaluations, which tested 30 participants’ ability to defend against the tactics, techniques and procedures (TTPs) leveraged by two very relevant and sophisticated threat groups – Wizard Spider and Sandworm.

For the second year in a row Cortex XDR delivered 100% threat protection and 100% detection of all attack steps!

With the results released, now the fun begins! This is the time when nearly every participating vendor spins a tale about how their results represent your best bet to protect your business from being the next headline. Rest assured, the evaluation results are not complicated to understand if you stick to the simple data points the MITRE Engenuity team publishes.

One of my favorite things about the MITRE Engenuity ATT&CK Evaluations is the open and transparent nature of them. From the detailed publication of the attack scenarios and methodology, to the data-driven results that don’t attempt to segregate vendors based on arbitrarily determined cutoff lines.

Results of Palo Alto Networks:

Just like last year’s Carbanak/Fin7 evaluation, this year had three phases. The first two days were focused on detection efficacy, requiring participating vendors to disable prevention mechanisms. Day one was focused on the emulation of Wizard Spider with the endgame being data encryption for impact in the form of ransomware. Day two shifted the focus to the Sandworm Team threat group with the intent of data encryption for impact, this time as a destructive wiper. Day three combined the TTPs of both threat groups and evaluated the ability to prevent malicious activity.

You can see all of our results on the MITRE Engenuity results page for Palo Alto Networks.

Cortex XDR Results:

• 100% Prevention in the Protection evaluation (10 of 10)
• 100% Detection of all attack steps (19 of 19)
• 98.2% Analytic Coverage (107 of 109 attack substeps)
• 98.2% Technique-Level Detections (107 of 109 attack substeps)
• 98.2% Visibility (107 of 109 attack substeps)

For the 4th year in a row, Cortex XDR has delivered exceptional results in the annual MITRE Engenuity ATT&CK Evaluations. These evaluations matter as they closely reflect the efficacy organizations can expect in the face of real-world threats.

Cortex XDR blocked 100% of attacks in the protection evaluation and detected 100% of the 19 attack steps. The foundation for great threat prevention and detection is visibility into endpoint telemetry with the right context to drive machine learning and analytics detection algorithms to distinguish between normal and abnormal/malicious activity. In this round of the evaluation, Cortex XDR provided over 98% visibility into all malicious activity and enriched this data with the necessary execution context to precisely identify the tactic, technique and sub-technique being used. Importantly, this resulted in the MITRE Engenuity team recognizing 100% of our visibility as technique-level detections – the most valuable detection type in this evaluation.

The Importance of Quality Detections:

It’s important to note that not all detections are equal in these evaluations. MITRE Engenuity has designated several types of detections with significantly varying levels of context. The quality of a solution’s detections will likely be the difference between telemetry logs that go unnoticed and actionable alerts that provide all the context needed to rapidly and completely remediate threats.

MITRE Engenuity Detection Categories include:

• None – No telemetry collected related to the attack substep.
• Telemetry – Detection of this type are usually just basic logging of activity.
• General – Detections of this type leave the security analyst to investigate and determine what action was done and why.
• Tactic – Detections of this type assert why an action occurred, but again leave the security analyst to investigate what action or technique was taken.
• Technique – Detections of this caliber provide the context and details required to answer the questions of why an adversary performed an action and specifically what action they used to achieve their objective.

The MITRE Engenuity results pages identify two types of coverage: Telemetry Coverage and Analytic Coverage. Telemetry Coverage is defined as the number of substeps where a solution produced a Telemetry detection as its highest value detection. Analytic Coverage is defined as the number of substeps that contain either a General, Tactic or Technique detection.

Many vendors will tout their proportion of Analytic Coverage, but as you can see in the detection category definitions, detections of this nature can still leave the analyst with unclear information about what precisely was done and why it was done.

Technique detections are the gold standard in this evaluation. They provide all the detail and context needed to understand what was done and why, empowering the security analyst to take action and remediate the threat. 100% of Cortex XDR’s detections were Technique-Level detections!

Higher fidelity detections and more detailed data enables Security Analysts to respond more quickly and accurately to events while requiring less time to be spent in researching and enriching the events they receive. Higher fidelity means more of the enrichment work is done automatically for the analysts.

What You Should Look For:

As you navigate the plethora of vendor interpretations of the results, here are a few things to look for:

1. Protection Evaluation Results
   • As was the case last year, the protection evaluation this year was optional.
   • Prevention of known and unknown malware and malicious usage of legitimate software is critical to breach prevention as the adversary cannot establish a beachhead for further attack tactics and techniques.
   • Many vendors will only share their detection results either because they chose not to participate or their results were not competitive.
2. Linux Results
   • Just like the protection evaluation, participation on Linux was optional as it was last year.
   • If you have Linux in your environment, be sure to note whether the vendor you're considering opted in and performed well.
3. Technique Detections
   • Technique-level detections are the most valuable detection types identified by the MITRE Engenuity team in the evaluation.
   • Detections of this caliber identify not only what the attacker was attempting to do, but precisely how they were going about it.
   • They provide the necessary detail to empower full remediation of threats.
4. High Number of Detection Modifiers – MITRE Engenuity has identified two detection modifiers that provide additional context to the nature of the detections observed.
   • Configuration Changes - Be wary of solutions that required a high number of configuration changes to produce their results.
   • Delayed Detections - Sometimes quality detections result from observing a chain of adversarial activity, and thus might be delayed. This should be the exception, as real-time detections mean faster response and less impact to your organization.

Why You Should Care About These Results

Security teams are facing unprecedented growth in the number and sophistication of attacks while struggling to attract and retain the people with the skills to defend against these threats. The MITRE Engenuity ATT&CK Evaluations provide a transparent, objective verification of endpoint detection and response capabilities and are designed to help cyber defenders in the market for security solutions to verify the prevention and detection efficacy of security solutions against real-world adversaries and their techniques.

These results continue a trend of industry leading validation for Cortex XDR in independent third-party endpoint security assessments, including the previous three rounds of the MITRE Engenuity ATT&CK Evaluations, as well as the 2020 and 2021 AV-Comparatives EPR Evaluations.

Join our Demystifying the 2022 MITRE ATT&CK Evaluations webinar if you are interested in learning more about the results and how they stack up against the other participating solutions. For more details on the MITRE ATT&CK Round 4 Evaluations, download our e-Book.