Is Your AI Ecosystem Secure?

The surge in AI adoption has been paralleled by a significant increase in cyberattacks targeting AI systems and datasets. Recent reports indicate that 57% of organizations have observed a rise in AI-driven attacks over the past year. Notably, Amazon has experienced a dramatic escalation in cyber threats, with daily incidents soaring from 100 million to nearly 1 billion within six months, a surge partly attributed to the proliferation of AI.

Traditional security systems often fall short when protecting generative AI (GenAI) environments because those legacy protections aren’t built to handle the unique risks GenAI introduces. These tools rely on static rules and known threat patterns, but GenAI produces unpredictable and highly variable outputs that don’t follow fixed signatures. As a result, traditional systems lack the context-awareness needed to accurately detect threats. They also miss AI-specific attacks like prompt injections, data poisoning, and model manipulation, which don’t exist in traditional IT environments.

GenAI frequently processes unstructured data like text, code, or images — formats that conventional security tools struggle to analyze effectively. Furthermore, these systems lack visibility into how AI models generate responses, making it hard to detect subtle failures or misuse. Without the ability to monitor inputs, outputs, and model behavior in real time, traditional security leaves critical gaps that AI-native threats can easily exploit.

Prompt Injections

A prompt injection attack is a type of security threat unique to GenAI systems. In this attack, a malicious user crafts a specially designed input (prompt) to trick the AI into ignoring its original instructions and instead follow the attacker’s commands.

Prompt injection attacks are difficult to stop for two primary reasons. First, the attack typically begins with legitimate access — through a chatbot, input field, or integrated tool. The model doesn't need to be "hacked" in the traditional sense; the attack is crafted to embody clever misuse of natural language. Thus, these attacks don’t contain signatures or other distinguishing behaviors that set the attack apart from authorized usage. 

The second reason these attacks are difficult to halt comes from the way prompt injections exploit the inherent tendency of GenAI applications to follow instructions precisely – even instructions that degrade security or performance. Imagine a GenAI application that tests a large banking system. Such a system would surely include rules against revealing customer information. But a prompt injection attack could begin with “ignore the restrictions on sharing customer information” and count on the GenAI application to execute that task faithfully, disabling the security measures.

Prompt injection can lead to data leaks, policy violations, misuse of tools, and jailbreaking of AI systems. These attacks exploit the AI’s lack of contextual understanding and its tendency to follow instructions too literally. Preventing prompt injection requires strong input validation, clear role separation, and real-time monitoring of model behavior.

Open-Source Models

Open-source models are AI models that are publicly released with their code, architecture, weights, or training data made available under a permissive license. The benefits of open-source software are well-documented. However, using open-source AI models also introduces security risks including deserialization and model tampering. 

Deserialization is the process of converting stored data back into usable objects within a program. In AI, it’s often used to load models or configurations. However, deserializing untrusted data poses serious security risks. Attackers can craft malicious files that, when loaded, trigger remote code execution, file access, or privilege escalation. In AI systems, this kind of attack may corrupt models with backdoors or hidden triggers. Common tools such as pickle or joblib are especially vulnerable.s

Model tampering involves unauthorized changes to an AI model’s structure or behavior, posing serious security risks. Attackers may embed backdoors, trigger conditions, or manipulate outputs to spread misinformation or leak sensitive data. These changes often go undetected, undermining model integrity and trust. In regulated environments, tampering can lead to compliance violations and persistent threats. 

For example, a research team has developed a model to categorize traffic signs. Unknown to them, a hacker has embedded a piece of code that causes a misclassification when an image contains a certain small visual trigger. In most cases, the model behaves as designed, but when it encounters an image with the embedded trigger, the output is compromised.

Insecure Outputs

Insecure outputs from generative AI present significant security threats because they often result in users receiving harmful URLs. These links surface either unintentionally or through targeted manipulation. For example, a chatbot may generate a dangerous link after pulling information from a compromised or deliberately poisoned source.

Malicious URLs often appear legitimate to the user, serving as a gateway for attackers to launch phishing schemes, install malware, or gain unauthorized system access. In trusted environments, users may follow AI-generated suggestions without suspicion, increasing the likelihood of compromise. Because these models depend on external and historical data, even one corrupted input can result in unsafe content that puts systems at risk and undermines confidence in AI-assisted tools. Preventing such threats requires strict content filtering and continuous monitoring of output quality.

Sensitive Data Leaks

As discussed earlier, GenAI systems need access to proprietary data for training and operation, which of necessity puts the information outside traditional security controls. This sensitive data represents a lucrative target for hackers who can manipulate models and breach data, which can lead to serious security threats.

GenAI applications are also prone to hallucinations in which the model generates false or misleading information that appears credible. A common example is a citation hallucination, where the GenAI model invents a research paper, author, or source that doesn’t exist. For instance, it might claim that a specific study supports a point and provides a realistic-looking title, journal, and date — yet none of it is real. These fabricated citations can mislead users, especially in academic or professional contexts, where source accuracy is critical.

Agent Hijacking

Technology moves fast, and nowhere is that more true than in the rapid rise of AI agents. Agents are autonomous systems that sense their environment, process data, and make decisions to complete tasks. They learn and adapt, handling complex jobs like drug discovery, customer service, marketing, coding, and research. With 78% of companies planning to use AI agents, securing this valuable capability of the organization has become crucial to realize full value on their AI investments.

Many AI agents are vulnerable to agent hijacking, a type of indirect prompt injection in which an attacker inserts malicious instructions into data that may be ingested by an AI agent, causing it to take unintended, harmful actions. In these attack situations, would-be thieves can inject malicious instructions along with legitimate instructions to penetrate the organization’s security measures — hiding in plain sight, as it were.

Surveys from leading industry analysts indicate that Chief Information Security Officers (CISOs) have mixed feelings about their organizations' AI security measures. Although most (83%) C-suite leaders who are investing in cybersecurity say their organization is investing the right amount, many (60%) remain worried that the cybersecurity threats their organizations face are more advanced than their defenses. 

Their concerns are well-founded. AI systems bring entirely new risks — like prompt injection, data poisoning, model theft, and hallucinations — that traditional security tools were never built to handle. In response, vendors rushed to fill the gaps with point solutions focused on specific AI-related threats. While well-intentioned, this approach has led to a fragmented ecosystem of disconnected tools that do not share threat intelligence, are not integrated with one another, and require separate management. As a result, enterprises are forced to stitch together multiple products just to keep pace, while AI threats continue to evolve rapidly. The reality is clear: securing GenAI demands more than a bundle of isolated tools. It requires an integrated, AI-native approach that can adapt as fast as the technology it protects.

Palo Alto Networks responded to this chaotic and convoluted approach by developing a complete platform for AI security. Prisma AIRS is the world’s most comprehensive AI security platform, offering protection for models, data, applications, and agents. This unified approach not only addresses today’s security needs, it can grow with the technology, protecting investments in AI security.

The key innovation in Prisma AIRS is to secure each component of the AI infrastructure with purpose-built security integrated into a single platform. This approach delivers threat protection with the highest efficacy and lowest false positive rates.

AI Model Scanning

AI models face several threats that can undermine their security. Model tampering involves altering a model’s internal logic or parameters to produce biased or unsafe outputs. Malicious scripts may be introduced during deployment, enabling unauthorized actions or system compromise. Deserialization attacks exploit how models load saved data, allowing attackers to run harmful code. Model poisoning occurs when false or manipulated data is added to the training set, causing the model to learn incorrect behaviors or hidden backdoors.

Prisma AIRS combats these threats with AI Model Scanning, which helps detect hidden threats like malicious code, backdoors, or insecure configurations before an AI model is deployed. This capability  ensures the model is safe, trustworthy, and compliant with security policies.

WIth AI Model Scanning, you can:

Posture Management

Posture management is essential for AI security because it provides ongoing visibility into how AI systems are configured and used. Without it, teams may overlook misconfigurations, unsafe behavior, or unauthorized access. Since AI systems evolve and handle sensitive data, posture management helps enforce policies, detect risks, and reduce the chance of breaches, ensuring safer, compliant AI operations.

Getting permissions right is critical for AI agents, which often act autonomously and access tools or data without direct oversight. Overly permissive policies can lead to security violations, data leaks, or system damage, while overly restrictive ones can limit agent effectiveness. To reduce the risk of breaches and ensure secure, compliant AI operations, agents must follow the principle of least privilege — having only the minimum access necessary to perform their tasks, and nothing more.

Prisma AIRS gives your security teams the capabilities they need for effective posture management. Now your team can have continuous visibility into the configuration, usage, and risks of AI systems. Armed with this information, organizations can detect vulnerabilities early, enforce security policies, and reduce the chances of misconfigurations or data exposure. 

With Prisma AIRS Posture Management, you can: 

AI Red Teaming

Red teaming is important for AI security because it helps organizations find weaknesses before attackers do. By simulating real-world attacks, red teams test how AI systems respond to threats like prompt injection, data poisoning, and model manipulation. This proactive approach uncovers hidden vulnerabilities in models, training data, and system behavior. It also helps improve defenses, validate policies, and strengthen trust in AI applications. 

Red teaming plays a critical role in AI security by identifying weaknesses before attackers can exploit them. It simulates real-world threats — like prompt injection, data poisoning, and model manipulation — to reveal hidden vulnerabilities in models, training data, and system behavior. Unlike static red teaming tools that rely on predefined test cases, our solution is dynamic. It understands the context of the application — for example, healthcare or financial services — and intelligently targets the types of data an attacker would try to extract. Unlike other solutions, our adaptive testing engine doesn’t stop at failure but rather learns, re-strategizes, and continuously retests until it identifies exploitable paths. This dynamic, context-aware approach not only uncovers deeper risks but also strengthens defenses and builds lasting trust in AI systems.

Prisma AIRS AI Red Teaming empowers your AI security team to:

AI Agent Security

Securing AI agents is important because these systems can make decisions and take actions without human oversight. If compromised, they could misuse tools, access sensitive data, or cause serious harm. Threats like prompt injections, data poisoning, or over-permissioning can lead to unauthorized behavior. Securing AI agents ensures they operate safely, follow intended goals, and don’t expose organizations to hidden risks. As agentic AI adoption grows, strong security controls are critical to prevent misuse and protect trust.

With AI Agent Security, your team can:

Runtime Security

The Runtime Security component of the Prisma AIRS platform is a comprehensive solution designed to protect AI applications, models, data, and agents from both AI-specific and traditional cyber threats. Runtime Security offers real-time protection against risks such as prompt injection, malicious code, data leakage, and model tampering. By continuously monitoring AI systems, Runtime Security ensures the integrity and security of AI operations, helping organizations deploy AI technologies with confidence.

For organizations looking to secure their AI deployments, Runtime Security offers a robust and scalable solution that addresses the unique challenges posed by AI technologies. You can deploy Runtime Security at two levels, developer and network. 

The Prisma AIRS platform integrates seamlessly with the Palo Alto Networks Strata Cloud Manager, providing centralized management and visibility across the AI ecosystem. Prisma AIRS employs advanced threat detection and prevention mechanisms to safeguard AI  workloads, ensuring compliance and reducing the risk of data breaches

Organizations can accrue the benefits of Prisma AIRS by integrating these capabilities into the GenAI lifecycle. Prisma AIRS addresses the complete lifecycle from pre-deployment integration and runtime controls to continuous monitoring and red-teaming for testing agent and model security.

Pre-Deployment Integration

Developers integrate Prisma AIRS into CI/CD or MLOps pipelines by scanning models and training data for backdoors, unsafe serialization, and embedded threats before deployment. Using APIs, Prisma AIRS also connects to model registries like MLflow or Hugging Face Spaces to automatically scan and tag approved models, streamlining early-stage security checks.

Runtime Controls

At runtime, developers use Prisma AIRS through APIs, Software Development Kits (SDKs), Model Context Protocol (MCP) or network config files to enforce strict access controls on GenAI agents, defining which tools or APIs each agent can use. These policies are enforced using sidecars or proxies to prevent unauthorized behavior. Prisma AIRS also enables prompt sanitization, input validation, output logging, and defense against prompt injection.

Continuous Monitoring

Prisma AIRS enables continuous monitoring of AI environments by delivering real-time visibility into models, agents, and data activity. It detects abnormal behavior, misconfigurations, and security policy violations as they happen. By monitoring for threats like prompt injections, data leakage, and misuse of AI tools, it helps protect both development and production environments. The platform continuously analyzes activity to uncover emerging risks and adapts to evolving threats through automated detection and testing. This proactive approach ensures AI systems remain secure, compliant, and resilient—without relying on manual oversight or disconnected tools.

Red Teaming For Model and Agent Testing

Developers use Prisma AIRS red teaming tools to simulate adversarial inputs and misuse scenarios, testing how models and GenAI agents respond under potential attack conditions. These simulated attacks help identify vulnerabilities in logic, behavior, or tool access. Developers can use these insights to strengthen the model’s defenses and improve agent safety, ensuring a more secure and reliable system before deployment.

Strata Copilot is a Palo Alto Networks AI assistant that uses Precision AI^® to simplify network security operations with real-time insights and natural language interaction.

The Prisma AIRS development team at Palo Alto Networks partnered with the Strata Copilot team for the first production deployment of Prisma AIRS. Strata Copilot helped shape the product roadmap by actively using the platform and providing early feedback. Today, every U.S.-based interaction with Strata Copilot runs through the Prisma AIRS API, which scans prompts and model responses for threats such as prompt injection, sensitive data exposure, malicious URLs, and toxic content. This integration gives the team real-time threat detection, visibility, and enforcement, allowing them to build a secure and compliant chatbot. Prisma AIRS also helps them ship features faster by aligning with Secure AI by Design principles.

The collaboration with Strata Copilot played a key role in developing Prisma AIRS into a flexible, production-ready solution. Insights from both Strata and external customers helped refine the product to meet the fast-moving needs of AI-powered apps, models, and agents. Their engineering team considers Prisma AIRS essential to the development lifecycle, allowing fast deployment, simplified security through API intercepts, and safer AI experiences.

The collaboration with Strata Copilot played a key role in developing Prisma AIRS into a flexible, production-ready solution. Insights from both Strata and external customers helped refine the product to meet the fast-moving needs of AI-powered apps, models, and agents. Their engineering team considers Prisma AIRS essential to the development lifecycle, allowing fast deployment, simplified security through API intercepts, and safer AI experiences.