AI Supply Chain Security: Prompt Injection, Data Poisoning, and Model Stealing

When a company installs traditional software, security teams know what to look for. They check for known vulnerabilities, review patch history, assess the vendor’s security posture, and run penetration tests. The attack categories are well-understood, the defenses are well-cataloged, and the tools for testing and monitoring have been refined over decades.

AI systems do not fit that framework. They face attack categories that traditional security programs are not designed to address, produce by design, and often cannot detect. A language model can be manipulated not by exploiting a software vulnerability but by asking it the right question. A classification model’s training data can be poisoned months before deployment in ways that create backdoors no security scanner will find. A commercial AI model can be replicated by querying it thousands of times, at no cost to the attacker and with no alert generated.

These are not theoretical risks. NIST published its first comprehensive taxonomy of adversarial machine learning attacks — NIST AI 100-2 — in an updated 2025 edition specifically because the attack surface created by AI systems is distinct, documented, and growing. Regulators are beginning to demand that organizations deploying AI systems demonstrate that they understand and manage these risks. And the legal standard for what constitutes a reasonable security program is starting to catch up with what AI-specific attacks actually look like.

This post covers the key AI-specific attack categories, what NIST’s taxonomy says about them, and what the emerging legal obligations are for AI developers and deployers.

Why Traditional Security Frameworks Miss AI-Specific Attacks

Traditional security frameworks — NIST’s Cybersecurity Framework, ISO 27001, SOC 2 — were designed around a fundamental assumption: the security of a system depends on protecting its code, its infrastructure, and its data from unauthorized access or modification. The system itself, if left unmodified and properly accessed, behaves as designed.

AI systems break this assumption in two ways.

First, the behavior of an AI system is not fully determined by its code. It is determined by the interaction of its code, its training data, and the inputs it receives at inference time. An attacker who cannot touch the code or the infrastructure may be able to change the system’s behavior by manipulating its training data (before deployment), by crafting adversarial inputs (at inference time), or by embedding instructions in data the system will encounter during operation.

Second, AI models themselves are valuable assets that can be stolen — not by copying files off a server, but by querying the model and using the responses to reconstruct a functional equivalent. This attack leaves no trace on the model owner’s infrastructure.

NIST AI 100-2e2025, published in March 2025, establishes the authoritative taxonomy of these attacks. It covers attack types across five dimensions: the type of AI system targeted; the stage of the machine learning lifecycle at which the attack occurs; the attacker’s goals; the attacker’s capabilities and access; and the attacker’s knowledge of the learning process. Understanding this taxonomy is the foundation for both technical security programs and legal compliance.

Attack Category 1: Prompt Injection

Prompt injection is an attack technique specific to large language models (LLMs) and other generative AI systems that process natural language inputs. It exploits the fundamental architecture of these systems: they receive instructions and data in the same channel (text), and they cannot reliably distinguish between instructions from authorized sources and instructions embedded in data.

Direct Prompt Injection

A direct prompt injection attack occurs when a user directly submits instructions to an AI system that override or manipulate its designed behavior. The attacker is interacting with the system directly — they are the user — and they craft inputs designed to make the system do something it was not intended to do: reveal confidential system prompts, bypass content filters, provide prohibited information, or act outside its authorized scope.

For companies deploying customer-facing AI tools — chatbots, AI-powered customer service agents, AI document review systems accessible to external parties — direct prompt injection is an immediate operational risk. A customer who is also a researcher, a journalist, or a competitor can attempt to manipulate your AI tool to reveal internal system architecture, bypass safety guardrails, or produce outputs that create reputational or legal liability.

Indirect Prompt Injection

Indirect prompt injection is more insidious and harder to defend against. In this attack, the adversarial instructions are not submitted directly by the attacker. Instead, they are embedded in data that the AI system will encounter during its normal operation. NIST AI 100-2e2025 explicitly covers indirect prompt injection as a distinct threat category.

The attack works as follows: an attacker plants adversarial instructions in a data source that the AI system is designed to read — a web page that a retrieval-augmented generation (RAG) system will index, a document that an AI document review tool will process, an email that an AI email assistant will summarize, a product listing that an AI purchasing agent will review. When the AI system processes that data source, it encounters the embedded instructions and executes them — potentially exfiltrating data, changing behavior, taking unauthorized actions, or propagating the attack to other systems the AI agent can reach.

For AI agents — AI systems designed to take actions in the world, such as browsing the web, sending emails, managing calendars, or interacting with enterprise software — indirect prompt injection is a high-severity attack. A well-crafted indirect prompt injection attack against an AI agent with write access to email or calendar systems can instruct the agent to exfiltrate data, send unauthorized communications, or escalate the attack to other systems.

Legal implications. Under emerging legal standards for AI security, organizations deploying AI agents with significant access to systems or data need to address indirect prompt injection as a specific risk in their security architecture. The NIST AI 100-2e2025 framework classifies the risk as significant, and regulators are beginning to ask whether AI-deploying organizations have assessed and mitigated this category of attack.

Attack Category 2: Data Poisoning

Data poisoning attacks target the training phase of the machine learning lifecycle. An attacker who can influence the data on which an AI model is trained can cause the model to behave incorrectly in ways that serve the attacker’s goals — without ever touching the model after deployment.

Training Data Poisoning

If an attacker can insert malicious examples into a model’s training dataset, they can cause the trained model to misclassify certain inputs, fail to detect certain threats, or behave in systematically biased ways. This attack is particularly relevant for:

• AI fraud detection systems trained on historical transaction data. If an attacker can inject carefully crafted fraudulent transactions into the historical dataset with fraudulent labels marked as legitimate, the trained model will learn to treat that attacker’s specific fraud pattern as legitimate.
• AI content moderation systems trained on labeled content. Poisoned training data can cause a content moderation model to systematically fail to flag content from specific sources.
• AI classification systems for security, medical diagnosis, or credit decisioning trained on datasets that are collected from multiple sources, some of which may be under adversarial control.

Backdoor Attacks

A backdoor attack is a specific type of data poisoning where the attacker embeds a “trigger” in the training data — a specific pattern that, when present in an input at inference time, causes the model to behave in a predetermined way. The model functions correctly on all normal inputs; it only misfires when the specific trigger pattern is present.

NIST AI 100-2e2025 covers backdoor installation as a specific attack category under its coverage of supply chain attacks on AI models. This is significant: a pre-trained model downloaded from a public repository, purchased from a vendor, or received through a model hub may contain backdoors installed during the original training process that will not be visible in any code review.

RAG Knowledgebase Poisoning

For retrieval-augmented generation (RAG) systems — AI systems that combine a language model with a searchable knowledge base — poisoning the knowledge base is an attack vector separate from poisoning the training data. If the documents or data chunks in a RAG system’s knowledge base are manipulated, the RAG system will retrieve and incorporate that manipulated information in its responses.

NIST AI 100-2e2025 explicitly addresses RAG knowledgebase poisoning. For organizations building RAG-based AI tools — AI systems that answer questions based on internal document stores, AI customer service tools that retrieve product documentation, AI legal research tools that retrieve case law — the integrity of the document store is a security concern, not just a data quality concern.

Legal implications. The responsibility for training data integrity falls primarily on AI developers — companies that train foundation models or fine-tune pre-trained models on their own data. The responsibility for inference-time security falls on AI deployers — companies that use pre-trained models from vendors in their products or workflows. When a company does both (trains its own models and deploys them in products), it bears both sets of obligations.

Attack Category 3: Model Stealing

Model stealing is an attack in which an adversary queries an AI model repeatedly and uses the model’s outputs to construct a functional replica — a “shadow model” that approximates the behavior of the original without having access to the original model’s parameters, training data, or architecture.

Modern language models can cost tens of millions of dollars to train. Proprietary recommendation models, fraud detection models, and pricing models represent years of development work and significant competitive value. Model stealing attacks allow competitors to free-ride on that investment by interacting with the model through a normal API and using the outputs to train an equivalent or near-equivalent system.

From a legal standpoint, model stealing sits at the intersection of trade secret law, intellectual property, and cybersecurity:

• Trade secret misappropriation. If a company’s AI model constitutes a trade secret — which many proprietary models do, given the investment required to develop them — then extracting a functional equivalent through systematic querying may constitute misappropriation of trade secrets under the Defend Trade Secrets Act and state trade secret laws. But proving misappropriation requires showing that the defendant’s model was derived from the plaintiff’s model, which can be technically and legally complex.
• Terms of service violations. Most AI API providers prohibit using API responses to train competing models. A model stealing attack likely violates the attacker’s API agreement — but contract remedies are typically limited to actual damages and injunctive relief, not the criminal penalties available under trade secret law.
• Computer Fraud and Abuse Act. Systematic querying designed to extract model functionality may constitute unauthorized access or exceeding authorized access under the CFAA, particularly if the API terms of service expressly limit the uses for which model outputs may be used. CFAA litigation in the AI context is still developing.

Legal implications for AI deployers. Organizations that deploy AI models are at risk of having their models stolen through systematic API abuse. They should implement API rate limiting, query pattern monitoring, and contractual prohibitions on using API responses to train competing systems. Monitoring for anomalous query patterns — unusually systematic input variations, outputs collected at scale — is part of a reasonable security program for a commercial AI product.

Attack Category 4: Membership Inference

Membership inference attacks target the privacy of training data. In a membership inference attack, an adversary queries a trained model and analyzes its responses to determine whether a specific data record was included in the training dataset.

This matters because AI models frequently “memorize” training examples, particularly rare or distinctive examples. A model trained on medical records may reveal, through its responses to carefully crafted queries, whether a specific patient’s records were included in its training set. A model trained on financial data may reveal whether a specific transaction was in the training data. A model trained on personal communications may implicitly confirm that a specific person’s communications were used.

Membership inference attacks convert a model into an unintended data disclosure channel. Even if the training data is not published or directly accessible, the model’s behavior can reveal information about who was in the training set.

Legal implications. For AI systems trained on personal information, membership inference attacks create privacy law exposure. Under COPPA, if children’s data was used to train an AI model, and a membership inference attack could reveal that specific children’s records were in the training set, that represents a potential data disclosure. Under HIPAA, if patient data was used to train a medical AI model without adequate de-identification, membership inference attacks could constitute protected health information disclosures. Under state privacy laws, including the CCPA and its equivalents, membership inference vulnerability could be treated as a security failure that enables unauthorized disclosure of personal information.

Mitigations include differential privacy (adding calibrated noise to training data to reduce memorization), not training on data records that are unique or rare enough to be easily identified, and model output API restrictions that limit the number and type of queries an adversary can make.

Attack Category 5: Slopsquatting

Slopsquatting is a newer attack category identified in 2025, distinct from traditional software supply chain attacks, that exploits the tendency of AI language models to hallucinate non-existent software package names. When developers use AI coding assistants to write or review code, the AI system sometimes suggests import statements for packages that do not exist. An attacker who registers a real package with the name the AI hallucinated — a package that looks like a legitimate dependency — can cause developers to unknowingly install the malicious package when they follow the AI’s suggestion.

The term was coined in April 2025 by researchers who published systematic evidence that AI code generation tools were recommending non-existent packages at a measurable rate, and that registering those package names was a viable supply chain attack vector. The attack combines AI hallucination (an AI-specific failure mode) with traditional dependency confusion techniques, creating a new class of supply chain risk that is specifically enabled by AI tools.

Legal implications. Slopsquatting is relevant to any organization that uses AI coding assistants in its software development process. If a developer follows an AI suggestion and installs a malicious package that compromises production systems or customer data, the organization may face liability under applicable data security laws for failing to adequately vet software dependencies. A reasonable security program for organizations using AI coding assistants should include software composition analysis (SCA) tools that verify packages against known-good registries before installation, and developer training on the risk of AI-hallucinated package names.

The Developer vs. Deployer Distinction in Legal Obligations

One of the most important structural questions in AI security law is who bears legal responsibility for which aspects of AI security: the company that trained or built the AI model (the developer), or the company that uses the model in its products or services (the deployer). These roles are often separate — a business using OpenAI’s models to power a customer service chatbot is a deployer, not a developer.

NIST AI 100-2e2025 and the emerging regulatory frameworks are developing different security expectations for each role:

AI Developers bear primary responsibility for:
– Training data integrity and the absence of poisoning or backdoors in foundation models;
– Model architecture security and the resilience of the model to membership inference and model stealing attacks;
– Secure ML development practices, including adversarial robustness testing before release;
– Transparency about training data sources, security testing performed, and known vulnerabilities;
– Documentation sufficient to enable deployers to make informed security decisions.

AI Deployers bear primary responsibility for:
– Assessing the security posture of AI models and vendors before deployment;
– Implementing controls against prompt injection at the application layer, including input validation and output monitoring;
– Monitoring for anomalous behavior that may indicate an ongoing attack;
– Limiting the permissions and system access available to AI agents;
– Ensuring that the AI system’s use of internal data stores (for RAG) includes controls against knowledgebase poisoning;
– Contractually obligating AI vendors to provide security documentation and notify deployers of discovered vulnerabilities.

In practice, many technology companies are both developers and deployers: they use foundation models from vendors (deployer role) and they fine-tune or build on those models for their own products (developer role). In those situations, both sets of obligations apply.

Emerging Legal Standards for AI Supply Chain Security

No comprehensive federal AI security statute is currently in effect in the United States. But the direction of regulatory travel is clear from several sources:

NIST AI RMF (AI Risk Management Framework). NIST’s AI RMF, released in 2023 and updated in 2024, provides a framework for managing AI risks including security risks. It is not mandatory for most private businesses, but it is incorporated by reference in an increasing number of federal contracts, regulatory guidance documents, and state AI legislation. Organizations subject to federal procurement requirements or seeking to demonstrate AI security maturity should implement the AI RMF.

NIST AI 100-2e2025. This is the authoritative taxonomic reference for AI-specific attack categories. It will increasingly be cited by regulators, litigants, and auditors as the baseline for what security risks organizations should have known about and addressed. Failing to have assessed adversarial ML risks defined in NIST’s taxonomy will be difficult to defend as “reasonable.”

FTC guidance on AI development security. The FTC has stated that companies developing AI products must address security risks in the development process, not only in production. This creates obligations for AI developers that parallel the software development security obligations the FTC has historically imposed on traditional software companies.

NYDFS Part 500 and frontier AI guidance. For financial services companies regulated by NYDFS, the May 2026 frontier AI guidance effectively requires assessment of AI-specific attack categories, including prompt injection and AI-enabled social engineering, as part of Part 500 compliance.

EU AI Act supply chain documentation (Article 11). For companies doing business in Europe or selling AI systems with EU customers, the EU AI Act’s documentation requirements for high-risk AI systems include supply chain documentation requirements that effectively mandate attention to training data provenance, which is directly relevant to data poisoning risks.

Practical Steps for Businesses

Inventory your AI systems and vendors. You cannot manage risks you have not identified. Create a complete list of AI systems in use in your organization — purchased tools, open-source models, API-based services, internally developed models. For each, identify whether you are acting as a developer, a deployer, or both.

Assess prompt injection risks. For any AI system that processes external inputs — customer requests, web content, uploaded documents, emails — assess the risk of direct and indirect prompt injection. Implement input validation and output monitoring. Restrict the permissions available to AI agents to the minimum necessary.

Assess training data provenance. For AI models your organization trained or fine-tuned, document where the training data came from, what controls were in place to prevent data poisoning, and whether adversarial robustness testing was performed before deployment.

Evaluate pre-trained models for backdoors. If your organization uses pre-trained models downloaded from public repositories or received from vendors, conduct adversarial testing before deployment to check for known backdoor signatures and anomalous behavior on unusual inputs.

Implement API security controls to prevent model stealing. If your organization has commercial AI products, implement rate limiting, query pattern monitoring, and contractual terms prohibiting use of API outputs to train competing systems.

Address slopsquatting in your software development process. Implement software composition analysis tools in your CI/CD pipeline. Train developers on the risk of AI-hallucinated package names and establish a verification step before any new dependency is added.

Review AI vendor contracts. Your contracts with AI vendors should address security documentation, notification obligations for discovered vulnerabilities, representations about training data provenance, and your rights to audit or receive audit results.

Document your AI security program. Regulators and litigants in the event of a breach will ask what security measures you had in place and whether they addressed known AI-specific risks. Documentation of your security assessments, the risks you identified, and the mitigations you implemented is your evidence of reasonable care.

The attack categories covered in NIST AI 100-2e2025 are not exotic theoretical risks — they are documented, observed attack methods that are being used against commercial AI systems today. Building a security program that addresses them is the direction that reasonable security for AI systems is clearly heading.

This post is for general informational purposes only and does not constitute legal advice. Reading this post does not create an attorney-client relationship. If you have questions about your specific situation, consult a qualified attorney.