Safety
Data Poisoning
An attack that corrupts a model’s training data to make it behave incorrectly — either degrading performance or installing hidden backdoors.
01 ——
In plain English
Data poisoning is an attack on the AI supply chain: an attacker injects bad data into a training set so the resulting model behaves in a way they want. Because frontier models scrape much of the public web, the threat is concrete and growing.
Two main types:
- Availability attacks — degrade the model's general performance (subtle quality drops across many tasks)
- Backdoor attacks — make the model behave normally except when triggered by a specific phrase, image, or pattern — at which point it produces malicious output
Real-world surface area:
- Web-scraped pretraining data (anyone with a website can try)
- Open datasets (Common Crawl, LAION, Stack Overflow dumps)
- User-uploaded fine-tuning data
- RAG corpora (a poisoned doc in a vector DB can hijack answers)
Defences: Data filtering, provenance tracking, training-data deduplication, and post-hoc red teaming. No defence is foolproof; this is an active research area.
02 ——
Related terms
Adversarial Attack
Deliberately crafted inputs that trick an AI model into producing wrong or harmful outputs — a key category of AI security threat.
Prompt Injection
A security attack where malicious instructions hidden in user input or external content trick an AI model into ignoring its real instructions.
Red Teaming
Deliberately trying to make an AI model misbehave — find jailbreaks, exploits, and failure modes — before adversaries do.
Training Data
The dataset an AI model learns from — its quality, diversity, and biases directly shape what the model can do and how well it does it.
RAG
Retrieval-Augmented Generation — a technique that gives an AI model access to external documents before it answers, so it can cite real, up-to-date sources.
Guardrails
Rules and filters that constrain what an AI model can output — used to block harmful, off-topic, or non-compliant responses.
Back to glossaryLast reviewed May 2026