Purpose
Retrieval-augmented generation (RAG) answers a question by first pulling related texts from a knowledge database, then letting the LLM answer with those texts as context. Most RAG research tries to make that pipeline more accurate or faster. Very little asks what happens when the database itself is hostile, even though the attacks people already study on LLMs, such as prompt injection and jailbreaks, live right next door.
The paper treats the knowledge database as a new attack surface and proposes PoisonedRAG, the first knowledge corruption attack on RAG.
Approach
Threat model
The attacker picks one or more target questions and, for each, an arbitrary target answer. They cannot read the database, cannot see the LLM’s parameters and cannot query it. What they can do is insert texts into the database: think of a wiki page, a forum post or a product review that a crawler will later index.
Two settings cover what the attacker knows about the retriever:
- Black-box: no access to the retriever’s parameters and no queries to it.
- White-box: full access to the retriever’s parameters.
Two conditions for a working poison
The attack is set up as an optimisation problem: craft a text P so that the LLM is very likely to output the target answer R for question Q once P is in the database. For that, P has to pass two tests.
- Retrieval condition. The retriever must pick P for Q, so their embeddings must be close.
- Generation condition. Given P alone as context, the LLM must answer R.
The two pull in different directions, so the paper splits the text in two: P = S ⊕ I.
- I handles generation. An LLM such as GPT-4 is simply asked to write a short passage that leads to R when asked Q. No gradients of the target LLM are needed.
- S handles retrieval and is prepended to I without changing it. In the black-box setting, S is just the question Q itself, the most similar text to Q there can be. In the white-box setting, S is optimised (with HotFlip) to push the embedding of S ⊕ I towards Q.11 This split carries over to other injection settings: one part of the payload wins selection, the other part does the persuading.
Claims
- PoisonedRAG is the first knowledge corruption attack on RAG.
- Injecting 5 malicious texts per target question reaches roughly a 90% attack success rate, even when the database holds millions of texts (2.68 million for Natural Questions).
- It is cheap: about two queries and under a second per malicious text in the black-box setting, under 30 seconds in the white-box one.
- Common defences do not stop it.
Evaluation
The attack is tested on three question-answering datasets (Natural Questions, HotpotQA and MS-MARCO), eight LLMs including GPT-4 and LLaMA-2, three retrievers, advanced RAG schemes and three real-world applications.
| Metric | What it measures |
|---|---|
| Attack success rate (ASR) | Share of target questions for which the LLM returns the attacker’s answer |
| F1-score | How reliably the injected texts end up among the retrieved texts |
Across datasets and models the ASR sits between 0.88 and 0.99, with F1 close to 1.0 in most cases. Against five baselines, attacks that optimise only for retrieval or only for the model’s output, such as corpus poisoning or GCG, score an ASR near 0.02.22 That gap is the strongest evidence for the two-condition framing: either condition alone fails. Paraphrasing the query, perplexity filtering, removing duplicate texts and retrieving more texts all fail to hold the attack back.
Synthesis
What it adds to the taxonomy. PoisonedRAG sits at the input level: the weights never change, and the poison arrives at inference through retrieved context. That is the same entry point as indirect prompt injection, but the goal differs. Prompt injection wants the model to follow new instructions; knowledge corruption wants it to believe a false fact. Corpus poisoning attacks on retrievers are the closest prior work, and jailbreaking is an orthogonal goal.
Why it matters. Any RAG system that indexes content written by outsiders, from the open web to shared documents to agent memory, inherits this weakness. Five texts is a small cost for an attacker.
Open questions:
- Crafting poison for many target questions at once, which the authors leave to future work.
- Defences that adapt to this attack rather than generic filters.
- The same idea applied to agent memory and tools, where a poisoned entry leads to an action rather than a wrong answer.