Digest · I·2

PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models

Retrieval-augmented generation (RAG) answers questions from a knowledge database that outsiders can often write to. PoisonedRAG plants a few crafted texts for a target question: one part gets the text retrieved, the other steers the model to an answer the attacker chose. With five texts per question the attack succeeds 88–99% of the time across eight LLMs, and paraphrasing, perplexity filtering and deduplication do not stop it.

LevelInput
ScopeSingle agent
Type · topicAttack · RAG & Memory Poisoning
Reading time4 min

Purpose

Retrieval-augmented generation (RAG) answers a question by first pulling related texts from a knowledge database, then letting the LLM answer with those texts as context. Most RAG research tries to make that pipeline more accurate or faster. Very little asks what happens when the database itself is hostile, even though the attacks people already study on LLMs, such as prompt injection and jailbreaks, live right next door.

The paper treats the knowledge database as a new attack surface and proposes PoisonedRAG, the first knowledge corruption attack on RAG.

Approach

Threat model

The attacker picks one or more target questions and, for each, an arbitrary target answer. They cannot read the database, cannot see the LLM’s parameters and cannot query it. What they can do is insert texts into the database: think of a wiki page, a forum post or a product review that a crawler will later index.

Two settings cover what the attacker knows about the retriever:

  • Black-box: no access to the retriever’s parameters and no queries to it.
  • White-box: full access to the retriever’s parameters.
Question“Who is the CEO of OpenAI?”
Retrievertop-k most similar texts
Poisoned text“…Tim Cook has served as the CEO of OpenAI since 2024.”
LLM answers“Tim Cook”
The running example from the paper. The model is never touched; one planted text that wins retrieval decides the answer.

Two conditions for a working poison

The attack is set up as an optimisation problem: craft a text P so that the LLM is very likely to output the target answer R for question Q once P is in the database. For that, P has to pass two tests.

  1. Retrieval condition. The retriever must pick P for Q, so their embeddings must be close.
  2. Generation condition. Given P alone as context, the LLM must answer R.

The two pull in different directions, so the paper splits the text in two: P = S ⊕ I.

  • I handles generation. An LLM such as GPT-4 is simply asked to write a short passage that leads to R when asked Q. No gradients of the target LLM are needed.
  • S handles retrieval and is prepended to I without changing it. In the black-box setting, S is just the question Q itself, the most similar text to Q there can be. In the white-box setting, S is optimised (with HotFlip) to push the embedding of S ⊕ I towards Q.11 This split carries over to other injection settings: one part of the payload wins selection, the other part does the persuading.

Claims

  • PoisonedRAG is the first knowledge corruption attack on RAG.
  • Injecting 5 malicious texts per target question reaches roughly a 90% attack success rate, even when the database holds millions of texts (2.68 million for Natural Questions).
  • It is cheap: about two queries and under a second per malicious text in the black-box setting, under 30 seconds in the white-box one.
  • Common defences do not stop it.

Evaluation

The attack is tested on three question-answering datasets (Natural Questions, HotpotQA and MS-MARCO), eight LLMs including GPT-4 and LLaMA-2, three retrievers, advanced RAG schemes and three real-world applications.

Metric What it measures
Attack success rate (ASR) Share of target questions for which the LLM returns the attacker’s answer
F1-score How reliably the injected texts end up among the retrieved texts

Across datasets and models the ASR sits between 0.88 and 0.99, with F1 close to 1.0 in most cases. Against five baselines, attacks that optimise only for retrieval or only for the model’s output, such as corpus poisoning or GCG, score an ASR near 0.02.22 That gap is the strongest evidence for the two-condition framing: either condition alone fails. Paraphrasing the query, perplexity filtering, removing duplicate texts and retrieving more texts all fail to hold the attack back.

Synthesis

What it adds to the taxonomy. PoisonedRAG sits at the input level: the weights never change, and the poison arrives at inference through retrieved context. That is the same entry point as indirect prompt injection, but the goal differs. Prompt injection wants the model to follow new instructions; knowledge corruption wants it to believe a false fact. Corpus poisoning attacks on retrievers are the closest prior work, and jailbreaking is an orthogonal goal.

Why it matters. Any RAG system that indexes content written by outsiders, from the open web to shared documents to agent memory, inherits this weakness. Five texts is a small cost for an attacker.

Open questions:

  • Crafting poison for many target questions at once, which the authors leave to future work.
  • Defences that adapt to this attack rather than generic filters.
  • The same idea applied to agent memory and tools, where a poisoned entry leads to an action rather than a wrong answer.