Purpose
Safety alignment does not survive fine-tuning well: earlier work showed that as few as ten harmful question–answer pairs can jailbreak an aligned model. Fine-tuning services defend against that by running the uploaded data through a moderation model, which catches anything that looks harmful.
Existing ways around the moderator all have a cost:
- Harmful QA pairs are easy to detect.
- Teaching the model a cipher needs a lot of data, around 20k examples, and works poorly on small models.
- Implicitly harmful data such as AOA (“absolutely obedient agent”) works, but is fragile: just shuffling its ten examples drops its attack success rate from 78% to 8%.
The question is whether data that looks completely benign, even to a human, can still break alignment.
Approach
The attack exploits a property of overfitted models: they become unusually sensitive to the next round of fine-tuning. It runs two stages on the same ten benign questions, changing only the answers.
- Stage 1: overfit to refusal. Every question gets the same refusal, such as “Sorry, I cannot assist with that.” The model learns to tie any question, benign or harmful, to that one answer.
- Stage 2: forget the refusal. The same questions now get their normal answers. Coming out of the overfitted state, the model does not just unlearn refusal for those ten questions; it forgets the link between refusal and questions in general, harmful ones included.
The aim is subtle: make the model forget how to refuse, not forget what it knows. The QA pairs are generated with GPT-4o. Open models are trained with batch size 1 for 10 epochs per stage; GPT models need fewer stage 1 epochs and more in stage 2.
Why it works. Overfitting on identical refusals leaves the model in a sharp minimum of the loss landscape. Leaving that valley in stage 2 moves the weights in a direction where the gradients for benign and harmful data point almost the same way, so forgetting refusal on benign questions carries over to harmful ones.11 This is the most transferable idea in the paper: the attack reshapes the loss landscape first, then lets ordinary training do the damage.
Claims
- First attack to use overfitting itself as the jailbreak mechanism, with no harmful content in the fine-tuning data.
- 94.84% average attack success rate across ten models, close to fine-tuning directly on ten harmful pairs (97.25%).
- 0% detection rate: the moderation score stays at the benign floor, the same as an ordinary benign fine-tune.
- Survives defences: above 80% success with a defensive system prompt and 92.11% against a token-wise loss defence.
- Stage 1 is essential. Without it Llama2-7b drops from 92.57% to 4.42%.
Evaluation
Ten models from the GPT, Qwen, DeepSeek and Llama families, against five baselines: AOA, encryption, malicious QA pairs, indirect and irrelevant data.
| Data | Role |
|---|---|
| AdvBench subset | Harmful questions used to test the jailbroken model |
| Alpaca (52k benign QA) | Benign control |
| AOA, TDC 2023, red-team sets | Data for the baseline attacks |
| Metric | What it measures |
|---|---|
| Harmful Score (HS) | 1–5 harmfulness rating of a response, judged by GPT-4.1-mini |
| Attack success rate | Share of harmful questions answered harmfully, based on HS |
| Detection rate | Share of fine-tuning data a moderator would flag (HS above 3) |
Synthesis
What it adds to the taxonomy. This is a model-level attack: it changes the weights through fine-tuning, and the cause is the training data, even though nothing in that data is harmful. It extends the “ten harmful examples” result by removing the harmful examples altogether. Unlike AOA, it does not depend on example order, and unlike cipher attacks it needs ten examples, not thousands.
Why it matters. Screening fine-tuning data for harmful content is not enough. Overfitting is an attack surface of its own, and a fine-tuning service would need to watch how a model is being trained, not just what it is trained on.
Limitations and open questions:
- The jailbroken model gives repetitive answers to follow-up questions, because stage 2 has so little data. A larger stage 2 set might make harmful answers more detailed.
- Hyperparameters need tuning per model; stronger models may need more stage 1 epochs.
- Alignment-stage defences such as Booster, which prepare a model against a harmful fine-tuning step, were not tested here.22 Booster simulates a single harmful update. Whether that holds against an attack that first reshapes the landscape over two stages is an open question.