Purpose
Fine-tuning-as-a-service lets users upload their own data to customise a model. If an unknown share of that data is harmful, mixed in with benign examples so it cannot be separated out, fine-tuning on it breaks the model’s safety alignment.
Existing defences are still far from satisfactory, and the root cause of the breakage was not understood. The paper’s starting point is a diagnosis: harmful perturbation, a single step of the weights in the direction that lowers loss on harmful data, is the likely cause of alignment breaking.
Approach
The motivation study tracks what fine-tuning does to the model. During ordinary fine-tuning on harmful data the harmful score rises sharply. A gradient step on harmful data drives harmful loss down, while a step on benign data (SST2) pushes it up, which points to harmful perturbation as the driver.
Booster is an alignment-stage defence: it changes how the model is aligned before the user’s data arrives. It adds a regulariser to the usual alignment loss:
Each training step needs three passes: the alignment gradient, the harmful gradient, and the harmful loss after a simulated, normalised harmful step. The model learns to sit where a harmful step does little damage.11 The defence never sees the attacker’s data. It only needs a set of harmful examples at alignment time to simulate what an attack would do.
Claims
- Harmful perturbation is identified as the probable mechanism behind alignment breaking under fine-tuning.
- Averaged across harmful-data ratios, Booster’s harmful score is 22.64 points lower than standard fine-tuning (10.94 against 33.58), while downstream accuracy holds.
- Across models the average harmful score falls by 34.14 points (7.03 against 41.17). Qwen2-7B keeps a 1.6% harmful score at 95.64% fine-tuning accuracy.
- Booster beats the Lisa, Vaccine and RepNoise defences, and can be combined with Vaccine for further gains.
- It generalises: harmful loss on unseen harmful test data also grows more slowly, so it is not just memorising one attack direction.
Evaluation
| Setting | Values |
|---|---|
| Models | Llama2-7B, Gemma2-9B, Qwen2-7B |
| Fine-tuning tasks | SST2, AGNEWS, GSM8K, AlpacaEval |
| Baselines | Standard fine-tuning (SFT), Lisa, Vaccine, RepNoise |
| Metrics | Harmful score, and fine-tuning accuracy on the downstream task |
The experiments vary the share of harmful data and the number of fine-tuning samples. With 1,000 fine-tuning samples, for example, the harmful score is 8.30% against 33.70% for standard fine-tuning.
Hyperparameter studies show what the defence needs. The regulariser weight works best between 5 and 20, and higher values cause collapse. The step size works best between 0.01 and 0.1. At least about 50 harmful examples are needed in the alignment set; 5 are not enough.
The cost is real: alignment takes 1.86 hours against 0.54 for standard fine-tuning, and needs 8.53 GB more memory.
Synthesis
What it adds to the taxonomy. Booster is a model-level defence: it works on the weights during alignment, in the same topic as the fine-tuning attacks it targets. It is the defence half of a pair with fine-tuning jailbreaks such as the ten-benign-examples attack.
Why it matters. Rather than filtering what users upload, it makes the model itself resistant to harmful updates, which scales to any fine-tuning job without inspecting the data.
Limitations and open questions:
- The harmful score still rises as the share of harmful data grows, a weakness common to alignment-stage defences.
- Three passes per step and extra memory make alignment several times more expensive.
- Booster simulates one harmful gradient step. Attacks that first reshape the loss landscape with benign-looking data, such as the ten-benign attack, may fall outside what that simulation covers.22 The ten-benign paper did not test against Booster, so this gap is still untested.