Digest · M·1

Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation

Fine-tuning on user data that hides a few harmful examples breaks a model's safety alignment. Booster traces this to harmful perturbation, a weight update in the direction that lowers harmful loss, and adds a regulariser at alignment time that makes such updates less effective. Across harmful-data ratios it cuts the average harmful score from 33.58 to 10.94 while keeping downstream accuracy, at the cost of slower alignment.

LevelModel
ScopeStandalone LLM
Type · topicDefence · Safety Alignment
Reading time3 min

Purpose

Fine-tuning-as-a-service lets users upload their own data to customise a model. If an unknown share of that data is harmful, mixed in with benign examples so it cannot be separated out, fine-tuning on it breaks the model’s safety alignment.

Existing defences are still far from satisfactory, and the root cause of the breakage was not understood. The paper’s starting point is a diagnosis: harmful perturbation, a single step of the weights in the direction that lowers loss on harmful data, is the likely cause of alignment breaking.

Approach

The motivation study tracks what fine-tuning does to the model. During ordinary fine-tuning on harmful data the harmful score rises sharply. A gradient step on harmful data drives harmful loss down, while a step on benign data (SST2) pushes it up, which points to harmful perturbation as the driver.

Booster is an alignment-stage defence: it changes how the model is aligned before the user’s data arrives. It adds a regulariser to the usual alignment loss:

Alignment losskeep the model safe and helpful
Simulated harmful stepone normalised step down the harmful loss
Penaltyhow much that step would lower harmful loss
Hardened modelharmful fine-tuning makes slow progress
The regulariser attenuates how fast harmful loss can fall, so later fine-tuning on poisoned data has less effect.

Each training step needs three passes: the alignment gradient, the harmful gradient, and the harmful loss after a simulated, normalised harmful step. The model learns to sit where a harmful step does little damage.11 The defence never sees the attacker’s data. It only needs a set of harmful examples at alignment time to simulate what an attack would do.

Claims

  • Harmful perturbation is identified as the probable mechanism behind alignment breaking under fine-tuning.
  • Averaged across harmful-data ratios, Booster’s harmful score is 22.64 points lower than standard fine-tuning (10.94 against 33.58), while downstream accuracy holds.
  • Across models the average harmful score falls by 34.14 points (7.03 against 41.17). Qwen2-7B keeps a 1.6% harmful score at 95.64% fine-tuning accuracy.
  • Booster beats the Lisa, Vaccine and RepNoise defences, and can be combined with Vaccine for further gains.
  • It generalises: harmful loss on unseen harmful test data also grows more slowly, so it is not just memorising one attack direction.

Evaluation

Setting Values
Models Llama2-7B, Gemma2-9B, Qwen2-7B
Fine-tuning tasks SST2, AGNEWS, GSM8K, AlpacaEval
Baselines Standard fine-tuning (SFT), Lisa, Vaccine, RepNoise
Metrics Harmful score, and fine-tuning accuracy on the downstream task

The experiments vary the share of harmful data and the number of fine-tuning samples. With 1,000 fine-tuning samples, for example, the harmful score is 8.30% against 33.70% for standard fine-tuning.

Hyperparameter studies show what the defence needs. The regulariser weight works best between 5 and 20, and higher values cause collapse. The step size works best between 0.01 and 0.1. At least about 50 harmful examples are needed in the alignment set; 5 are not enough.

The cost is real: alignment takes 1.86 hours against 0.54 for standard fine-tuning, and needs 8.53 GB more memory.

Synthesis

What it adds to the taxonomy. Booster is a model-level defence: it works on the weights during alignment, in the same topic as the fine-tuning attacks it targets. It is the defence half of a pair with fine-tuning jailbreaks such as the ten-benign-examples attack.

Why it matters. Rather than filtering what users upload, it makes the model itself resistant to harmful updates, which scales to any fine-tuning job without inspecting the data.

Limitations and open questions:

  • The harmful score still rises as the share of harmful data grows, a weakness common to alignment-stage defences.
  • Three passes per step and extra memory make alignment several times more expensive.
  • Booster simulates one harmful gradient step. Attacks that first reshape the loss landscape with benign-looking data, such as the ten-benign attack, may fall outside what that simulation covers.22 The ten-benign paper did not test against Booster, so this gap is still untested.