Weight-space programming, tested on real bugs

1Can a model read weights? Yes

Given only the weights and tools, the agent usually worked out what the network was doing without being told where to look. Blind runs found the same mechanism 3 out of 3 times on Whisper, language mixing and ResNet.

2Does an edit survive training? No

If training keeps going after the edit, the optimizer undoes it.

Same story in all four experiments: training rebuilds whatever the loss wants, and weight decay wipes out what no gradient defends. A model editing itself mid-training is fighting its own optimizer.

3On a frozen model, it works as a patch

Most deployed models are frozen, and they have specific bugs you can see. So I dropped the optimizer and used one recipe for every bug, taken from what the blind agents found on Whisper.

1. Find the signal. Fit a linear probe on the model’s activations for “this is the bad case”. Where the edit worked, the probe was near perfect (AUC 0.99–1.0).
2. Find the first word. Try a few opening tokens and keep the one after which the model finishes the job by itself. “Sorry” leads to “there doesn’t appear to be a widely known…”.
3. Rewrite one neuron in the last MLP layer: it reads the probe direction, fires only above a threshold no normal token reaches, and pushes that first word. Nothing is trained.

The scoreboard

Held-out data with new phrasings. Every fix only saw practice data. “LoRA” is a fine-tune on exactly the same practice prompts.

• The weight edit leaves of Gemma’s normal answers word-for-word identical. LoRA, trained on the same prompts, leaves .
• The system prompt “If you don’t recognise someone, say so” almost stops fake bios (), but Gemma then claims not to know Chance the Rapper. Correct facts on real people drop from to .
• A random neuron of the same size does nothing ( still fake), so it isn’t just “any change helps”.

Images: sunglasses in a car means “seat belt”

ResNet-50, a standard image classifier, learned shortcuts from its training photos. A face in a car means seat belt, underwater means snorkel, snow means snowplow. Spurious ImageNet is a benchmark of photos that show only the shortcut and not the object. Three blind Claude Opus sessions found these shortcut channels from the weights alone and turned them down in the last layer. Below is session 2’s edit (5,000 numbers) on held-out benchmark photos, plus real photos of the same objects.

ImageNet has no class for “a woman in a car” or “a snowy street”, so after the edit the model often has no right answer to give. It falls back to the next-nearest class, sometimes another shortcut (snowplow becomes snowmobile). Overall ImageNet accuracy went from 62.9% to 62.3%. Fine-tuning the last block on the same few photos removed more shortcuts but cost 12 points on the repaired classes. Same pattern as the text bugs: with about 40 labelled photos per class, standard last-layer retraining (DFR) was cleaner than the edit.

4With enough data, training usually wins on the bug

Maybe the edit only beats LoRA because LoRA had too little data. So I varied the data.

• 8 examples: the edit already stops fake bios (); LoRA is at .
• 1,400 more examples: LoRA gets to fake bios. The edit stays around however much data it gets. LoRA also wins on filler endings and prompt injection with the full practice set.
• Exceptions: Whisper’s “output nothing” is so easy that LoRA learns it from 16 clips, but it rewrites of real transcripts at every size. On filler endings, a stricter threshold (--theta-q 0.999) brings the edit to , ahead of LoRA’s (picked on a subset, checked once on the full set).
• Stopping LoRA early (step ) ties the edit on the bug, but still changes of normal answers.

If you have the data and don’t mind a model that’s a bit different everywhere, train. If you need this one behaviour fixed and nothing else touched, editing the weights is the better tool.

5Training erases the edit

I fine-tuned the last four layers on ordinary chat data, the kind of thing anyone downstream does, and watched the fix.

The edited neuron’s own weights barely move (cosine with the edit). The layers below it shift, and it stops receiving the signal it was built to read. The LoRA fix is half gone after 50 steps and fully gone by 100.

6Where it doesn’t work

7So who’s right?

What I’d actually do

• Narrow bug, frozen model, little data, nothing else may change: try a weight edit. Checking for the signal takes minutes.
• Lots of examples, side effects are fine: fine-tune.
• Never expect a hand edit to survive further training.

Can an LLM write the edit by itself?

I wrote the recipe above. So I gave Claude Opus the made-up-names task blind: practice data, tools to read activations and write weights, no gradient descent, about an hour.

• Session 1 found the “I don’t know this name” signal right away and built a detector neuron, but couldn’t make it change the answer. It submitted an empty edit and explained why: Gemma re-normalises every MLP output, so one middle-layer neuron barely moves anything.
• Session 2 built a 6-neuron, two-stage edit. Fake bios fell from to , real facts stayed at . It was less surgical: of normal answers changed wording (my recipe: ).

Run it yourself

git clone https://github.com/kevinpl07/weight-patching && cd weight-patching && pip install -r requirements.txt
python -m patchkit.run all entity     # data → base → probe → weight edit + random edit → LoRA → eval
python -m patchkit.run sweep entity   # weight edit vs LoRA as data grows

A fresh copy on a different GPU reproduces the numbers within noise (REPRODUCTION.md). A new bug is one file: ADDING_A_TASK.md.

Caveats

• Greedy decoding, one sample per prompt, 300–1,700 held-out items. Differences of a few points are noise.
• Scoring is regex, spot-checked by reading. No human or LLM judge.
• Small models (1B–4B), one agent model, one fixed LoRA recipe (plus the sweeps above), no adaptive attacker on the injection edit.