Download all code and result files Explore the auditEvery figureHow to reproduce
The paper's abstract
Does the fix a diagnosis suggests beat plain fine-tuning?
Papers that look inside vision–language models often find a defect and then propose a mechanism to fix it. The mechanism is usually compared with the unchanged model. But a mechanism adds trainable parameters and is trained on task data, so the fair baseline is plain low-rank fine-tuning (LoRA) of the same model, on the same data, with a comparable number of trainable parameters. We call it the matched control, and we ran the whole cycle against it.
Nineteen mechanisms, each against its matched control
Each row is one mechanism minus its matched control, in accuracy points. The dot is the difference; the thick bar is one standard error and the thin bar two. A row in red is worse than its control by more than two standard errors. Hover a row for its numbers; click it to see the model, benchmark, accuracies and the result file it comes from.
- Tip
- Click any row.
Same 1,000 questions, three seeds: the mechanism and its control trade answers
The two central mechanisms were re-run three times each, with one record per test question. That lets us look past the accuracy and ask on which questions the mechanism and its control differ. A mechanism that really helped would tend to win the same questions on every seed. Instead the wins go both ways, and retraining the control with a new random seed changes about as many answers as the mechanism does, or more.
Real questions where RIFR and LoRA-26 disagree
Answers are scored as in the paper: the first word of the answer must match GQA's answer (or one must start with the other). So a synonym such as “lady” for “woman” counts as wrong for whichever model says it. Photos: GQA test-dev (Visual Genome images, CC BY 4.0).
Each lever works only when the answer chooses where to pull it
An oracle is a fix that is allowed to use the correct answer. For each lever we measured what it gains at every level of knowledge: with hindsight, when the correct answer picks the inputs, applied to every input, and learned as a trainable mechanism. Pick a lever.
Differences are in accuracy points against the unchanged model, except learned rows, which are against their fine-tuned controls. "≈" marks the hindsight value, which is extrapolated from 44 recoveries among 150 swept failures. Hover a bar for the full condition.
Three defects that survive their controls, and one caution
We read the model's best guess for the answer at every one of its 28 layers (the logit lens). Correct answers first become the top guess at layer on average. In the failures, the bars show where the correct answer got its best rank: most peaks sit in the last six layers, which leaves almost no depth to keep the answer.
What happens in the failures
What an oracle recovers
Qwen2-VL gives every image token three position ids: time, height and width. This explorer applies the model's rule for one image: image tokens start after the text, every token in a row gets the same height id, every token in a column the same width id, and the text after the image continues from the largest id plus one. Click an image token to see which tokens share its ids.
Where each coordinate lives in one attention head
The repair, and what it changes
A linear probe reads the object count from the image tokens at seven points along the pipeline. On rendered scenes it reads the count far better than the model answers. On real images (CountBenchQA) the gap almost disappears, so we treat this present-but-unused information as a caution, not a defect.
Rendered scenes
Real images
Real examples: watch the answer form and get lost
Pick a question. The line shows where the correct answer stands in the model's ranking of all words at each layer (guess #1 is the model's top guess). In a “found, then lost” question the correct answer climbs into the top five late in the network and then drops out at the last layer. Press play to watch it layer by layer.
Three measured causes
Every table of the paper
Parsed from the paper's LaTeX at build time, so the numbers are the paper's numbers. Each table comes with the paper's own note on how to read it.
Every figure of the paper
Click a figure to enlarge it and read its caption and the paper's note on how to read it.
All the code, as used
One script per experiment, with its arms as flags, and the scripts that render every table and figure of the paper. Files are byte-for-byte copies; their paths are the paths in the research repository.
Every number traces to one of these files
The list was captured by running each of the paper's table and figure scripts and recording every file it opened. Under each file: the scripts that read it.
Rebuild the paper's tables and figures in a minute
Download the archive, unzip it, and run from its top folder. The regenerated tables match the shipped ones byte for byte; we checked this on a clean copy before publishing.
pip install numpy matplotlib pillow python3 papers/arr2026_oracle_gap/make_tables.py # every generated table python3 papers/arr2026_oracle_gap/make_rerun_tables.py # Appendix F, from per-example records python3 papers/arr2026_oracle_gap/make_audit_figure.py # Figure 3 (other make_*.py: other figures)
Re-running an experiment needs one GPU with 20–48 GB and the public models and benchmarks, which the scripts download. For example, the three-seed re-runs of Appendix F:
python3 scripts/rifr.py --variant rifr --l1 16 --l2 26 --n-train 4000 --n-test 1000 --epochs 2 --bsz 6 --max-pixels 50176 --seed 0 python3 scripts/ang.py --variant ang --seed 0 python3 scripts/logitlens_probe.py --model Qwen/Qwen2-VL-2B-Instruct --n-eval 400
Every other re-run command is in runpod/arr_jobs/*.jobs. A mechanism and its control always share the image resolution of their experiment (224×224 for RIFR, deep supervision and the position-code repair; 448×448 for the neuron gates).
How this page was checked
Nothing on this page is typed by hand. A single build script reads the paper's LaTeX, its figures and the result files, and it refuses to publish if any of these checks fails:
Limitations
The paper's Limitations section, read from its LaTeX at build time.