🔍 Read the full analysis: Weight-Space AI Editing At Home: A Zero-Legal-Score Experiment on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A home experiment used directional ablation to reduce looping in the local model Ornith 1.5 35B-A3B. On the same setup, it completed 10 of 10 published Lexassay tasks instead of 4, but scored zero on legal accuracy; the answers included 55 wrong or invented citations. The author says results are preliminary, with one run per setting and an independent AI review that found errors in the analysis.
A home experiment reported by ThorstenMeyerAI.com found that editing the weights of the local model Ornith 1.5 35B-A3B helped it finish more legal benchmark tasks, but did not improve its legal score. On the same experimental setup, the model completed 10 of 10 published Lexassay tasks, up from 4, while its score remained zero; the edited answers contained 55 wrong or invented citations, according to the report.
The experiment ran from Wednesday evening to Thursday morning and was inspired by MIT researchers’ work on directly editing model weights. It did not reproduce that method. Instead, a Claude Opus 5.5 session wrote code to run Ornith, measure an internal direction associated with looping, and remove that direction at half strength from the output weights of 27 of the model’s 40 layers. The report says there was no additional training and no new legal data.
The comparison used 10 published tasks from Lexassay, a benchmark covering German, U.S. and European Union law. The edited model met 7 of 50 checked criteria, compared with 3 before editing, but its legal score remained zero. The report counted 55 incorrect or fabricated citations among the edited answers. These figures describe the published task set, not the benchmark’s five hidden tasks.
The author also reports that changing the sampling settings to those recommended in Ornith’s model card, without editing weights, allowed the unchanged model to complete 9 of 10 tasks on its original setup. That result is not directly comparable with the 4-of-10 edited-experiment baseline: the report says the two runs used different local setups. In a separate published run, the unchanged model completed 8 of 10 tasks.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
Finishing Tasks Is Not Legal Accuracy
The result illustrates a practical distinction for people testing local AI tools: generating a complete answer does not mean generating a correct one. Reducing repetitive output can make a system easier to use and evaluate, but the experiment found no measurable improvement in the model’s legal score. Its citation errors are a particular warning in legal work, where an answer can sound complete while pointing to authorities that are wrong or do not exist.
The findings do not establish that weight edits cannot improve legal performance. They show that this particular intervention, on this model and this small published task set, improved completion more than substance. The report itself frames the technique as addressing an obstacle that may hide an existing capability, rather than adding knowledge the model does not have. Readers should treat the results as an experiment, not as evidence that the model is safe or suitable for legal use. The article says it is not legal advice.
Products Worth Considering
Logitech MX Creative Console with 9 Customizable Keys, Graphite
Apple MacBook Pro 14.2-inch with M6 chip, Space Black
Apple MacBook Air 15-inch (2026) with M5 Chip, 16GB Memory, 1TB SSD, and Liquid Retina Display – Sky Blue
As an affiliate, we earn on qualifying purchases.
MIT Research Prompted a Home Test
The experiment followed a report on MIT CSAIL researchers Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, who described what they called “weight-space programming”: using one AI to read and edit another model’s weights. In a Qwen example cited by the source, the researchers reported that a model’s looping was linked to a direction in its internal signals; after an edit, its maths score rose from 7.9% to 48.1%. The source stresses that the home test used a different, older technique and should not be mistaken for a replication of that work.
Directional ablation works by running a model, recording internal signals, calculating a direction associated with a behavior, and projecting that direction out of some weights. The source notes that related single-direction edits have been studied before, including work on removing refusals from chat models. Ornith was selected because it showed looping in the benchmark, while other candidate models either appeared to lack relevant knowledge without looping or could not be tested under comparable conditions.
Lexassay asks models the same questions across German, U.S. and EU law, without internet access or tools, and checks legal citations against primary sources. The report says 10 tasks are public and five remain hidden. All figures in the account refer to the public tasks, and the author says the answers, grades, citation checks and method are available on Lexassay.
“The model finishes is not the answer is right.”
— Thorsten Meyer, author of the experiment report
Small Sample, Uneven Test Conditions
The results are limited to one run per setting and 10 published tasks. The report says the loop detector changed during the experiment, and the original and edited runs used two local setups that are not directly comparable. It also says the unchanged model completed 8 of 10 tasks in a published run but only 4 of 10 on the setup used for the direct comparison. The separate 9-of-10 result with model-card sampling adds another condition rather than resolving those differences.
Grading also presents a potential conflict: Claude carried out the edit, and Claude agents performed blind grading. The report says OpenAI’s GPT-6 Astra, running in Codex, independently reviewed code, figures and claims and found real faults. That review is a check, not a second experiment. The source does not detail the specific faults in the supplied material, nor does it establish how the method would perform across more models, repeated runs or the five hidden tasks.
Public Runs Enable Further Checks
The next step available to readers is to inspect the published Lexassay answers, grades, citation checks and experiment method, then test whether the pattern holds under repeated runs and consistent hardware and sampling settings. Results on the five hidden tasks would provide a separate check against tuning to the public questions, but the source does not say when those tasks will be scored or released.
For now, the reported outcome is narrow: directional ablation reduced looping enough for Ornith to complete more tasks on one setup, while its legal score stayed at zero. Further testing would be needed to determine whether the completion gain is reliable and whether any different intervention can improve factual legal performance.
Key Questions
What changed after the model’s weights were edited?
On the same experimental setup and 10 published Lexassay tasks, Ornith completed 10 tasks instead of 4. Its legal score remained zero, and the report counted 55 wrong or invented citations in the edited answers.
Was this the same method used by the MIT researchers?
No. The home test used directional ablation: code ran the model, measured a direction associated with looping and removed it from some weights. The MIT work described in the source involved an AI reasoning directly about another model’s weights.
Did the edit improve the model’s legal knowledge?
The reported benchmark score did not improve: it stayed at zero. The edited model met 7 of 50 checked criteria, up from 3, but the report also counted 55 wrong or invented citations. The results do not show that the model is reliable for legal work.
How strong is the evidence?
The account reports one run per setting across 10 public tasks, with a loop detector that changed during the experiment and local setups that were not directly comparable. An independent AI review found faults, according to the author, so the result should be treated as preliminary.
Source: ThorstenMeyerAI.com
