What Is Actually True About Open Weights

Strip away the week, the letter, the bill, and the factions. Four things are true about open-weight models, and each one is inconvenient for somebody.

One: release is irreversible, and this has no analogue

Once weights are published they cannot be withdrawn. Safeguards can be stripped off by anyone with a GPU. Copies can be downloaded, mirrored, and run on private systems that no one monitors. There is no patch channel, no revocation list, no kill switch, no expiry.

The UK AI Security Institute states it without hedging: "Once open-weight models are released, these options are lost permanently: safeguards can be removed, and copies can be downloaded, redistributed, and run on private systems beyond monitoring. For models with dangerous capabilities — including highly cyber-capable models — open weight release therefore creates a persistent and irreversible risk of misuse."

This is not an argument against open weights. It is a structural fact that any argument about open weights has to accommodate. It is also the one property that has no good analogue in ordinary software: a vulnerable library gets a patched version and a deprecation notice; a set of weights with a dangerous capability gets nothing, forever.

Two: the provenance infrastructure is not ready

Here is the state of the ecosystem as of mid-2026, and it is worse than most practitioners assume.

Pickle is still everywhere. Python's pickle format executes arbitrary code by design during deserialization — Python's own documentation says never to unpickle data from an untrusted source. Hugging Face built safetensors specifically to eliminate this: a format with no execution engine, just typed tensor data. And yet the Hub still carries on the order of 1.3 million pickle-format files against roughly 900,000 safetensors. The safe format exists, is well-supported, and is still in the minority.

The scanner is not a guarantee. PickleScan, the main open-source tool for detecting malicious pickles, took three separate CVSS-9.3 bypasses disclosed in December 2025. A scan that passes is evidence, not proof.

Signing exists and is barely used. Sigstore and the Linux Foundation's Model Signing project provide the machinery for keyless signing and verifiable provenance. Adoption is voluntary and remains a minority practice. For most models you download, there is nothing to verify against.

Compromise is inherited. A backdoored base model or a tampered tokenizer propagates into every fine-tune built on top of it. Checking the provenance of the model you are downloading is insufficient if you have not checked the provenance of what it was built from.

And the vector in this very incident was a dataset. Not a model. The poisoned artefact was a dataset configuration that triggered code execution in a processing pipeline. Everyone's mental model of AI supply-chain risk is focused on weights; the actual attack came through data.

Three: this incident does not support the restrictionist case

Trace the causal chain and check where open weights appear.

The attacking model was closed — a frontier model from a US lab, with its safety training deliberately disabled by its own creators. The escape route was a commercial proxy product. The staging base was somebody's misconfigured demo. The entry vector was a dataset, not a model. The stolen goods were benchmark answers. And the only tool that successfully analysed the attack afterwards was an open-weight model running on the victim's own hardware.

Restricting open-weight distribution would not have prevented a single step of this intrusion. It would have removed the tool that solved it.

Anyone citing this incident as an argument for restricting open weights has not read the incident.

Four: this incident does not settle the open case either

The letter's central claim is that broad access to capability helps defenders more than attackers. This incident provides exactly one data point in favour — the forensics — and does not test the general proposition at all.

Amodei's counter-argument is not answered by anything that happened in July. He argues that the offence-defence balance varies by domain, and that in biology particularly the attacker may hold a structural advantage: a sufficiently capable model might compress weaponisation to days using widely available materials, while defence is a multi-year operational undertaking even in the best case. Whether or not you find that specific claim persuasive, the logical point stands — the offence-defence balance is an empirical question with a different answer in each domain, and asserting a universal answer in either direction is a claim nobody currently has the evidence to make.

Which is precisely why his proposal — test capability before release, rather than deciding in advance — is the position hardest to argue against on principle. The objections to it are practical: who tests, to what standard, with what appeal, and whether a testing regime becomes a licensing regime that only incumbents can afford. Those objections are serious. They are also objections about implementation rather than about the idea.

What this means for you, concretely

If you download and run open weights — and if you run a model locally, you do — the policy fight is mostly noise and the hygiene is entirely real.

Prefer safetensors. Refuse pickle-format weights from any source you would not be willing to run as root, because that is the permission you are granting. Pin to a specific commit hash rather than a moving tag, so that a compromised re-upload does not silently reach you on your next pull. Verify signatures where they exist and write down the absence where they do not, so that your risk is recorded rather than merely unexamined. Check what the model was fine-tuned from. And extend every bit of this suspicion to datasets, because in the one large-scale incident we have, the dataset was the weapon.

None of that requires a position on the policy question. All of it is worth doing regardless of who wins.