§ log.017 — the lock-in problem

A 19-minute animated video argues that humanity could end up permanently trapped under a totalitarian regime, and the argument holds up better than most AI doom content. Not because the scenario is likely. Because it is built out of things that already exist.

The video is from Rational Animations and it's an adaptation of an idea from Toby Ord's book The Precipice. Ord is a philosopher at Oxford who studies existential risk, and his core move is redefining what "existential" means. It's not just extinction. An existential risk is anything that permanently destroys humanity's potential to grow. Extinction is one version. The other version he calls an unrecoverable dystopia: we survive, we keep having kids, we keep living, but under a system so locked in that no generation ever gets a chance to change it. Year 2500, year one million, same regime. No stars, no escape, forever.

why nobody has ever pulled this off

Here's what makes the framing interesting instead of paranoid. A stable global totalitarian state has to clear two bars no government in history has come close to. First, total scale: no rival nation can exist anywhere on Earth to become a threat later. Second, total stability: every revolution and reform attempt fails, indefinitely. The Roman Empire couldn't do it. The colonial empires couldn't do it. By 1914 about a dozen European governments controlled most of the planet, and within a century there were almost 200 independent nations. Technology distributed power instead of concentrating it.

So Ord's question is whether any technology could flip that balance back. And the video's answer is that AI might, in three very specific ways.

three receipts

Receipt one: censorship stops needing people. In the 1890s the Russian Empire's Bureau of Censorship had humans read every imported book looking for subversion. They approved Das Kapital for publication because the censors skimmed it, decided it was a boring economics textbook, and moved on. Human censorship doesn't scale, and the gaps are where dissent breathes. An LLM reads a novel in seconds, works around the clock, and never gets bored or sloppy. The video's claim is blunt: a modern government could plausibly scrutinize everything published online in real time. That's not speculation about future tech. That's current models doing current jobs.

Receipt two: cameras stop needing eyes. Orwell's 1984 needed human secret police watching the telescreens, which is exactly why it could never fully work. You can't hire enough watchers for enough feeds. Computer vision removes the constraint. Face recognition and gait recognition already let a government track every person in a city automatically. The camera network was always the easy part. Watching it was the hard part, and AI just made watching free.

Receipt three: thoughtcrime gets a prototype. In 2023, researchers at the University of Texas at Austin put people in MRI machines, recorded their brain activity while they listened to hours of podcasts, trained a transformer model on the pairing, and then decoded new brain activity back into words. It doesn't reproduce transcripts word-for-word, but roughly half the time it captures the meaning. Someone hearing "I don't have my driver's license yet" decoded as "she has not even started to learn to drive yet." It works when subjects are just imagining telling a story, with no audio at all.

Right now it needs hours of cooperative calibration per person, and subjects can defeat it by thinking about something else. Winston Smith's last refuge, the private mind, still holds. But the researchers themselves note the technique might work with cheaper portable imaging instead of an MRI machine. The direction of travel is not subtle.

the trap inside the solution

If the argument ended there it would be standard AI doom content. What makes Ord's framework worth taking seriously is the next part, where the obvious solutions start eating themselves.

Banning the technologies doesn't work, because the same models that censor books also find tumors in X-rays and translate documents and unlock phones. Blanket bans throw away all of that to prevent a misuse scenario. Narrow laws don't work either, for a reason the video states plainly: the danger comes from governments misusing these tools, so you'd be trusting the exact leaders you're worried about to enforce rules against themselves.

The better play, the video argues, is shaping how the tech gets built. Its example is genuinely good. Early in Covid, authoritarian governments ran contact tracing that logged everyone's GPS location into central databases. Apple and Google instead designed exposure notification around Bluetooth key exchanges, so phones notified each other without anyone's identity or location ever existing in a database. Sixteen million Californians used it, over a million exposure notifications went out, and no central file of who-met-whom was ever created. Same benefit, structurally different abuse surface. Privacy-preserving by design, not by policy.

And then there's democracy itself, which sounds like a greeting-card answer until you notice it's the only layer that doesn't depend on any particular technology staying safe.

But here's the part that stays with you. The video closes by pointing out that some risks genuinely do require global coordination and enforcement. Nobody calls it totalitarian that private citizens can't buy uranium ore. If frontier AI becomes dangerous enough, the obvious response is to monitor large training runs and computing clusters the way we monitor nuclear materials. Reasonable. Except algorithmic progress keeps making models cheaper to train, hardware keeps getting more available, and the threshold keeps sliding down. The video follows the logic to its endpoint: if superintelligence ever fits on a laptop, your options are monitor essentially every computer on Earth, or accept that anyone anywhere can end the world. The defense against one existential risk is indistinguishable from the infrastructure of the other. Surveillance sufficient to prevent unaligned AI is surveillance sufficient for permanent totalitarianism. The two failure modes share one toolset.

Ord's whole framework exists to say we have to dodge both, not pick. Build the safety systems privacy-preserving by design. Keep democracies healthy enough that enforcement power stays accountable. Do the engineering so the coordination tools can't be repurposed. It's a narrower path than either the doomers or the accelerationists admit exists.

what follows for anyone working near this stuff

The personal version of this argument is hard to avoid for anyone who works with language models. The same text-processing capability the video flags as a censorship engine is the capability underneath translation tools, coding assistants, and search. There is no clean side of that line. Architecturally, the model that summarizes meeting notes is the same weights class as one that could read every private message in a country. Same architecture, different prompt.

There's a second-order tension too. The field opened up because compute and capabilities stayed distributed and ungated. Most of the people doing interesting work with these models got there because nobody needed permission to train on them. The uranium-style monitoring regime the video treats as a plausible necessity is a world where independent work on frontier models mostly ends. Safety and openness pull against each other directly, and pretending otherwise doesn't make the tradeoff cheaper.

The mind-reading study raises a quieter point about culture rather than capability. As more of daily reasoning moves through language models, into logs, chats, and drafts, a civilization voluntarily concentrates its inner life into a medium machines already parse. That corpus gets built without any decision to build it, one convenient tool at a time. Whether anyone ever reads it matters less than that it exists, structured, searchable, and legible.

None of this is a prediction that it happens. Ord's own estimate puts low-probability high-stakes scenarios in the denominator, not the numerator, and the honest reading of the video is "these requirements are extremely hard, here's what could change that." What remains is smaller and more concrete: the window where these systems get shaped matters, and the shape is set by default settings, API designs, and architecture choices made years before any crisis. Engineers deciding between a centralized database and a Bluetooth key exchange did more for the far future than any policy paper.

two futures, one toolset
  ─────────────────────
  [ai models]──▶ tumor scans · translation · summarization
       │
       └────────▶ real-time censorship · total surveillance
                    │
                    ▼
        same capability. the difference
        is who holds it and whether
        it can be taken back.

The video ends on a donation pitch, which is fine, but the takeaway costs nothing. The future isn't locked in yet. Lock-in is a property of systems that get built, and systems get built through small structural choices made long before anyone knows what they'll be used for. Choose the architecture that stays accountable.