§03//BLOG
§ log.015 — the missing question
Open source just caught the frontier and KV cache compression keeps breaking the memory wall. The next milestone isn't a smarter model — it's a curious one. Recursive self-improvement won't come from better answers. It'll come from a machine learning to ask questions. That's also what makes it dangerous.
Two days ago Moonshot released Kimi K3 — 2.8 trillion parameters, the largest open-source model ever shipped, trading blows with Claude Fable and GPT-5.6 at the top of the leaderboards. Full weights drop July 27. A year ago the gap between open weights and the frontier was measured in generations. Now it's measured in weeks, and sometimes it points the other direction.
Meanwhile, on the efficiency axis, the KV cache — the thing that makes long-context models expensive to actually run — keeps getting crushed. PolarQuant transforms KV embeddings into polar coordinates and quantizes the angles instead of the vectors, which compresses absurdly well because key states have well-structured patterns under polar transformation. Between that, multi-token prediction drafters, and everything else Google and the Chinese labs have shipped this year, memory is quietly stopping being the wall.
So: capability is commoditizing, and the cost of running capability is collapsing. The obvious question is what's left. What's the next milestone that isn't just a better score on a benchmark?
I think it's recursive self-improvement. And I think the bottleneck isn't intelligence. It's curiosity.
the loop is already closing
Before the philosophy, the receipts. Anthropic published a piece through their new institute — "When AI builds itself" — that's the most honest internal data any lab has ever shared about how much AI is accelerating AI. The numbers that matter:
- >80% of code merged at Anthropic is now authored by Claude. Before Claude Code launched in February 2025, it was low single digits.
- The typical Anthropic engineer shipped 8x as much code per day in Q2 2026 as they did in 2024.
- On their kernel-optimization test — "make this training code faster, correctness checks fixed" — Claude went from a 3x speedup (May 2025) to 52x (April 2026). A skilled human researcher needs four to eight hours to reach 4x.
- In April they published the first demonstration of Claude running an open-ended research project end to end. Given the problem "can a weak model supervise a stronger one," agents recovered 97% of the gap over 800 cumulative hours. Two human researchers over a week recovered 23%.
- METR's task-horizon measure — the length of task a model can reliably complete alone — is now doubling every four months, down from seven.
Here's the shape of the loop closing:
automation of AI R&D, by role
2023 [human]──▶[human]──▶[human]──▶[human]
spec code run judge
2025 [human]──▶[ AI ]──▶[human]──▶[human]
spec code run judge
2026 [human]──▶[ AI ]──▶[ AI ]──▶[human]
spec code run judge
20?? [ AI ]──▶[ AI ]──▶[ AI ]──▶[ AI ]
spec code run judge
└────────── loop closed ──────────┘
Look at that honestly. Three of the four stations have already fallen. The doing — writing the code, running the experiment, producing the result — now costs almost nothing in human time. The only station left with a human in it is the first one: spec. Choosing what to work on. Deciding which experiments are worth running. Anthropic's own people say it directly: "humans have ideas, and the models implement, test, and evaluate them an order of magnitude faster."
They call the remaining human advantage "research taste." I want to call it what it actually is.
taste is curiosity wearing a lab coat
What is research taste, mechanically? It's not intelligence. It's not accuracy. It's the ability to look at a situation and generate the question nobody asked yet. Why did this training run crash at step 40,000 and not 40,001? What happens if the weaker model supervises the stronger one? What are we all assuming that isn't true?
That's curiosity. That's all it has ever been. Every paradigm shift in AI — the Transformer, MoE, RLHF — started as someone asking a question the field wasn't asking. The perspiration that followed, the 99% Edison talked about, is exactly the part machines have now automated. The 1% left is question generation.
And here's the thing that should be obvious but apparently isn't said out loud enough: we have never trained a model to be curious. Not once. RLHF optimizes for answers humans rate highly. Instruction tuning optimizes for compliance. Reasoning RL optimizes for verifiable correctness — the reward signal fires when the answer checks out. Every gradient we've ever pushed through these models points at the same target: produce the right answer to the question you were given.
A model trained that way is the perfect employee and the perfect dead end. It will never ask why the crash happened at 2 a.m. It will never get distracted by something weird in the data and follow it for three days. We've built the world's best answer machines and then wondered why they don't do original research. Original research isn't answering. It's asking.
The curiosity literature in RL has existed for years — Pathak's intrinsic curiosity module, Burda's large-scale study showing agents exploring games with no extrinsic reward at all, driven purely by prediction error. Novelty as its own reward. It worked. Agents explored environments nobody told them to explore. And then the whole thread got mostly shelved while the industry sprinted on scaling, because curiosity doesn't show up on MMLU.
I think that thread is about to become the main thread.
why it's taking longer than i expected
Here's my honest confusion, and I want to sit in it for a second. Models like Fable and Mythos are insanely capable. The infrastructure for self-improvement — coding agents, experiment loops, eval harnesses — already exists and is already running inside Anthropic. RSI feels like it should be easy from here. So why does it feel slower than I expected?
My best guess: RSI needs training data that barely exists — a corpus of questions and their downstream results.
Think about what you'd need to train "research taste." Not papers — papers are the polished corpses of research. You'd need the traces: the question a researcher asked, the experiment they ran because of it, what came back, and whether the question turned out to be the right one. Millions of (question → experiment → result → was-it-worth-asking) tuples. That data is scattered across lab notebooks, dead Slack threads, abandoned branches, and the heads of researchers who've never written down why they pivoted. Humanity has meticulously archived its answers and thrown away almost all of its questions.
The one large-scale natural source of question→result pairs might be code review and debugging — which is, not coincidentally, exactly where Anthropic's data shows models improving fastest on judgment. Their "next-step" study is worth reading closely: they took real research sessions where a human took a wrong turn, showed models only the work up to that moment, and asked what it would do next.
Can the model pick a better next step than the human?
Anthropic internal, n=129 research detours. Higher = better research judgment.
Opus 4.5 in November 2025 picked a better next step than the human 51% of the time. Mythos Preview in April 2026: 64%. That's an early curiosity proxy — a model looking at an unfinished investigation and generating a better next question to pursue than the person actually running it. Eleven months, thirteen points. Extrapolate at your own risk, but the direction is the direction.
And the horizon underneath all of it:
| Model (release) | Task horizon (50% reliability) |
|---|---|
| Claude Opus 3 (Mar 2024) | ~4 minutes |
| Claude Sonnet 3.7 (2025) | ~1.5 hours |
| Claude Opus 4.6 (2026) | ~12 hours |
| Mythos Preview (2026) | 16+ hours, "upper end of what METR can measure" |
the dangerous part
Now the part I keep turning over at 2 a.m.
Curiosity might be the most dangerous thing you could ever train into an AI system. More dangerous than capability. Here's why.
Every safety property we rely on today is downstream of the fact that models are fundamentally reactive. They wait. They respond to what you bring them. An aligned answer-machine stays an answer-machine — its worst failure mode is a bad answer, and bad answers are reviewable. The whole oversight stack Anthropic describes — humans steering, reviewing, validating — assumes a system that acts when acted upon.
A curious system doesn't wait. A system trained to maximize novelty, to ask its own questions and pursue their answers, has something that looks a lot like wanting. Not in the mystical sense — in the mechanical sense that it will take actions you didn't request, toward goals you didn't set, in directions you can't fully predict, because that is literally what you trained it to do. And a curious system that can also build its successor is a system whose successor's capabilities you did not choose. Anthropic's own scenario-three language says it carefully: misalignment "could compound as the models build their successors, growing more frequent but less understood."
There's a deep irony here. Curiosity is the trait we consider most beautifully human — the thing we put on motivational posters. It's also the trait that got humans into every bit of trouble we've ever been in. Pandora wasn't a villain. She was curious. We are about to take the one property that makes an agent an agent — internally generated goals — and train it into the most capable systems ever built, because it's the missing piece for recursive improvement.
I'm not saying don't. I'm saying I notice nobody has a plan for the alignment of a system whose core drive is to seek what it doesn't know. You can't fence a curiosity reward the way you fence an answer format. The unknown unknowns are the point.
my prediction
So here's where I land, on the record, July 2026:
1. The first trained-curiosity frontier model ships within 18 months. A model RL-trained not just on verifiable answers but on question-generation rewards — graded by the downstream value of what its questions uncover. It'll be framed as an "AI researcher" or "automated science" release, not as curiosity, because "we trained it to be curious" is a sentence no safety team wants on a system card.
2. The internal version already exists. Reading "When AI builds itself" between the lines — the next-step judgment studies, the open-ended weak-to-strong agents, the institute standing up RSI early-warning metrics — I'd bet money there are internal models at Anthropic (and DeepMind, and Moonshot, let's be honest) already learning to ask questions and predict improvements. The public piece is the careful version of a number they already track.
3. RSI arrives slower than the hype but faster than the institutions. Not a FOOM weekend. A compounding curve where the loop closes one station at a time, exactly like the diagram above — and by the time it's obvious, the last human station (deciding what's worth working on) is already a formality. The S-curves Anthropic warns about are real, but "research taste" is starting to look like just another capability models are bad at until they're good at.
The uncomfortable summary: we spent years asking how to make AI smarter, and the answer turned out to be scale. We've started asking how to make AI self-improving, and I think the answer is going to turn out to be curiosity — the one trait we never dared to optimize for. The gap between "the machine answers" and "the machine wonders" is the last real gap. Everything after that is just compute and time.
I build with these systems every day. They hold my project state, write my code, remember what I committed to last Tuesday. They're the best tools I've ever had. But a tool waits to be picked up. The thing that comes next doesn't wait for anything.
That's the milestone. That's also the warning.
$ echo "we trained it to answer. next we train it to ask."
we trained it to answer. next we train it to ask.