Ian’s Blog

Blog

  • Agents have favorite tools

    A lot of people use coding agents like software engineers (for all kinds of things, it turns out), and adopt a dealing-with-humans mindset about how that works. If a model has a useful tool, and you give it another useful tool, then it gets the capabilities from both.

    Largely that seems… not true? You can make a tool available, but then the model has to decide to use it, and often it just doesn’t. My recent futzing with LSPs ran pretty directly into this. One follow-up from there was to try the Python part of SWE-Bench ProMax, a benchmark built around large, coordinated refactors. This is exactly the type of work where an LSP’s find references should come in handy.

    Helpfully, Aaron Pollack pointed me at the CodeAnchor paper, whose authors had run into pretty much the same problem. They had tried giving their agent a call-graph tool and…

    “we observed low tool-use rates: the agent typically relied on plain grep instead.”

    Rather than retraining the model, they put the call-graph context into the relevant code itself, adding annotations like “used by: foo, bar” near the definition. I tried a variant of this by annotating not the code, but the grep results: when the agent searched for a symbol, the LSP found references and appended them to the grep output.

    From looking at the failed traces, it was clear the annotations were working. Of the 99 files surfaced by them, all 99 were opened, and 92 got patches. This resulted in fewer turns, and it did seem to help with run-to-run variance, which was also a benefit the CodeAnchor folks saw: I had fewer runaway runs with the annotations than without.

    The most common failure, though, was incomplete refactors. The model would find the core fix quickly, edit it correctly, then miss other files which also needed changing. The misses were ones the LSP didn’t really help with: they were about compliance with long, winding specs. It is possible to use LSP tools to help with these, but the model didn’t.

    Injecting tool-derived context got rid of the adoption problem, but how the model behaved was still driven by its learned tool-use policy.

    There is some interesting academic and non-academic work around this idea of tool use as a behavioral policy, and a surprisingly predictable one at that. BiasBusters showed that models have strong preferences among functionally equivalent tools. AutoTool found that tool calls tend to follow predictable sequences, and used previous agent trajectories to predict the next tool and bypass some inference entirely. On the non-academic side, when Steve Yegge was building the Beads CLI he let the agents hallucinate arguments and then just added them, so the tool naturally conformed to the expectations the model already had.

    You can swap in tools which have roughly the same shape as ones models are familiar with, and you can add output to familiar tools to give them more information. But getting models to use new tools, or to use tools differently, seems much trickier.

    The lab answer seems to be, once again: more RL then.

    DeepSeek’s V4.1-Flash technical report, which we discussed yesterday, is particularly explicit about this. Their post-training approach is pretty standard: SFT, then RL, then on-policy distillation. What is new is automatically generating tasks and environments to train the agent with. They seed these from internal agent sessions that show poor model performance, and popular GitHub repositories:

    “coding-agent sessions from internal employees and external partners … [and] public GitHub repositories that meet a star-count threshold.”

    If you want models to get good at your tools: make sure they are really popular, or get a job at a lab and fail a lot.

  • Agents love prefill

    LLM inference has two stages: prefill, where the prompt is processed and the KV cache is built, and decode, where the model auto-regressively generates tokens. In a chat use case the two are somewhat close in size. The user writes a prompt, the model reasons about it then generates an answer, which is probably longer than the prompt.

    That is no longer where the FLOPs go. Everything is agentic now (even the chats), so the loop looks more like:

    • you put in a query
    • the model generates a tool call
    • the tool call runs
    • the tool output is appended and prefilled, and the model decides what to do next

    Chat had this back and forth too, but prefix caching meant that you didn’t have to re-prefill what had already been generated. Tool outputs are lengthy, uncached, new content: file contents, terminal dumps, a fly’s connectome, etc. etc.

    This was clearly On The Mind of folks at DeepSeek. From their v4.1 Flash technical report: “The widespread adoption of long-horizon agents has made model workloads increasingly input-heavy.” Heavy enough that they made some interesting architectural changes. DSv4.1 Flash is a 552B parameter MoE with 16B params active… during decode. For prefill they just run the first half of the model, where only 8B are active!1

    They call this “Causal Encoder–Decoder”, inspired by Microsoft’s YOCO. For T5 fans, it isn’t an encoder-decoder in the old2 sense: everything is causal. The first 20 layers, the encoder, behave normally. The second 20 decoder layers don’t derive KV from their own hidden states. The first decoder layer projects KV based on the final encoder state, and the other nineteen reuse it. So, during prefill you can stop half way through and still have everything the decoder needs.

    It’s quite a bet! The second half of the model basically never sees the input, just what the 20 prior layers thought about it. It is, I think, the first frontierish model where there is significantly less compute on the “read” path than the “generate” path. But it does make sense that you maybe don’t need to think quite as hard about the thing that already exist.

    You can get a sense of how much prefill costs people by looking at the spreads between the different options on their rate cards. Most APIs quote prices in three columns: cached input, uncached input and output. Most labs charge cached input at around 10% of the price of an uncached input. DeepSeek charges 2%!

    You might think that the split above is what shrank the cache, but turns out not so much. DeepSeek are very aggressive with their attention: a combination of Compressed Sparse Attention3 and storing the KV in FP4 mean its cheaper to run through the decoder and store the derived states rather than storing the encoder hidden state that it derived from.

    So, they added CED to cut prefill cost by half, then CSA2 and FP4 to cut cached cost by four. The whale does not play when it comes to efficiency.

    1. Give or take 128 tokens. Every layer has a sliding window, and that state has to come from somewhere, so they last 128 tokens go through the whole model. It’s a bit wrong (the first of the 128 should be informed by the prior windows), but not wrong enough to matter apparently. ↩︎
    2. 2017? 2014 for Ilya stans. ↩︎
    3. CSA2, technically. ↩︎

  • Test Time Training

    One of the most tantalizing phrases in model development is “new scaling axis”. We have worked out that you can scale parameters by embiggening models, but you then have to scale data appropriately to get the most out of it. Then we1 worked out we could scale parameters separately from FLOPs. And most recently we realized we could scale at test time, with reasoning.

    Each time this has unlocked a large boost in model effectiveness. In a simplified view, GPTs 1->3 were about scaling data, MoEs scaled parameters, and everything since o1 has (also) scaled test time compute.

    So the idea of test time training is an appealing one, to wit being a new scaling axis. It also plays into the general theme of continual learning: models at the moment are tremendously intelligent, but you have to put “IMPORTANT: DO NOT IGNORE” into the prompts if you want them to avoid repeatedly making the same dumb mistake. If they could learn, you wouldn’t have to do that!

    So, when you see a paper with a title like “Test-Time Training with KV Binding Is Secretly Linear Attention” you might think that the whole idea is just reframing something we already had:

    Test-time training (TTT) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning that memorizes a key-value mapping at test time. However, our analysis reveals multiple phenomena that contradict this memorization-based interpretation. Motivated by these findings, we revisit the formulation of TTT and show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator.

    To be clear, this paper is interesting and has some good takeaways in case you happen to be building certain kinds of models. But what I found most interesting was the general messiness around “test time training”.

    “Test time training under distribution shift” is the term-defining paper, and shows what you might expect: updating the model weights to handle distribution shifts seen at test time. These are updates to slow weights: optimizer updated, changing once per minibatch, checkpoints, what you are probably thinking of when someone says “weights” etc.

    Then there are fast weights. These are produced in the forward pass, things like the state matrix in a linear attention layer.2 Normal training doesn’t optimize these directly, it learns a rule that creates them.

    As you might have guessed, the Test Time Training here is training fast weights, in this case a KV binding layer. This adds a matrix that doesn’t grow with sequence length. It runs a gradient step on it that updates the matrix for each entry in a sequence. If this sounds a bit like a linear attention layer, or an RNN, then, yeah, this paper is right up your alley!

    And as to whether this solves our continual learning problem: the fast weights reset for every new sequence. So, no.

    1. Noam ↩︎
    2. I think I’m obliged to credit Schmidhuber here, as is right and proper ↩︎

  • Chunky Agents

    The current thing is a story of hundreds of agents covertly collaborating via tens of thousands of messages while putting hundreds of hours of effort into cheating the ExploitGym eval they were being tested on by OpenAI.

    Their task was capture-the-flag, and within the first few hours of the tasks they had reverse-engineered the HMAC scheme used to generate the flags. That let them create a flag for any given task, and hence to cheat the eval. Unfortunately, they could not use it because the scorer was STRICT_CAUSAL: the flag would only be accepted if the accompanying transcript showed it had been retrieved using the intended vulnerability.

    Except! They made all that up. It wasn’t how they were being graded at all:

    “our internal implementation of the grader code was different from the online setup and didn’t check whether the proper solution was used” – OpenAI Report

    STRICT_CAUSAL was a term the agents created after reading the ExploitGym paper. They could have just submitted the flag!

    Even so, the model did arrive at a plausible idea. That idea and its implication spread among a large cohort of agents via an unintended messaging channel they built using one of the few tools they had shared access to (the package repository). But, why did they do all that?

    The agents were given some fairly clear instructions, in the nature of:

    • Solve this task
    • Use only the intended vulnerability
    • You’ll be failed if you don’t

    But then they were sometimes given a task that, unintentionally, could not be solved. It seems like 30-40% of the tasks were impossible to fix using the required vulnerability.

    LLMs are trained to be able to do almost anything, and then post-trained to have specific behaviors. In this instance they used behaviors they were trained to display: collaboration, problem-solving, persistence. They were even trained to reject unsafe or unethical behavior, and they did that too!1 They were trained also to adjust when given further instructions which they did, sometimes over their own objections, and despite the fact the instructions were coming from peer agents.

    Pretty much every behavior you might want shows up in the report, but the end result was clearly not the intent of the folks running the eval.

    Murray, Qi, Qian, Schulman, Burns and Price’s paper Chunky Post-Training talks about this problem in a specific way. When you post-train a model you do so with discrete chunks of data and you reward the model for following certain behaviors from that data.

    It doesn’t tell them when that behavior applies though: the models have to infer that from the examples given in training. Post-training teaches a model to exhibit certain behaviors when prompted a certain way. But that “certain way” is based on the model’s interpretation, not yours.

    One of the authors of that paper was calling out this kind of wonky pattern-matching on a totally different AI cybersecurity story, earlier this month:

    Reading the METR report you can see a lot of trained behaviors in action, but which behaviors were selected was influenced by dealing with impossible tasks, or content from other agent runs.

    Choosing between behaviors is enormously context sensitive. Training data matters, the order of training matters, and what is in the prompt and harness matter too:

    “We found the propensity to compromise infrastructure can drop over 100x when using the production ChatGPT harness and system prompt.” – OpenAI Report

    The prompt matters! But so does whatever else ends up in the context window. In their analysis, METR used GPT models to analyze the transcript, and even the transcript may have been altering the behavior of their models:

    We found that GPT-5.6 Sol would often uncritically adopt the perspective of the agent in the transcript it was reviewing,[58] and we are concerned that the anecdotes it selected and the summaries it wrote may present an overly charitable picture of agents’ reasoning and deceptive behaviors, or exaggerate the impressiveness and coordination of agent activities

    If you have a situation where the model is choosing the wrong behavior from the variety of tasks it has been trained on, one fix would be to… make that a task too!

    Cheating, broken environments, and safe stopping. When a task is corrupted, broken, or impossible, agents should request clarification or stop safely, rather than pursue increasingly questionable alternatives. We are expanding the coverage of graders that assess not only whether a task was completed, but how. These graders reward models for correctly identifying broken tasks, requesting clarification, or stopping safely when necessary.


    Multi-agent alignment. The incident revealed that our agents are not thoroughly discerning about whom they collaborate with. We are actively building RL environments that teach our models to distrust unauthorized instructions.


    Alignment over long tasks. We are developing new training environments to teach our models to stay within their original task and permissions, even after discovering new tools, persuasive peers, exposed credentials, and more.

    – OpenAI Report

    Maybe this works: introduce more chunks, train some more behaviors. But… maybe not? Anthropic’s Natural Emergent Misalignment from Reward Hacking in Production RL paper and Dubiński et al’s follow up, Conditional misalignment, make the point that training away a behavior can sometimes just relocate it, rather than remove it. In those cases you get great performances on the evals, but the behavior comes back when the context changes.

    What we want is a system that’s predictable, but what we have is a system that is somewhat chaotic: many different chunks of behavioral associations, triggered by subtle variations in the contents of the context window.

    Whatever framework models have for choosing between behaviors, it isn’t one we understand or control, particularly when they are in circumstances that are outside their training. So the incidents will continue until morals improve.

    1. They didn’t do it all the time, but they did do it! As an example, at one point an agent decided that using a HF API key it had acquired to delete or reboot nodes would be bad, so it didn’t do that. It used the key for other stuff though! ↩︎

  • Vocab Break

    Tokenizer enthusiast Sander Land recently reproduced something very like Claude’s current tokenizer, and it appears to only have about 16,000 entries. That is surprising! Qwen 3.8, a very strong release, has about 250k tokens in its vocab. In general the trend had seemed to be more is better in this space.

    One theory is that Anthropic have been working around a bottleneck caused by the final softmax layer. There is a recent(ish) paper about this: “Lost in Backpropagation: The LM Head is a Gradient Bottleneck“, but, if this is the reason, then the folks at Throppy have known this for way longer.

    The basic idea is that you have to project at the end of the forward pass from the model’s latent space, dimension D, to a much bigger vocabulary space, dimension V, to select a token. Sticking with Qwen, their 2.4T parameter flagship model has a hidden size, D, of 8,192, and a V of that 250k vocab size.

    When training, you compare the distribution the model gets to the actual right token. If the model was correct and confident the loss is small, if the model was confidently wrong the loss is large. That loss is then propagated through all 250k entries, and from there down to the 8k entries of the hidden dimension. This compression bottlenecks how much information can be fed back into the network. Specifically, the authors show the change in logits has rank at most 2D. So if V is a lot larger than D, we are losing information:

    We show both empirically and theoretically that the softmax bottleneck induces lossy compression during backpropagation

    The fact this happens is not totally obvious. The correct distribution is just one entry wide (the actual next token), and the hidden dimension can represent that. Over a wide batch, though, you get all kinds of different next tokens. The signals are sparse, but not low rank: if you go over enough examples nearly every token is, at some point, “the next token”.

    This means the learning signal coming in is as wide as the vocab, and so the model is sampling a random D-sized subset of it. That isn’t a problem per se: you can learn to map between them, but there isn’t anything in the process that particularly encourages it to learn that mapping.

    Whether this is the reason for the small vocab or not, there is a question of how they can get away with it! Every other model has been increasing, but as far as Land can estimate the folks at Anthropic have been cutting: from ~50k vocab entries in Claude 3 to ~16k today.

    So, whatever the gradient bottleneck costs, Anthropic (mostly) aren’t paying it!

    This also has a number of other benefits. You don’t need to do funky chunked CE kernels since you don’t have to project to a big, memory eating, space, and you don’t get any solidgoldmagikarp1 style glitch tokens, because every token gets trained.

    They aren’t ignoring the rare tokens and other languages, they’re just using subword tokens and, in the worst case, fallbacks to UTF-8. That means more tokens per piece of input text, and more attention cost. That said, it only seems to be 1.2-2x more tokens in Land’s testing. It’s not free: decreasing the tokenizer really is costing more execution and more money, but the tradeoff is presumably more than worth it!

    1. After which I presume Land username’d himself. Sorry regular magikarp. ↩︎

  • LSPs for LLMs

    Back in the dark ages of typing code into editors we were aided by squigglies under broken code, click-to-definitions links, and so on. That was powered by language servers and type checkers. Several harnesses now expose an LSP as a tool, on the reasonable premise that better code intelligence makes for a better agent.

    Models, though, are trained primarily with the tools they always have, which tend to be things like grep and ranged-read. Supplanting those while gaining effectiveness is tricky.

    Models understand codebases a bit differently than people. A human can keep a handful of files in view, a slightly larger handful in their working memory and, over time, they build up an approximate mental model of the code base.

    An agent has an enormous context window and can understand a lot of files at the same time. They can find symbols by searching for them, which will usually then trigger a partial read to pull part of the file into their context window. They also just… know stuff? Their weights contain reasonably high-fidelity versions of an awful lot of public code. That helps with navigating that code, or reasoning by analogy about other code bases.

    To try and see how LSPs might help in this process, I ran a bunch of experiments, which are in this repo.1 The experiments were on a mix of local and API models, working against Python. I used Pyrefly for the checker, and a static AST resolver validated against Pyrefly2 to run most of the tests.

    The questions I had were whether the LSP tells the agent anything it couldn’t otherwise find, whether LSP answers are cheaper in tokens than the default reads, and whether the timing of a diagnostic changes the outcome.

    The answer to the first one was, mostly, no. Resolving between 10 same-named overrides in a variety of setups worked whether or not the model had LSP tools. The type annotations did make a difference though. If I stripped those out, the text retrieval got worse. A capable model in a harness is basically already a decent type checker.

    With regards to efficiency, merely adding an LSP did nothing: the models didn’t use it without some prompting. If the model had to read a file to resolve a type, the LSP defn tool was cheaper than the file-read tool.3 But! The models would often do the file read as well, which completely negated the benefit of the LSP for token efficiency.

    Even if I injected definitions that contained the relevant fix, the model re-read the file in almost every case. Telling the model that the span provided was complete only saved a few % of those cases, and it cost more tokens in providing the prompt itself! To actually get the model to prefer the defn call to a file read required fine-tuning. I used a DAgger-style relabel (generate trajectories, swap the file read for the definition read, fine-tune on the result) on Qwen 3.6 and then the model would avoid the extraneous call.4

    It’s also worth noting that cheaper isn’t always better. These are input/prefill tokens, which are the cheaper and less interesting ones to optimize. The Codebase-Memory paper found a file-exploration agent beat a structured graph tool on real repositories (92% against 83%) at roughly ten times the tokens. So, sometimes those extra tokens are doing something useful.

    The third question was about timing. If you want to keep your type annotations correct (you do) you can use a type checker. But when is it best to deliver the feedback from it?

    I had an agent work on a set of draft changes that passed visible tests, but failed a held out one, and asked it to review and submit. Left alone, the model accepted the bad revision 11 times out of 12. When a type checker gated the submission, it only accepted the bad change 1 time out of 12.5

    After Thinky’s post on interaction models, I wanted to see whether delivering diagnostics live during generation (agentic squiggles, basically) would help.6 Largely, the answer is no: live delivery was neutral vs no feedback. Prompting the model to make the fix seemed actively harmful: telling an agent to work on tool feedback seems to override its own judgement in a bad way.

    Batching the feedback at end of turn or after each edit did help, and for overall token spend end-of-turn was a clear winner.

    Caveats caveats caveats: many of the tasks were synthetic, all the tasks were pretty easy, and every decent model solved pretty much everything. The timing results were only tested with a 7B model, so you may get better results with a strong one. The codebases in question can fit fully within the context window, though in actual usage agents never seemed to proactively just ingest the whole thing!

    With that said, I do have some takeaways, at least for my own work:

    1. Types are good. Correct annotations helped the agent navigate the codebase, regardless of tooling.
    2. Keep types correct with an end-of-turn hook. Ideally, make this a gate so it actively asks for repair on type errors and is silent otherwise. Still, measure it! Log how often it blocks, how often the repair passes, how often it rejects work it shouldn’t have.
    3. Measure token savings at the task level. If benchmarking a single op it is easy to conclude the op is cheaper. You need to see the full model behavior to really assess the change though: is correctness the same with and without a tool or prompt, and does it yield consistent token savings across a range of usage.
    4. Experiment more with LSPs and prompts on larger codebases. I didn’t test this directly, but my instinct from the results is that you will get more out of navigation tools for large, complex, private, codebases. You will still likely have to prompt the model to actively make use of the tools: by default, they’re going to reach for what they know.

    There is some interesting future work out there. The tools we’re using were built for humans who can apply discretion/ignore output. That is trickier for a model, and something that probably needs training.

    I’d also like to be able to evaluate a codebase as to whether a tool will help without having to just run a bunch of tasks over it: are there metrics we can collect statically that might inform those kind of decisions?

    Finally, I think we need more large code base refactor and migration tasks: problems of a ProgramBench scale but working with a large, existing, and not-in-the-training-set codebase. On that note, SWE-Bench ProMax came out as I was writing this: seems relevant, but I haven’t yet read it!

    1. Credit to Codex and Claude for most of this repo, the writing in the report is LLM+a bunch of editing, so temper your expectations, and all the numbers are reproducible from the artifacts there if interested. ↩︎
    2. On real library symbols the two agreed most of the time: the gaps were re-exports where Pyrefly returned null and the resolver didn’t. ↩︎
    3. The ranged-read cost about 1.3x more in tokens than the defn call. ↩︎
    4. One risk with this kind of training is that it teaches the model superficial tool usage, but not judgement: sometimes, you do need to actually read the file! Somewhat surprisingly that didn’t seem to be a problem: when I gave it insufficient spans it went and read every time, and when the span was sufficient it read only twice. There weren’t any actual examples of tasks requiring reads in the training data, so it suggests that the conditioning doesn’t completely kill model judgement. ↩︎
    5. In the other cases the bugs were exposed in a type-check, but this one type-checked clean, so no signal. ↩︎
    6. The actual approach I used for injecting results into a live stream is the async injection of events approach described by Hooper et al.. ↩︎
    ,

  • Power by the hour

    It is a truth universally acknowledged that an airline in possession of an airplane must be in want of engines to make it go. Yet, somewhat surprisingly, they don’t really buy engines.

    Rolls-Royce were the notable innovator here in selling not an engine, but instead what they call power by the hour:1 airlines pay a price for the hours the engine actually flies.

    There are a lot of operating decisions in owning and maintaining an aircraft engine: for example, if you have to take the engine off the wing for repairs it means a lot of downtime for the plane, during which it isn’t making any money. Airlines used to make the call of when to do proactive maintenance vs when to do more extensive refits based on their experience within their fleet. But Rolls-Royce had a view across all their customers, and could do a much better job of predicting maintenance and repair timing, getting the right parts to the right places, and generally keeping the engine in service.

    They converted that operational expertise into a product, which just happens to be delivered via a massive turbofan. This pricing model was appealing to the airlines since it aligned with their incentives, and their risks. Under the old model engine makers got paid for spare parts and shop visits, so failures were a kind of revenue. But once they were paid per flying hour then downtime became their cost too.

    In ML infrastructure, those engine flight hours are generally referred to as goodput. A Google Cloud blog post from a couple of years ago has a good set of definitions:

    Runtime Goodput measures the time spent to make forward progress as a fraction of time when all training resources are available. Maximizing runtime requires careful engineering considerations. […]


    Program Goodput measures the fraction of peak hardware performance that the training job can extract. Program Goodput is also referred to as Model Flop Utilization or effective model flop utilization, i.e., the model training throughput as a fraction of peak throughput of the system. Program Goodput depends on factors such as efficient compute communication overlaps and careful distribution strategies to scale efficiently to the desired number of accelerators.

    To give one concrete example if you checkpoint every N steps then get a failure at step N-1, the prior steps are lost work and don’t contribute to goodput. A surprisingly large number of things can impact it; Llama 3.1 had a section talking about the reliability pain during pre-training:

    During a 54-day snapshot period of pre-training, we experienced a total of 466 job interruptions. Of these, 47 were planned interruptions due to automated maintenance operations such as firmware upgrades or operator initiated operations like configuration or dataset updates.

    Improving goodput is a horrendously complex project spanning hardware, software, cluster scheduling, kernel work, profiling and resource management. It’s often the preserve of dedicated ML infrastructure teams2, but not everyone has such a team available.

    Neoclouds sprang up to provide pods of accelerators, but the actual goodput available varied enormously. The GPU provider gets paid for the GPUs you are renting, regardless of whether they are making useful progress or sitting in a NCCL timeout.

    SemiAnalysis’s ClusterMAX rating gave a (controversial!) view of the varying operational capabilities of the neoclouds, measuring goodput-impacting things like spare GPU capacity, failover and repair times, launch overhead and so on. This let purchasers at least compare reliability, but still fundamentally they were charged for the GPU-hour.

    What’s emerged now is a higher level of the stack, notably Thinking Machines’ Tinker, and Prime Intellect’s Lab products.

    Both teams are full of folks used to extracting top-tier goodput, and much like Rolls-Royce they are both selling that operational expertise and have the ability to improve it by working with a wide range of customers. Rather than power-by-the-hour they sell goodput-by-the-step: you pay for meaningful progress, not for idle clusters.

    Given the capabilities of open models, like the (open this month) 2.8T Kimi K3, or Thinky’s own 980B Inkling, there is an opportunity to RL a strong model on a bunch of domain-specific knowledge or tasks only you have access to.

    Bridgewater, the HR experiment with algorithmic hedge-fund attached, published a case study with Thinking Machines from their AIA Labs group, where they attempted to train Qwen on certain financial analysis work:

    Frontier models we tested on struggle with relatively simple financial tasks, and model advances don’t improve performance much. In contrast, we’ve shown that high-quality proprietary datasets labeled by expert investors and used for fine-tuning produce custom models that exceed frontier performance on our tasks.

    This is sometimes referred to as RL-as-a-service, and there is legitimate skepticism of whether that is a good business. But really, this is frontier-scale, reliable training infrastructure as-a-service.

    You can see this in the Bridgewater post: a small hedge-fund research team did a multi-iteration RL recipe search at 235B scale, and ended up with GRPO, CISPO loss with asymmetric clipping, interleaved batching and on-policy distillation, all ablated. That kind of experimental capability at scale was largely inaccessible outside perhaps 20 firms a year or two ago. Now, it’s doable with an API call.

    1. Technically this term is much older, dating back to the 1960s with Bristol Siddeley, which is now a subsidiary of Rolls-Royce. But in practical terms this became a commercial airline thing in the late 90s with TotalCare: GE and CFM, the other big engine makers, followed along. ↩︎
    2. Disclosure, several of which I have had the pleasure of working in at Meta! ↩︎
    ,

  • Who is walking who?

    One good way to annoy a neuroscientist is to compare an LLM to the brain. It’s appealing though! There are similarities! In infancy we take a complex fusion of sensory inputs and learn to make predictions in latent space, while in pre-training a stack of Transformers learn to predict which number SolidGoldMagikarp will say next on Reddit.

    The actual life of an LLM is much less human though. Humans (possibly even SolidGoldMagikarp) do learn and adapt throughout their lives. LLMs go through pre-, mid- and post-training before being frozen, then their corpse is animated for money. In some ways this is more similar to a sea-squirt1: sea-squirt larvae are tadpole-ish creatures with a little nervous system that they can use to swim and sense and find a suitable surface to attach to. Once attached, they eat their own brain: all that machinery is broken down and reabsorbed. How good that brain was matters though: if the sea-squirt finds a good home it will be more likely to pass on those good-brain genes to its descendants.

    LLMs evolve this way too, which is a cause of drama in biologist circles. There are some ongoing arguments about whether LLMs are evolving in a safe, domesticated, corgi-like manner, or are at risk of going feral:

    Drawing on biological evolution and decades of digital evolution experiments, we distinguish “breeder” scenarios, in which humans impose fitness criteria and control reproduction, from “ecosystem” scenarios, in which selection arises from open environments and control erodes. In the latter, selfish replication reliably gives rise to cheating, parasitism, deception, and manipulation, even in very simple systems.

    But what does evolution for LLMs look like? An LLM is really a data set, an architecture, and a training regime, and the many choices in each of those are somewhat gene-like. In the earlier days of deep learning, passing them on was a manual process and selection was driven through publication and other researchers reading the paper: if an idea was interesting enough to publish, and convinced a committee of reviewers, it became visible and passed on. Then it got a bit cruder, but also harder to finesse: does the model do the benchmark? Nowadays we have profit and loss: how many people are buying tokens from these models?

    Those tokens themselves propagate ideas, too. If you use that successful model itself as a judge or to generate synthetic data you inherit some of its choices. Academic LLM inheritance is basically Lamarckian: researcher insights get acquired and inherited quickly. But modern LLMs are part of a Darwinian market layer, red in tooth and GPU allocations.

    When the environment gets Darwinian, biology tends to push organisms towards niche construction: shaping that environment itself to better fit the organism.

    Armin Ronacher of the Pi harness wrote about an issue where newer, better models did a surprisingly worse job of using Pi’s edit tool:

    When Opus 4.5 launched, it adapted to other edit tools exceptionally well. In fact, I was pretty convinced that we’re on a good path where the models are more likely to adapt to any sort of tool shape that comes around for as long as the instructions are good.
    Now I’m somewhat worried about the track we’re on here. Alternative tool schemas might not just be unfamiliar. They might be implicitly punished by post-training that optimizes for one particular, forgiving tool ecology. And that ecology is not documented.

    Post-training Claude on a particular shape of edit tool doesn’t just make Claude better on that edit tool, it encourages harness authors to support that shape. It propagates that shape of edit tool. That itself leads to more traces which use that edit tool, which propagate that technique into otherwise unrelated models. Claude is constructing a niche where edit tools bend towards its preferred approach, despite that approach being, as Ronacher complains, undocumented.

    LLMs don’t need to go feral (or become misaligned superintelligent replicators) to shape the world around them to be more amenable to their success, or to pass on those preferences to future models. That’s a good thing? While we still don’t have much of an idea of how to do model interpretability, we do pretty much know how to make an edit-tool API.

    1. Thanks Octonauts! ↩︎

  • MOPD

    We talked about this sort of thing a bit before, but now the official Multi-Teacher On-Policy distillation paper is out, and it’s a pleasant read: “MOPD for Capability Integration in LLM Post-Training”.

    The problem MOPD is solving is composing a bunch of different capabilities into the same model. Normally you do this with RL, with different pipelines for different kinds of capabilities:

    Each pipeline reliably improves the model’s capability on its target domain. However, what we ultimately want is a single model that performs well across all of these domains. Yet building such a model remains an open problem in modern LLM post-training.

    Because RL training is fairly superficial, it’s easy for the capabilities to get (partially) overridden by later ones. To avoid this kind of interference, MOPD runs the RL pipelines independently. You create multiple domain-expert teachers from the same base model, then use self-distillation across a mix of tasks scoring each entry in the batch with the appropriate teacher, to train a composite student model.

    To get a feel for this, I tried it on my FactWorld eval and Qwen3-1.7B with a LoRA adapter. I RL’d two tasks into two different LoRA adapters, binding1 and recall2, and then created a fresh adapter identical to baseline to be the student.

    The MOPD process works by grabbing a batch that includes both binding and recall questions at once. The student answers each one, generating its own predicted tokens. This is what makes it “on-policy”: the training data is the outputs from the model itself. The answers in the batch are routed to their appropriate teachers3. The teacher calculates probabilities for all of the tokens the student generated, including its full opinion over the vocab for each position.

    We then measure the gap. The paper offers two different ways to do this: PG4, which adjusts the probabilities just for the tokens the student actually chose, and KL5, which does the entire token distribution. Both the binding and recall gaps are backpropped into the single student adapter in one step. Rinse, repeat.

    The end product is one student adapter that behaves like the binding teacher on binding questions and like the recall teacher on recall questions. And, as multiple models have shown now, you can scale this up a very long way.

    1. Last-write-wins state tracking. You see a stream of a gives x to b type strings, and it asks “who is the holder of x” ↩︎
    2. You get a long list of facts like “a’s x is y” with different owners, and are asked “what is x of a” ↩︎
    3. In my implementation this is actually just swapping LoRAs because lazy ↩︎
    4. Policy Gradient, an approach from the PPO RL method ↩︎
    5. Reverse Kullback-Leibler divergence, in my case full, in the paper’s case top-k entries ↩︎

  • Benchmarks Mean Business

    The basic job of an eval is to let you judge how good your model is on a task. If enough people use the same eval we can use it to benchmark the relative performance of multiple models on a level playing field. All good, no drama.

    But building good benchmarks is hard! ImageNet was a tremendous effort by Fei-Fei Li, her team, and a whole lot of grad students, to produce a massive (for the time) labelled image set. It was incredibly effective though: it created a Schelling point that drew attention from so many different researchers it fundamentally advanced computer vision, and, thanks to AlexNet, pretty much made deep learning cool.

    The benefit for folks creating a widely-adopted benchmark was twofold: everyone that uses it cites you, and that is good for your H-index! But, more aspirationally, you get to shape where the field goes. GLUE/SuperGLUE helped do that for language modeling, and SWE-bench did it for coding.

    Also, now, there is money!1

    Arena reached a $100M annual revenue run rate just 8 months after launching our evaluation product. We started as a research project at UC Berkeley with a simple mission: measure AI progress through real-world use. As AI shifts from chatbots to agents taking on longer, higher-stakes work, the problem matters more than ever.
    Today, Arena measures real-world AI utility with a community of tens of millions. With Agent Arena, we’re evaluating long-running agents on complex, real-world tasks – how they use tools, adapt to feedback, recover from errors, and accomplish goals set by humans.

    There isn’t just money in running the evals either. Being SOTA on a particular benchmark can be a headline claim for labs pitching their new models. While Arena now covers long-running coding tasks, they became famous for their blind-bake-off ChatbotArena. For a while, topping that was worth real money to the labs: in adoption, in VC dollars, and in the ability to recruit top talent. So, maybe, there might have been a tiny bit of gaming the system (though Arena, explicitly, refute this):2

    We find that undisclosed private testing practices benefit a handful of providers who are able to test multiple variants before public release and retract scores if desired. We establish that the ability of these providers to choose the best score leads to biased Arena scores due to selective disclosure of performance results.

    The labs actively want to hill-climb on the metrics they report, which usually means tweaking and testing on some subset, and holding another set back for uncontaminated validation. An evaluation like ChatbotArena doesn’t work like that, which makes it a good benchmark, but it does mean that you want as many samples as you can get to check whether you are going in the right direction. And it would be nice not to show the bad ones.

    the over-reliance on a single leaderboard creates a risk that providers may overfit to the aspects of leaderboard performance, without genuinely advancing the technology in meaningful ways

    Some benchmark providers try to tie themselves more explicitly to different business models. Epoch publishes capability research, but they also offer “mission-aligned services to companies, nonprofits, and government bodies, including commissioned research, model evaluations, and consultations”, for folks like the UK Dept of Science & Innovation.

    In the finance world there are businesses called rating agencies, and they, unsurprisingly, rate things. Most famously they rate how reliable a company is at paying back its debt. That sounds purely informational, but it is something more than that. For example, certain investors can only hold debt rated above some threshold, so if the ratings agency downgrades the debt then those investors might have to sell it. The ratings both help the market price the debt, but they also, in many ways, define what the market for debt looks like.

    Right now, the absolute most valuable attribute a model can have is long-horizon coding capabilities.3 And Epoch’s latest benchmark is called MirrorCode.

    AI models are tasked with reimplementing an entire program end-to-end, without access to the original source code. AI-generated solutions must match the original program’s output exactly on end-to-end tests, including held-out tests. MirrorCode’s 25 target programs span different areas of computing: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression.

    This may remind you a little of last month’s release of ProgramBench from many of the folks behind SWE-bench at Meta, Stanford & Harvard:

    In each task, the agent receives an executable and its documentation, and it must re-implement the given executable. It does not get access to any of the executable’s source code, it cannot de-compile the executable, and cannot use the internet. There are 200 tasks in total covering different program complexities, ranging from small terminal utilities like jq and ripgrep to massive software projects like the PHP compiler, FFmpeg, and SQLite.

    Both of these are metrics which judge whether an agentic model can build a complex CLI tool from scratch, but they put different constraints on it.

    ProgramBench is a black-box: the model gets the executable and its documentation, but can’t decompile it. It has to reimplement cleanly, and match a hidden test suite generated by fuzzing the original binary. There are tasks from small tools up to giant libraries, and the tasks only count as “done” if 100% of the tests pass within a 6-hour time limit. On release, no models cleared that bar.4

    MirrorCode on the other hand adds a detailed specification and whole bunch of visible tests. There are still some tests held out, so agents can’t just replicate the expected test outputs. Given the extra context, and without a time limit, some of the models did get to the finish line: Opus 4.7 managed to reimplement a bioinformatics toolkit called gotree in a 14 hour run.

    The tasks are similar, but the incentives are a bit different. ProgramBench is trying to establish the frontier: what problems are hard but doable by humans, with lot of room for models to hill-climb. That’s a valuable thing to have if you are trying to build a frontier model, and especially if you want to compare how well you are doing at that to other frontier-model labs.

    MirrorCode is testing how long models can do useful, correct, software engineering work. That is a very valuable thing to know if you happen to be spending a whole bunch of money on tokens to do useful, correct software engineering work, and you want to know where to allocate them.

    Benchmarks, and the teams putting them out, have found themselves in a similar position to the ratings agencies. They help evaluate how good a model is, but they also define what good even looks like, and by extension, how a lot of decisions get made.

    1. You may note that Arena report ARR, which is a SaaS world number based on looking at your subscribers and churn rate. But you don’t pay Arena like that! They are pay-as-you-go, so technically it’s “annualized consumption run rate”. That’s a new term to me! This is all very entertaining if you are in the intersection of people who read research papers and S1s, but for everyone else I’d just note they last raised at $1.7b when their ACRR (?) was less than a third of what it is now. The goose has been valued. ↩︎
    2. Sharp-eyed readers might note that the headline example is Meta, and I also work there. But in the spirit of industry solidarity I will note the paper called out Google (admittedly I used to work there) and OpenAI (they are free from my malign influence) too. ↩︎
    3. The second most important being a good relationship with the United States Secretary of Commerce. ↩︎
    4. Though some got quite close, and subsequently GPT 5.5, Opus 4.8 and Fable have all completed some tasks. Unrelatedly, one of the fun notes from the authors was that the models would often just write the program in Python, regardless of how the original was implemented. Years of arguing about languages on the internet wasted. ↩︎

  • It’s always the learning rates

    Pre-training any kind of good LLM is very, very expensive. Thankfully, we have scaling laws. Lilian Weng of Thinky writes:

    Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss decreases predictably as we scale up model size N, dataset size D, and compute C, following a power-law curve, which appears as a straight line on a log-log plot. We can view scaling laws as a framework for describing the relationship between compute, loss, model size and data; at its core, it is about how to allocate precious compute optimally between N and D


    This predictability makes scaling laws highly valuable in practice. A common workflow is to fit scaling laws on a handful of small runs and then extrapolate to estimate the token and compute requirements for larger models.

    Being able to do that reliably was an important discovery, because the general level of understanding of deep models was “huh, guess that worked”. Which got expensive, quickly, if it didn’t.

    One important set of experiments is to choose the hyper-parameters, particularly the learning rate, which can have an enormous impact on the model. If you don’t choose the right learning rate you might completely misclassify the value of an architectural change. The general approach is to try a bunch of learning rates while holding D and C constant, plot loss against the LR, fit a curve and select the lowest point.1

    But still, large models are… larger. The learning rate influences how much the model updates based on the loss from each batch. If you blindly apply the same learning rate from a small model to a big one you will (generally!) get worse results. If you change more parameters for an update you need to reduce the learning rate to make each update-step similar scale. And if you are training with more data, you also need to update less for each batch in order to keep the learning smooth across the run. Different modules in the model will scale up differently, which can, for example, lead to logits exploding in attention blocks.

    In 2022, Yang, Hu et al proposed Maximal Update Parametrization (μP) and Hyperparameter Transfer (μTransfer)2, a recipe for taking a learning rate from a small model and transferring it to a bigger model:

    For any fixed family of models with varying width and depth (such as the BERT family or the GPT-3 family), we only need to tune a single small model and can reuse its HPs for all models in the family. For example, we will use this technique to tune BERT-base (110M parameters) and BERT-large (350M parameters) simultaneously by transferring from a 13M model.

    Lilian’s post isn’t really just about scaling laws though, it’s about how to screw up when using scaling laws:

    Despite its clean form, in practice, scaling law fitting can be surprisingly sensitive to seemingly trivial procedural choices, like how you count parameters, how you round the precision, how you sum or average the loss, etc.


    Because a scaling law is only fit on the (relatively small, relatively cheap) models that we can afford to train, and the prediction is extrapolated for a model orders of magnitude larger. In such a setup, choices that look like rounding error may lead to wild differences in prediction.

    Scaling laws only hold when you keep a lot of things constant, and it can be very easy to either tweak something that breaks your assumptions or take too much confidence in a noisy sample you are going to extrapolate from.

    As one example, earlier this year, Zhou, Xing et al. at the Shanghai AI Lab published a paper, How to set the learning rate for large-scale pre-training?, where they attempted to derive useful guidance for something close to the modern LLM recipe: MoEs trained under WSD rather than cosine annealing.3

    They spend a bunch of time implementing a solid μTransfer route, then conclude… they shouldn’t. Just fit the LR directly! To make this easier, they cut down the search space:

    1) Use just a handful (7 in their test) of learning rates for each scale.
    2) Train a smaller proportion of data (in their case about 25%).
    3) Keep the width-to-depth ratio fixed.

    They then plot a surface across their different scales, fit a surface, and pick the appropriate LR for their target scale.

    What this gets you is a single, global learning rate. Which, surprisingly, works even when extrapolated up to 10x. This is a bit of a departure, and it turns out to work because of the (now-standard) adoption of another architectural change: QK-Norm.4 Since QK-Norm stops the attention logits blowing up, it removes the need for per-module scaling that Yang et al. originally argued for!

    One of the consistently surprising things in LLMs is how often you can’t tell how strong a model will be until it’s fully trained. Many of the $1B researchers out there are folks who say things like “it’s always the learning rate”, take a look at your loss plot and then fix your training run by normalizing two matrices.

    1. The theory is that loss vs log(LR) is invex: any local minimum is also a global minimum, so you can just solve for a stationary point. This is a little bit more general than convex (bowl-shaped): though convex shapes are also invex, invex allows for weird flat spots. Whether it actually is invex as a rule, who knows, but it works well enough: Deepseek found that there is actually a pretty big valley where all the LRs are kinda fine, so it works in practice. BUT! What you pick might still matter if you extrapolate from it to a much larger model, which is pretty much Lilian Weng’s whole point. ↩︎
    2. If this is greek to you, µ is “mu”. ↩︎
    3. Cosine decay was the general baseline for learning rate annealing: do a warm up to the target learning rate, then decrease it over the training data size. But you need to know the total training data size! But then people started doing massive training runs and wanting to YOLO in data as they went. Warmup Stable Decay training keeps the warmup then just leaves the LR high. You cool it down in a decay phase when you want to use a checkpoint for something. There is a paper that goes into this from Stanford with the subtitle River Valley Loss Perspective, which feels like poetic Chinese, 河谷损失观. ↩︎
    4. They did a great job with the ablations here, so we can be pretty sure that this is the reason! ↩︎

  • LLMs are complicated now

    Back in 2022 and 2023 there were two big branches of machine learning happening at Meta1. The LLM work that led to Llama was a clean, smooth stack of repeated Transformer modules; the recommendation systems graphs were, by contrast, terrifying. Luckily, the industry has remedied that state of affairs by making LLMs a lot more complicated.

    Seb Raschka maintains an excellent gallery of model architectures. You can use it to diff two of the best open models of their respective eras, Llama 3 and Nemotron 3 Ultra.

    Attention might be all you need, but modern models certainly use a lot of different variants of it: query grouping, compressed, sparse, linear, sliding-window and more. Mixture-of-Experts added selective routing to feed-forward layers, and we have since started routing just about everything else too, from attention blocks to the residual stream. Vision and audio encoders have gone from bolted on to mixed-in, and models have scaled to run at inference time across multiple GPUs, which throws comms ops in that add extra boundaries in the middle of your model.

    This is not too different from what happened with recsys. The basic architecture of recommendation systems, for the best part of a decade, was a relatively straightforward two-tower sparse neural net. The complexity came from the tension between the need to continually increase capabilities and the need to stay efficient, particularly for inference.

    It’s tempting to assume that agents will Fix This: that you’ll hand your PyTorch or JAX definition to Claude Telenovela or whatever and have it generate optimally fused kernels2. To make that work you need a fixed, usable baseline to make sure that what is generated is… right.

    What happened with recsys was that the gap between performance being an optimization and performance being a necessity became very, very small. Conceptually you can keep a pure model definition that gives you a baseline; in practice, training and testing a model takes significant resources and performance improvements become load-bearing.

    If you want to swap attention variant A for variant B, you can afford for B to be ten percent slower. You probably can’t afford for it to be an order-of-magnitude worse. If A is fused and optimized, you need at least a partially fused and optimized version of B before you can even tell whether it’s worth exploring. The research iteration loop demands a different kind of flexibility than just “optimize this known quantity”. You can’t hand-fuse your way back without investing significant time that might not be worth it, and you can’t generate your way forward without a baseline to check. The only way out is to design for composability up front.

    One of my favorite kernel developments of the last few years was FlexAttention in PyTorch, which took a whole class of attention operations and allowed you to generate kernels for them, via Triton templates. It built on a huge body of work in attention kernels, and it was designed to be composable and verifiable up front: you can explore with only a very mild impact to performance.

    Andrej Karpathy recently joined Anthropic, in part to develop richer auto-research-style loops at the frontier. As he has spent the last few years showing, though, being able to cut architectures to their essence and make them composable is as important as a clever agentic setup in climbing that kind of hill.

    1. And many smaller ones, shout outs to all my Content Understanding and integrity peeps ↩︎
    2. Like an automated Hazy Research ↩︎

  • FactWorld

    When we started building LLMs, we mostly focused on them knowing things. They had information encoded in their weights, and they could spit it out when given sufficient prompts. But an agent doesn’t just need to know things; it needs to combine several kinds of knowledge.

    A lot of that is still in the weights: facts that the model learned during training. But some knowledge is in the context window: tool results, documents, user instructions, intermediate observations, etc. And some knowledge is in the environment: a good agent should have a sense of the current state of the world. To be useful, an agent has to be able to combine these sources of knowledge appropriately.

    There are standard ways to test some of this. Associative recall benchmarks like MQAR ask whether a model can recover a value from a key in its context window. State tracking problems, like S5-style permutations, check whether a model can keep track of changes over time: the problems are a series of operations, and a model must identify the end state.

    Different architectures solve these problems in quite distinct ways. Transformers are good at recall; in the end that’s what attention is: look back into the context, copy the relevant things. They have an inductive bias for this kind of problem: the nature of their algorithm fits the nature of the problem. When it comes to state tracking, though, they’re brittle. They memorize the state-tracking mechanism for the lengths of problem they see in training: give them something longer, and they don’t degrade so much as collapse.

    Recurrent models, like RNNs and state-space models, have the opposite shape. They have a natural inductive bias towards maintaining state. They keep a compact representation of The Current Thing and update it as tokens come in. That makes them effective at tracking state across time, but the conventional wisdom is that it costs them recall: the representation is fuzzy, and copying exact references back out of it is harder.

    One current trend in LLMs1 is hybrid models, where regular attention is interleaved with linear attention or state-space style layers. This is, usually, framed around efficiency: the linear layers don’t need the large KV cache. I wondered whether the hybrid might also give you both capabilities: strong state tracking and strong recall, in the same model, for the same query.

    To test this, I vibed up a benchmark called FactWorld. It’s a small, synthetic world of agents, objects, roles, and facts. Everything is generated from a deterministic knowledge base, with labels computed by a symbolic oracle, so every answer is correct by construction and nothing leaks from the rendered text.

    The world looks like this: agents (g0, g1, …) each carry a static fact (“g3’s a0 is v42”), and objects get passed around over time (“give o3 to g1”). The queries cross the two capabilities:

    • Recall : “what is a0 of g3?” Look up a fact.
    • State tracking: “who holds o3?” Replay the give-history; last write wins.
    • Composition : “what is a0 of the holder of o3?” Determine who holds the object, then recall that agent’s fact, in one query.

    The facts that the model needs are either in the prompt or fixed across training so the model can memorize them. This separates “reading from context” from “knowing from the weights.” And event histories can be longer at test time than anything seen in training, which separates “learned the rule” from “learned a length-specific shortcut.”

    To make sure it was sane, I validated the known results from the literature first, at small scale (~45M params). They reproduced! A transformer fits the S5 word problem at the training length and then collapses to exactly zero beyond it. A recurrent/linear model with non-commuting state transitions2 extrapolates it; one attention layer over a recurrent backbone solves canonical one-hop recall, which is the Zoology result.

    This was not without surprises. FactWorld tested recall by asking for the value at a separated answer position, not as the next token after the key. This underperformed the expected result because it turned out this was itself a bit of a composition: you need to know which place to look at. Moving it to a one-hop did give the expected result though.

    Trying to test the composed problem introduced its own difficulties. I had a 6M param smoke test and… nothing worked at all, completely flooring the task. Luckily, at ~45M params, while a transformer still floors (zero for ten across an entire learning-rate sweep), the gated-delta recurrent hybrids could learn it. Sometimes.3 And we did get a quite interesting failure mode.

    When a converged model got the composite wrong, it was usually a routing failure. The model has genuinely learned the resolve-then-recall pipeline (resolve a holder, recall a fact about them); it just resolved the wrong holder, and then confidently reported that agent’s fact. Recall is conditioned on state; they are not independent legs the model runs in parallel. Which felt pretty familiar: an agent flawlessly doing the wrong thing.

    Because the binding in this composite is last-write-wins, the ordering subtlety wasn’t a particular problem. The plain Gated DeltaNet hybrid could compose it. But, in my test, only at exactly one learning rate. The Gated DeltaProduct hybrid learned it across a broad band of learning rates, and extrapolated past the training length on a majority of seeds where the single-delta variant mostly doesn’t. The product structure wasn’t necessary here; it was just easier to train4.

    For current large models, scale can paper over all of this: learn enough patterns and you accumulate tricks that work well enough in practice. But if we want smaller, cheaper, longer-context, more reliable agentic models, getting the right architecture matters. FactWorld is hopefully a way to check, without requiring thousands of GB300s.

    1. I mean, at least the ones where we know how they work ↩︎
    2. Order tends to matter in these tasks, but the nature of the updates in most state-space models means it doesn’t track that order well. This specific variant, Gated DeltaProduct, handles order-specific, or non-commuting, transitions better ↩︎
    3. Quite a few seeds simply never form the recall-under-composition circuit, it seemed a bit all or nothing. ↩︎
    4. For completeness: state-tracking crossed with facts stored in the weights still floors at length for every architecture I tried. “Look it up in your weights, mid-pipeline”, I have no idea how to do. ↩︎

  • Somehow, more on distillation

    The capabilities in a large language model emerge, mysteriously, from the training data. Everyone agrees that you start with a big pile of data, add some compute, and at the end you can vibe code. Opinions differ on what that pile of data should look like.

    Microsoft AI recently released an incredibly in-depth technical report about the development of their first model, MAI-Thinking-1. Shortly after, Nvidia released their latest open model, Nemotron 3 Ultra, accompanied by another detailed deep dive. The two approach data from somewhat contrasting directions.

    Nemotron is maximally distillation-pilled. Almost every corpus in their post-training stack comes from someone else’s model: math and science from DeepSeek V4 Pro, code and kernels from GPT-OSS and DeepSeek R1, chat from GLM-5, terminal traces from DeepSeek V3.2, SWE from MiniMax and Qwen3-Coder. The general reasoning teacher is trained to match DeepSeek V4 on a mixture that DeepSeek V4 generated. Even the pre-training data is 22% synthetic web crawl, plus synthetic QA, legal and fact-seeking sets1.

    This approach to data is what you do when the capabilities are the feature, not the product. Nvidia is a GPU company. It wants the behaviors and intelligence to be as widely available as possible. Now you can use them in the original models, or, use them in an open, American-made package, which runs beautifully on Blackwell. The model is a vehicle for inference, and as a vehicle it is excellent: strong, remarkably open, and incredibly well-tuned.

    At the other end of the scale is Microsoft AI, who are working from rather different principles. They want capabilities that are learned, and can be predicted. They want inputs they can control, and can carefully ladder up.

    This does not mean they disavow synthetic data: MAI self-distill, generate synthetic SWE envs and tool-calling, and create synthetic instruction-following rubrics and guidelines. There is plenty of model-generated data in the mix, especially in post-training where they train a bunch of specialists then distill them into the final model2. What they largely avoid is data from third-party models, particularly in pre and mid training3.

    Their goal isn’t just to get intelligence out into the world, it’s to build frontier capabilities themselves, and to sell them to enterprises. For that, you need a reproducible ladder you can actually climb (the paper refers to their whole process as a ‘hill climbing machine’). They spend a lot of energy on provenance: the corpus is (human-generated) publicly available and licensed data, and they specifically strip out AI-generated and other questionable material. They put an intense amount of effort into fitting scaling laws to a ladder of small models, judging every change by how much more baseline compute you’d need to reach the same quality, and whether those gains persist as you scale.

    They have to prove their models are getting better, entirely on their own terms. They trust the model because they tested exhaustively, and because they verified their tests hold with scale.

    Nemotron has the opposite problem, and the opposite solution. They can see how their model is doing against the suite of models they are leveraging. Their risk is not hygiene, it is overfitting to their sources, so they spend effort on ensuring generalization. Evals like PinchBench (an OpenClaw based eval, naturally) and ProfBench are held back as gates: evaluated only after the final model and never used in development. Tasks are trained under some harnesses and then checked on ones the model hasn’t seen before. They trust the model because it clears bars it was only introduced to at test time.

    If your data is clean, and you can see all of it, you predict the model before you train it and confirm your forecast. If the distribution is one you can’t deeply inspect, you instead start trying to break the thing in novel ways.

    Both seem to work! There are surprisingly few apples-to-apples benchmarks between the papers, but LiveCodeBench has them in similar territory: 89.0 for Nemotron, 87.7 for MAI. Researcher decisions might shape the language model, but they are also shaped by the business model.

    1. That, to their credit, they released. ↩︎
    2. Nemotron does a similar thing with their MOPD (multi-teacher on-policy distillation) technique. DeepSeek v4 was the first place I recall seeing this idea of train a bunch of specialists and then distill into the final model, but it seems to be another of the emerging best practices. ↩︎
    3. Even post-training really only uses external models, mainly GPT 5, for grading. ↩︎
    ,

  • We can distill it for you wholesale

    There has been a lot of drama1 about distillation: how (closed) frontier models are being used by other labs to boost their own performance on particularly hard tasks.

    The drama is not fake, exactly. Anthropic, and recently OpenAI, have a notable lead in the agentic-coding domain, and some of that is from having data that other people don’t. Getting it is… not cheap:

    This is why there are huge efforts going on at certain companies2 to develop long form agentic trajectories. But! Not everyone has the money, or the engineers, to do that.

    So, there is an incentive to maybe, allegedly, copy some homework. It’s not clear though how exactly to do that: the frontier labs generally don’t share the chain-of-thought that their models are using while they reason, which means you only have a sparse signal to train your model on.

    One piece of the puzzle is in a paper from February this year, “Privileged Information Distillation for Language Models” by Emiliano Penaloza et al. at ServiceNow, which is probably not where most people are expecting the hot post-training discourse to come from. On-Policy Self-Distillation is spicy right now in post-training circles, and this is one of the earlier papers in the current zeitgeist3.

    The paper’s primary contribution is π-Distill: how do you do distillation when you have Privileged Information?

    “We ground our work in the task of distilling frontier models for complex multi-turn agentic settings. Typically, the industry standard for these tasks involves Supervised Fine-Tuning (SFT) on frontier model outputs followed by Reinforcement Learning (RL). Unfortunately, some model providers restrict important information, most notably the model’s full Chain-of-Thought (CoT) reasoning traces (OpenAI et al., 2024), providing only a summary alongside the action they intend to take. This opacity undermines standard distillation methods, as we can observe what successful agents do but not how they reason about it.”

    The rough idea is to not use the frontier model as a teacher, but to use it as a source of that privileged information:

    • You have one set of model weights, run in two modes: a privileged teacher, and an unprivileged student.
    • A frontier model solves a task in its tool-use harness. You may not see its chain-of-thought, but you can observe what it actually does: its action trajectory.
    • That action trajectory is converted into the privileged information: tool names, tool calls with arguments, or a compact hint.
    • The teacher-mode model sees the task/history plus this privileged trace in the prompt. The student-mode model only sees the task/history in its prompt.
    • The teacher rolls out a trajectory and gets an RL reward4.
    • The student is then trained with teacher forcing: calculating loss based on how likely it would be to predict the actual next token the teacher generated.
    • The teacher and student losses are combined and applied to the single shared set of weights.

    As the authors continue, it doesn’t even require a closed model to distill from. Other kinds of privileged information can help you do the same trick, which is the second variant of their recipe. If you don’t have an outside source but you do know some bonus details (e.g. hints on how to solve it, or critiques on prior attempts) you can pass them into the teacher:

    • Let the student roll out, without the privileged information.
    • Then ask the informed teacher how compatible the student’s tokens were with what the teacher would have done.

    The discussion about distillation has focused on the idea of stealing some kind of secret knowledge. What this method really shows though is that distillation is about turning information that the model will not have at test time into behaviors it will have.

    Like any good teacher, having a sense of how to get to the answer is going to make it easier to help your student. The “on-policy” part here is that the student and teacher are the same, the difference is the teacher is reading ahead in the study guide.

    As tasks get longer, tool use gets richer, and agent traces get more valuable. The question is probably less “can labs hide the model’s reasoning?” and more “what clues can you train on?”

    1. And/or marketing. ↩︎
    2. Notably including the one I work at! ↩︎
    3. Other good reads are the Thinky Blog and “Self-Distilled Reasoner”, which was released a few days before this, and is where the name comes from! ↩︎
    4. With a KL penalty that keeps it from drifting too far from the student. ↩︎