Paper Summary: The State-Prediction Separation Hypothesis

"The State-Prediction Separation Hypothesis" Authors here note that complex LLMs are not just doing next-token prediction, but also maintaining an internal state. They find by explicitly separating these functions (simple masking) they can get same performance for cheaper!

Full paper here: https://arxiv.org/abs/2607.01218

Masking mechanism is simple: Every token gets a parallel "predict token" inserted with it. A sliding window includes the last K predict tokens, along with the usual full history of real tokens. Next token prediction is only done on the predict tokens.

There's an inverse way of looking at this: at every token, the model gets a "thinking token". This is the opposite end to having all the "chain of thought" at the end of the input but before the output. An interesting followup idea is: can we get the same result with having thinking space every N tokens? (The algorithm here would need nontrivial adjustments). In some ways, this could look very similar to the Token Superposition paper.

(Token superposition paper:)

N.B. the headline picture of "2.6x more token efficient" is taken from *before* learning rate decay phase (linear decay at the last 10% of steps). There is also a small ~7% inference cost, at least in their implementation Results are still a good and significant gain!

Beyond the paper: in the age of Reinforcement Learning -- Doesn't this remind us of actor-critic architecture? One part of the model tracks the state; the other handles the action. I wonder if this will enable e.g. easy+cheap PPO implementation.

This is a really well-written, easy-to-read paper! It's my favourite kind of paper: we had a simple intuitive insight, we found a way to implement it, and it gives us multiplicative gains! Very well done to the authors 😊