Why I Stopped Trusting Validation Loss

My last post was about failing to validate a world model. It assumed something I never argued for, namely that you need a world model to evaluate a driving policy in the first place. Why not just hold out some data, measure the loss, and pick the policy with the lower number?

Two things I read this month say that doesn’t work. They come at it from completely different angles and neither is about driving, but they land in the same place, and it’s a place that makes the problem I couldn’t solve considerably more urgent than I’d been treating it.

The first one, four things that shouldn’t happen

Seohong Park, an RL researcher at Berkeley, wrote up four mysteries he ran into after moving into robotics. His starting observation is that real demonstration data behaves nothing like the data in the benchmarks everybody tests on. Human demos are narrow, temporally correlated, and smoothly random in a way scripted benchmark data isn’t. So he built a simulator that fakes those properties. Piecewise Hermite splines, randomized grasp angles and contact points, occasional mistakes and recoveries. Then he reproduced four phenomena that shouldn’t be reproducible.

Overfitting helps. Train longer, performance keeps climbing, while validation flow loss climbs too. That’s not a subtle effect, and it gets worse. A 10,000-episode dataset sometimes beats a 50,000-episode one drawn from the same distribution, averaged over four seeds with four independently sampled datasets each.

Acting blind beats reacting. Have the policy predict 25 moves and execute all of them without looking again, and it works. Have it predict one move, look at the result, predict the next, and the policy doesn’t touch the cube. This is with infinite data, since he regenerates fresh demos continuously and trains on each batch once, so it isn’t overfitting or scarcity. Adding history conditioning, which should close the expressivity gap, makes it worse, even though the history-conditioned policies fit the data better.

The policy has to be enormous. A 37-dimensional, single-task, state-based pick-and-place needs residual MLPs around eight layers deep and 4,096 units wide before it works well. Doubling the width to 8,192 improves it further, so that isn’t even the ceiling. That’s on the order of half a billion parameters for a task where you’d reach for three layers of 512 without thinking about it.

Feature scaling matters with infinite data. Two feature sets containing identical information, differing only in per-dimension scaling, produce nearly identical flow losses and clearly different performance. Scaling up the object’s xyz coordinates relative to everything else improves things further.

His diagnosis for all four comes down to test-time distribution shift. The policy drifts into states the training data never covered, and every one of these mysteries is really about behavior out there rather than behavior on the data you have. His read is that this may be unfixable except by scaling until nothing is out-of-distribution, which is his explanation for why LLMs don’t suffer from it the way robot policies do.

The bit I keep coming back to is smaller than any of the four. It’s an aside where he notes that the metric you’d actually want is the error between generated and target actions under the state distribution the policy itself induces at test time, rather than under the dataset’s state distribution. But that metric is generally inaccessible, since you don’t have target labels for the test-time states.

That’s the whole problem in two sentences.

The second one, the thing everyone believed was wrong

The other is a paper out of CMU, MIT and Toyota Research called Much Ado About Noising. It asks why diffusion and flow policies beat plain regression on manipulation benchmarks. The standard answer has been multi-modality. A human might sensibly do several different things from the same state, regression averages them into mush, and a generative model can represent the full distribution.

They tested it and it’s wrong. The key methodological move is that they hold architecture constant. They implement regression policies inside the exact same backbones the diffusion papers use, by setting the noise and time index to zero, which as far as I can tell nobody had done before. Under that comparison, across 28 benchmarks, generative and regression policies come out level almost everywhere. Gaps show up only on a handful of high-precision tasks.

Then they go after multi-modality directly, three ways.

First they sample many actions at states where several answers should obviously be valid. Push-T, where the robot shoves a T-shaped block into a target outline, has configurations where pushing from the left and pushing from the right are equally sensible. Franka Kitchen has independent subtasks, so opening the microwave first and turning the burner first are both fine. These are the two places in the standard benchmark suite where distinct modes should be least controversial. They get single clusters at both.

Second, they replace stochastic sampling with the mean action. If the policy really were representing several distinct options, averaging them would produce something between two valid choices, which is usually invalid. Performance barely moves.

Third, they rebuild the dataset from a deterministic policy in a deterministic environment, so multi-modality is impossible even in principle. The gap between generative and regression policies persists anyway.

What’s actually doing the work turns out to be two things. Injecting noise during training, and generating the output over several steps where every intermediate step gets supervised. Both are procedural habits rather than distributional ones. Keep them, drop the distribution-fitting entirely, and you get MIP, an ordinary regression policy that predicts a single action, refines it over two supervised passes, and matches flow across state, image and point-cloud benchmarks at two function evaluations instead of nine.

The part I found most useful comes later, when they look for why MIP and flow win. They expected better reconstruction on held-out data, and instead found that every method achieves roughly the same validation loss. What separates the good policies from the bad ones is a metric they had to invent, which they call manifold adherence. Roughly, whether the action you produce is still plausible when the observation is off-distribution. Validation loss doesn’t discriminate. Off-manifold error does.

Worth being precise about the metric, because “plausible” is doing a lot of work. They perturb expert trajectories to reach states slightly off the training distribution, ask the policy what it would do there, and measure how much of the predicted action falls outside the span of expert actions at nearby states. Only the component orthogonal to that span counts. Distance to the nearest expert action doesn’t, which is what you’d probably assume they meant.

The numbers on state-based Tool-Hang, a high-precision insertion task from Robomimic and the one where the performance gap between methods is widest, averaged over three architectures and 32 states.

Metric	Regression	SF	RR	MIP	Flow
Validation L2	0.073	0.071	0.062	0.069	0.074
Off-manifold L2	0.058	0.061	0.057	0.043	0.032

Validation loss sits between 0.062 and 0.074 across five methods whose success rates run from 0.31 to 0.74. Residual regression has the best validation loss of the five and the worst performance of the five.

One caveat that a fast read would miss. Off-manifold error cleanly separates flow and MIP from the other three, but it doesn’t order the three below them. RR has the lowest off-manifold error of that group at 0.057, and the worst success rate at 0.31. Straight flow has the highest error at 0.061 and the best success rate of the three at 0.54. The metric identifies a property the winners share. It isn’t a dial where a lower number means a better policy.

MIP’s objective is two regression terms, both aiming at the true action. The first is ordinary regression from the observation. The second asks the same network to recover that action from a noised version of it, at one fixed noise level. At inference you run the first, feed its output into the second, and stop. Two function evaluations.

Flow instead predicts a velocity field across a continuum of noise levels and integrates it. That continuum is the distributional part, and MIP doesn’t have it. Everything MIP kept is procedural. The noise and the second pass survive, the distribution fitting doesn’t, and performance holds.

They’re also honest that they can’t explain the mechanism. There’s theory saying generative models learn to project onto the data manifold, but nothing explaining why that bias would be stronger than a well-trained regression model’s, and they say so plainly rather than gesturing at it.

What they agree on

Two groups, different problems, different methods, neither one about driving. Park is chasing why robot policies behave strangely on human data. The CMU group is auditing a design choice everybody copied without checking. They arrive at the same finding from opposite directions.

The number you compute offline does not tell you which policy is better.

Park watches validation loss rise while performance improves. The CMU group finds identical validation losses across methods with very different success rates, and has to build a new metric because the obvious one doesn’t discriminate at all.

I don’t think either paper frames this as its headline result. For Park it’s the thread connecting four mysteries. For the CMU group it’s a step on the way to explaining MIP. But it’s the part that matters most to me, because if offline metrics don’t rank policies, then you have to actually run the policy to find out which one is better.

That’s the distinction I flagged above. Scoring a policy against a recording is cheap, but the recording plays out the same way no matter what the policy says, so you never see the consequences of its decisions. Letting the policy’s choices change what happens next means either real miles, which are expensive and still can’t answer what-if questions, or a simulator you trust.

Which is where my last few weeks went.

The part that’s uncomfortable

The whole argument for generative world models as evaluation substrate rests on this. You can’t answer “what would have happened if the policy had swerved” from a recording, and rare events are too rare to collect at the volume closed-loop evaluation needs. Fine. But that argument only bites if the offline alternative genuinely doesn’t work, and I’d been treating that as background rather than something to establish. These two results establish it better than I could have.

So closed-loop evaluation isn’t a nicer way to do this. It’s the only way. And the thing standing between us and closed-loop evaluation at scale is whether anyone can tell a trustworthy generated rollout from an untrustworthy one, which is the question I spent a week failing to answer.

That’s a worse position than “I couldn’t build the checker.” It’s “I couldn’t build the checker, and there’s no fallback.”

Three groups, three scorers, none audited

The pattern shows up a third time, inside the CMU paper itself. They needed a way to tell whether an action was plausible off-distribution. Nothing existing worked, so they built manifold adherence, and then wrote plainly that they can’t explain why it works. There’s theory saying generative models learn to project onto the data manifold, but nothing explaining why that bias should be stronger than a well-trained regression model’s.

That’s the same shape as everything in my last post. An automatic evaluator standing in for judgment you can’t collect at scale, load-bearing for a central claim, and unmeasured. Theirs might be excellent. Mine turned out to be blind in three separate ways, and I only found out because I went looking.

Park needs a metric he says is inaccessible. The CMU group builds one and can’t account for it. I built one and found it couldn’t see the thing it was built to detect. Three groups, three attempts, three different problems, and in none of the three cases does anyone know how much to trust the instrument.

Where this leaves me

Not anywhere different, practically. I’m still building a geometric consistency score in a synthetic environment, for the same reason as before. It’s the only place I can validate the instrument against ground truth instead of against another instrument’s opinion.

What changed is how I’d describe the problem to someone who asked why it matters. A week ago I’d have said generated rollouts are useful and we should check whether they’re trustworthy. Now I’d say the field is being pushed toward closed-loop evaluation because the offline alternative doesn’t work, and nobody has an audited instrument for deciding whether a closed-loop rollout is worth believing.

Park’s benchmark is due for release around October, which I’m waiting on. A controlled environment with scripted demonstrations, ground-truth state, and enough realism to reproduce the mysteries is close to what I’d want anyway, and it would be a much better starting point than building the whole thing myself.