Mez Gebre

Why I Stopped Trusting Validation Loss

2026-09-02T00:00:00+00:00

My last post was about failing to validate a world model. It assumed something I never argued for, namely that you need a world model to evaluate a driving policy in the first place. Why not just hold out some data, measure the loss, and pick the policy with the lower number?

Two things I read this month say that doesn’t work. They come at it from completely different angles and neither is about driving, but they land in the same place, and it’s a place that makes the problem I couldn’t solve considerably more urgent than I’d been treating it.

The first one, four things that shouldn’t happen

Seohong Park, an RL researcher at Berkeley, wrote up four mysteries he ran into after moving into robotics. His starting observation is that real demonstration data behaves nothing like the data in the benchmarks everybody tests on. Human demos are narrow, temporally correlated, and smoothly random in a way scripted benchmark data isn’t. So he built a simulator that fakes those properties. Piecewise Hermite splines, randomized grasp angles and contact points, occasional mistakes and recoveries. Then he reproduced four phenomena that shouldn’t be reproducible.

Overfitting helps. Train longer, performance keeps climbing, while validation flow loss climbs too. That’s not a subtle effect, and it gets worse. A 10,000-episode dataset sometimes beats a 50,000-episode one drawn from the same distribution, averaged over four seeds with four independently sampled datasets each.

Acting blind beats reacting. Have the policy predict 25 moves and execute all of them without looking again, and it works. Have it predict one move, look at the result, predict the next, and the policy doesn’t touch the cube. This is with infinite data, since he regenerates fresh demos continuously and trains on each batch once, so it isn’t overfitting or scarcity. Adding history conditioning, which should close the expressivity gap, makes it worse, even though the history-conditioned policies fit the data better.

The policy has to be enormous. A 37-dimensional, single-task, state-based pick-and-place needs residual MLPs around eight layers deep and 4,096 units wide before it works well. Doubling the width to 8,192 improves it further, so that isn’t even the ceiling. That’s on the order of half a billion parameters for a task where you’d reach for three layers of 512 without thinking about it.

Feature scaling matters with infinite data. Two feature sets containing identical information, differing only in per-dimension scaling, produce nearly identical flow losses and clearly different performance. Scaling up the object’s xyz coordinates relative to everything else improves things further.

His diagnosis for all four comes down to test-time distribution shift. The policy drifts into states the training data never covered, and every one of these mysteries is really about behavior out there rather than behavior on the data you have. His read is that this may be unfixable except by scaling until nothing is out-of-distribution, which is his explanation for why LLMs don’t suffer from it the way robot policies do.

The bit I keep coming back to is smaller than any of the four. It’s an aside where he notes that the metric you’d actually want is the error between generated and target actions under the state distribution the policy itself induces at test time, rather than under the dataset’s state distribution. But that metric is generally inaccessible, since you don’t have target labels for the test-time states.

That’s the whole problem in two sentences.

The second one, the thing everyone believed was wrong

The other is a paper out of CMU, MIT and Toyota Research called Much Ado About Noising. It asks why diffusion and flow policies beat plain regression on manipulation benchmarks. The standard answer has been multi-modality. A human might sensibly do several different things from the same state, regression averages them into mush, and a generative model can represent the full distribution.

They tested it and it’s wrong. The key methodological move is that they hold architecture constant. They implement regression policies inside the exact same backbones the diffusion papers use, by setting the noise and time index to zero, which as far as I can tell nobody had done before. Under that comparison, across 28 benchmarks, generative and regression policies come out level almost everywhere. Gaps show up only on a handful of high-precision tasks.

Then they go after multi-modality directly, three ways.

First they sample many actions at states where several answers should obviously be valid. Push-T, where the robot shoves a T-shaped block into a target outline, has configurations where pushing from the left and pushing from the right are equally sensible. Franka Kitchen has independent subtasks, so opening the microwave first and turning the burner first are both fine. These are the two places in the standard benchmark suite where distinct modes should be least controversial. They get single clusters at both.

Second, they replace stochastic sampling with the mean action. If the policy really were representing several distinct options, averaging them would produce something between two valid choices, which is usually invalid. Performance barely moves.

Third, they rebuild the dataset from a deterministic policy in a deterministic environment, so multi-modality is impossible even in principle. The gap between generative and regression policies persists anyway.

What’s actually doing the work turns out to be two things. Injecting noise during training, and generating the output over several steps where every intermediate step gets supervised. Both are procedural habits rather than distributional ones. Keep them, drop the distribution-fitting entirely, and you get MIP, an ordinary regression policy that predicts a single action, refines it over two supervised passes, and matches flow across state, image and point-cloud benchmarks at two function evaluations instead of nine.

The part I found most useful comes later, when they look for why MIP and flow win. They expected better reconstruction on held-out data, and instead found that every method achieves roughly the same validation loss. What separates the good policies from the bad ones is a metric they had to invent, which they call manifold adherence. Roughly, whether the action you produce is still plausible when the observation is off-distribution. Validation loss doesn’t discriminate. Off-manifold error does.

Worth being precise about the metric, because “plausible” is doing a lot of work. They perturb expert trajectories to reach states slightly off the training distribution, ask the policy what it would do there, and measure how much of the predicted action falls outside the span of expert actions at nearby states. Only the component orthogonal to that span counts. Distance to the nearest expert action doesn’t, which is what you’d probably assume they meant.

The numbers on state-based Tool-Hang, a high-precision insertion task from Robomimic and the one where the performance gap between methods is widest, averaged over three architectures and 32 states.

Metric	Regression	SF	RR	MIP	Flow
Validation L2	0.073	0.071	0.062	0.069	0.074
Off-manifold L2	0.058	0.061	0.057	0.043	0.032

Validation loss sits between 0.062 and 0.074 across five methods whose success rates run from 0.31 to 0.74. Residual regression has the best validation loss of the five and the worst performance of the five.

One caveat that a fast read would miss. Off-manifold error cleanly separates flow and MIP from the other three, but it doesn’t order the three below them. RR has the lowest off-manifold error of that group at 0.057, and the worst success rate at 0.31. Straight flow has the highest error at 0.061 and the best success rate of the three at 0.54. The metric identifies a property the winners share. It isn’t a dial where a lower number means a better policy.

MIP’s objective is two regression terms, both aiming at the true action. The first is ordinary regression from the observation. The second asks the same network to recover that action from a noised version of it, at one fixed noise level. At inference you run the first, feed its output into the second, and stop. Two function evaluations.

Flow instead predicts a velocity field across a continuum of noise levels and integrates it. That continuum is the distributional part, and MIP doesn’t have it. Everything MIP kept is procedural. The noise and the second pass survive, the distribution fitting doesn’t, and performance holds.

They’re also honest that they can’t explain the mechanism. There’s theory saying generative models learn to project onto the data manifold, but nothing explaining why that bias would be stronger than a well-trained regression model’s, and they say so plainly rather than gesturing at it.

What they agree on

Two groups, different problems, different methods, neither one about driving. Park is chasing why robot policies behave strangely on human data. The CMU group is auditing a design choice everybody copied without checking. They arrive at the same finding from opposite directions.

The number you compute offline does not tell you which policy is better.

Park watches validation loss rise while performance improves. The CMU group finds identical validation losses across methods with very different success rates, and has to build a new metric because the obvious one doesn’t discriminate at all.

I don’t think either paper frames this as its headline result. For Park it’s the thread connecting four mysteries. For the CMU group it’s a step on the way to explaining MIP. But it’s the part that matters most to me, because if offline metrics don’t rank policies, then you have to actually run the policy to find out which one is better.

That’s the distinction I flagged above. Scoring a policy against a recording is cheap, but the recording plays out the same way no matter what the policy says, so you never see the consequences of its decisions. Letting the policy’s choices change what happens next means either real miles, which are expensive and still can’t answer what-if questions, or a simulator you trust.

Which is where my last few weeks went.

The part that’s uncomfortable

The whole argument for generative world models as evaluation substrate rests on this. You can’t answer “what would have happened if the policy had swerved” from a recording, and rare events are too rare to collect at the volume closed-loop evaluation needs. Fine. But that argument only bites if the offline alternative genuinely doesn’t work, and I’d been treating that as background rather than something to establish. These two results establish it better than I could have.

So closed-loop evaluation isn’t a nicer way to do this. It’s the only way. And the thing standing between us and closed-loop evaluation at scale is whether anyone can tell a trustworthy generated rollout from an untrustworthy one, which is the question I spent a week failing to answer.

That’s a worse position than “I couldn’t build the checker.” It’s “I couldn’t build the checker, and there’s no fallback.”

Three groups, three scorers, none audited

The pattern shows up a third time, inside the CMU paper itself. They needed a way to tell whether an action was plausible off-distribution. Nothing existing worked, so they built manifold adherence, and then wrote plainly that they can’t explain why it works. There’s theory saying generative models learn to project onto the data manifold, but nothing explaining why that bias should be stronger than a well-trained regression model’s.

That’s the same shape as everything in my last post. An automatic evaluator standing in for judgment you can’t collect at scale, load-bearing for a central claim, and unmeasured. Theirs might be excellent. Mine turned out to be blind in three separate ways, and I only found out because I went looking.

Park needs a metric he says is inaccessible. The CMU group builds one and can’t account for it. I built one and found it couldn’t see the thing it was built to detect. Three groups, three attempts, three different problems, and in none of the three cases does anyone know how much to trust the instrument.

Where this leaves me

Not anywhere different, practically. I’m still building a geometric consistency score in a synthetic environment, for the same reason as before. It’s the only place I can validate the instrument against ground truth instead of against another instrument’s opinion.

What changed is how I’d describe the problem to someone who asked why it matters. A week ago I’d have said generated rollouts are useful and we should check whether they’re trustworthy. Now I’d say the field is being pushed toward closed-loop evaluation because the offline alternative doesn’t work, and nobody has an audited instrument for deciding whether a closed-loop rollout is worth believing.

Park’s benchmark is due for release around October, which I’m waiting on. A controlled environment with scripted demonstrations, ground-truth state, and enough realism to reproduce the mysteries is close to what I’d want anyway, and it would be a much better starting point than building the whole thing myself.

Three Ways to Think You've Validated a World Model

2026-08-29T00:00:00+00:00

I set out to test a distillation question. I ended up writing a paper about why that question was premature.

The plan was simple, on paper. Vista, OpenDriveLab’s action-conditioned driving world model, generates a rollout: give it a starting frame and a trajectory, and it imagines the next 25 frames of driving. It ships with 50 sampling steps by default. Distill it down to 4, or 2, or 1, and you get a much cheaper rollout, if the quality holds. The question I wanted to answer was whether a policy-ranking verdict survives that distillation. Rank two driving policies with Vista: does the ranking stay the same at 50 steps and at 2?

That’s still an interesting question. I haven’t answered it. Instead I spent most of this project discovering that the question has two silent prerequisites nobody in the driving-world-model literature seems to state out loud, and neither one held up when I actually checked. This is the story of finding that out, what it cost to check properly instead of assuming, and what I’m building next because of it.

Prerequisite one: does the model even listen to you?

Vista takes an 8-number trajectory: four future waypoints, each a (lateral, forward) pair in meters, at half-second intervals. Feed it a left turn, it should generate a left turn. Feed it a right turn, a right turn. That’s the entire premise of using it for closed-loop policy evaluation. The model’s response has to be attributable to the policy’s actual decision.

First bug, and if you’re adapting Vista to anything other than nuScenes you will hit this too: the axis order is (lateral, forward), not (forward, lateral). The failure mode is nasty precisely because it doesn’t look like a bug. You get blurry, indecisive turning behavior, output that reads as “the model doesn’t turn very well” instead of “the input is wrong.” I only caught it by triangulating three independent sources: the paper, the original author’s own preprocessing script, and a direct statistical decode of the nuScenes annotation data Vista was trained on. If your Vista turns look weak or wrong, check your axis order before you conclude anything about the model.

Fixing that revealed a second problem, and this one isn’t a bug. It’s arithmetic. Vista encodes each trajectory value through a sinusoidal embedding built for diffusion timesteps, which run in the hundreds to thousands. Real driving trajectories are single-digit meters, three orders of magnitude smaller than what the embedding was tuned for. I checked whether that actually matters instead of assuming it either way, and it does. Cosine similarity between the embeddings of +3m and -3m (same magnitude, opposite direction, exactly the pair a left-vs-right decision depends on) is 0.735. Cosine similarity between +3m and +15m (same direction, five times the magnitude) is 0.662. Lower. At realistic driving scale, Vista’s own conditioning discriminates magnitude more reliably than direction. Behavioral tests agreed: no reliable directional response at realistic magnitude, and only a real but seed-dependent effect at ten times the scale.

I didn’t introduce that bug. It’s a property of Vista’s own embedding design, and you can verify it yourself with ten lines of numpy and no GPU. It means the first prerequisite, that a generated rollout is attributable to the policy’s action, doesn’t cleanly hold at the magnitudes a real policy actually produces.

Prerequisite two: can anyone tell if a rollout is any good?

Say the conditioning problem didn’t exist. There’s a second prerequisite underneath the ranking question: something has to look at a generated rollout and say whether it’s a fair test of the policy or an artifact of the generator falling apart. A rollout can look plausible frame-to-frame while a static pole bends, or the ego vehicle’s implied speed does something a real car can’t do. So I built a geometric checker: VGGT, a feed-forward 3D reconstruction model, validated first against real Waymo pose (11.5cm mean position error, 0.987 heading correlation on a real 49-degree turn) and cross-checked against real lidar depth.

Then I tried to use it, and this is where the project turned into a different paper. Four times a check looked like it had found a real problem. Three of those four turned out, once I built the control I should have built from the start, to be measuring something else entirely.

The sharpest one: a scale-consistency score checking whether a rollout’s implied camera speed evolves smoothly. On a real, dead-straight clip it looked clean. On a real 49-degree turn, at a 5-frame averaging window, it reported 4.6% deviation, comfortably under the 5% threshold I’d pinned in advance. A clean pass. The actual ground truth, on the exact same clip, was 13.3%. A clear fail. The checker wasn’t being cautious. It was confidently wrong: it reported smoother implied camera motion than the real clip actually had, and widening the averaging window made things worse, which ruled out “just needs more data” as the explanation.

Then I ran the same statistic on the straight clip and found it was wrong there too, just in the opposite direction: jumpier than reality on the easy clip, smoother than reality on the hard one. Same estimator, two opposite failure directions. I checked this wasn’t a coincidence by sweeping the averaging window from 3 to 15 frames on both clips. On the straight clip, the gap to ground truth shrank as the window widened, the signature of high-frequency noise averaging itself out. On the turning clip, the gap grew as the window widened, the signature of a low-pass filter smoothing away real, slower variation instead of noise. That crossover, not just the two headline numbers, is why I call it a bandpass distortion rather than a simple too-strict or too-lax bias.

My best explanation, and I want to be clear it’s an explanation rather than something I isolated with an ablation: a joint-attention model is trained to build one temporally consistent story out of the whole clip, and that’s exactly the property that would make it bad at flagging temporal inconsistency, since flagging it means contradicting the story it just built.

The near-miss that would have been a confidently wrong published result

Before I trusted that finding, I hit a smaller, uglier version of the same problem. It’s the one I’d want another engineer to see before they publish a negative result of their own. Extracting the recovered heading change on that same turning clip, my first pass returned -1.6 degrees against a ground truth of 49.0. Wrong sign, off by an order of magnitude. Read on its own, that’s a near-total failure to recover the turn, and it would have been easy to believe, because it matched a limitation I half-expected going in.

It wasn’t real. The position trajectory, computed completely independently, visibly curved and matched ground truth to 11.5cm. A straight predicted path can’t be rigidly aligned onto a genuinely curved one and look that good, so the heading number and the position number couldn’t both be telling the truth. The bug was a coordinate-convention mismatch. I’d extracted heading by reading yaw directly off VGGT’s rotation matrix under the wrong axis convention: Waymo’s vehicle-frame assumption, applied to OpenCV’s camera-frame output. The number that came out wasn’t heading with some noise on it. It was a completely different physical quantity wearing heading’s name.

The generalizable lesson isn’t “check your axis conventions,” though that’s true here too. It’s about what to do when two independent measurements of the same thing disagree: trust the one with fewer assumptions baked in. Position, computed from consecutive frame deltas, has no rotation-matrix convention to get wrong. Heading, decoded from a rotation matrix, does. A plausible-looking negative result that happens to confirm something you already suspected is exactly the shape of finding that turns into a confidently wrong published claim, if you have no independent check to catch it.

The pattern that showed up three times

By the time I’d chased down why a sparse-point rigidity score kept passing rollouts it shouldn’t have, I’d found the same design mistake twice in my own code. The first version treated any point that never reached enough tracking confidence as simply excluded from scoring. That’s the wrong default for this metric specifically, because low confidence isn’t independent of rigidity. Thin, high-frequency structure like poles and wires is low-confidence by construction, so the filter removes exactly the structure most likely to reveal a problem. A rollout falling apart geometrically and a rollout with no data at all look identical under a pooled pass/fail statistic. The fix: report the unscoreable fraction as a first-class output.

The second version, one level deeper, evaded the exact fix I’d just built. It used an adaptive confidence threshold, set to each frame’s own median. That sounds principled. It generalizes across clips with different absolute confidence scales. It’s actually the same bug wearing a disguise: the threshold gets set by whatever surface happens to dominate a given frame, so a perfectly trackable point can fail simply by sharing a frame with something more confident elsewhere. Reporting the unscoreable fraction within one run doesn’t catch this, because the bias doesn’t look like missing data. It looks like a systematic skew toward whichever surface is nearest or most reflective, invisible unless you go looking for it specifically.

Here’s what that subtle deformation actually looks like, next to the real, unwarped frame and the large positive control used to confirm any checker in this story could still fire on something obvious:

The third time is the one that convinced me it isn’t just my code. I ran the same subtle, carefully controlled deformation against GeCo, a published geometric consistency checker I had no hand in writing. It missed the deformation completely: no separation from the unwarped clip. Before trusting that null result, I ran a positive control, a large, obvious deformation, the kind GeCo’s own published validation uses to confirm the metric fires at all. It fired clearly. The pipeline worked, and the subtle warp genuinely sat inside the range GeCo reads as unremarkable. Then, instead of just accepting “not much signal,” I checked why the response was smaller than the visual deformation suggested, and found a clean, monotonic dose-response: pixels displaced under 5px lost almost no coverage, pixels displaced over 90px lost twenty times as much. The most-deformed pixels weren’t scoring badly. They were disappearing from the denominator, in proportion to how deformed they were. Silent selection again, in a system I didn’t write, using a motion model I didn’t train.

Twice in code I control, found by fixing them. Once in code I don’t control, found only because I ran a positive control to rule out a wiring bug and got a dose-response instead. If there’s one operational takeaway from this whole project, it’s this: a scoring pipeline should never report a number without also reporting what fraction of the data that number is actually based on. “Zero problems detected” and “the checker had no opinion” look identical in a summary table. They are not the same claim.

What honoring a pre-committed stopping rule actually costs

Partway through, I pinned a rule before I had any results to argue about: any geometric score gets at most two failed validation attempts. Fail twice, and it drops from a pass/fail gate to a reported diagnostic. I wrote that down because the rigidity score had already burned most of a week: five separate validation attempts, each one individually reasonable, each one turning up one more confound worth ruling out before the next. That’s exactly how a week disappears one week into a fourteen-week plan. Not one bad decision, just a chain of locally justified “one more check.”

The rule did its job, in a way I didn’t love in the moment. Scale consistency’s redesigned, self-contained version, the one that doesn’t need ground truth, which is the actual production requirement, got exactly two attempts. The first found a false pass on the turning clip. The second, a window sweep, confirmed it wasn’t a window-size artifact: VGGT’s own reported deviation trended toward zero as the window widened, while the real deviation stayed substantial. Two strikes. Demoted. No third redesign, no “but this version is different.” Writing the rule down before I had a result to protect is the only reason I actually stopped there instead of finding a reason this particular case was special.

Where this leaves the project

I did not answer “does a policy-ranking verdict survive distillation.” I found that the question presupposes two things, attributable conditioning and a trustworthy geometric checker, and neither one holds without real, load-bearing work first. That’s the paper I ended up writing, Three Ways to Think You’ve Validated a World Model. It argues that building trustworthy automatic evaluators, not sampling cost and not generation fidelity, is the actual binding constraint on using generative world models for driving evaluation. I also put together a companion notebook repo so the numbers above are checkable rather than just asserted: no GPU required for four of the five notebooks, and the fifth needs nothing more than a laptop and 60MB of bundled data. I’m not releasing either publicly yet, but if you want to read the paper or run the notebooks yourself, reach out and I’ll send them over.

What’s next follows directly from what killed this project’s original scope. The driving geometric checker never got the one thing it actually needed: ground truth for the geometric coherence of a generated scene. The scene never existed, so there was nothing to validate the instrument against except another model’s opinion, and that instrument turned out to be structurally blind to exactly what it was asked to measure. A synthetic environment solves this by construction. Control the world, and you know its exact geometry, poses, and depth for every frame, so a geometric reward can be validated directly against truth instead of against another instrument’s opinion.

The plan is staged, so it produces a result at every stage instead of betting everything on the last one. Stage one: build a geometric-consistency score in a synthetic environment and validate it against known ground truth, the step the driving version never reached, and report its sensitivity floor, the smallest deformation it can detect. Stage two: use that score as a rejection filter on a batch of generated rollouts and check whether downstream conclusions actually change between the filtered and unfiltered library. That second half might be the whole result on its own. If filtering changes nothing downstream, that’s a real finding about how much geometric coherence actually matters, and it costs no training at all. Stage three, only if the first two justify it: use the validated score as an RL reward and fine-tune a generative model against it directly.

I don’t know yet whether a synthetic result transfers to driving. Thin structures, weak parallax, and sparse texture are domain-specific failure modes a clean synthetic environment won’t reproduce, and I’m not pretending otherwise going in. But proving the mechanism works somewhere you can actually check it beats assuming it works and finding out the hard way, which is more or less the summary of everything above.

Audio Classification: A Weekend of Experiments (FAAAAHH!)

2026-02-17T00:00:00+00:00

I went down an audio classification rabbit hole. What started as “how hard can ESC-50 be?” turned into a weekend of building CNNs, Transformers, and finally catching up on self-supervised learning, only a few years behind everyone else.

This post documents everything I tried, what worked, what didn’t, and the lessons learned.

The Dataset: ESC-50

The ESC-50 dataset contains:

2,000 audio clips (5 seconds each)
50 classes of environmental sounds
5 folds for cross-validation

With only 40 samples per class, this is a challenging dataset that rewards good representations over brute-force memorization.

The Supervised Baseline

ResNet-34 from Scratch

Classic approach: convert audio to mel spectrograms and treat them as images for a CNN.

batch_size = 32
learning_rate = 1e-3
optimizer = AdamW(weight_decay=0.01)
scheduler = OneCycleLR

Result: 80.5% accuracy

Test-Time Augmentation

I noticed validation performance was inconsistent, so I added TTA, averaging predictions across multiple augmented versions of each test sample.

def predict_with_tta(model, spectrogram, n_augments=5):
    predictions = []
    for _ in range(n_augments):
        aug_spec = apply_augmentation(spectrogram)
        pred = model(aug_spec)
        predictions.append(pred)
    return torch.stack(predictions).mean(dim=0)

Result: 83.5% accuracy (+3% from TTA alone!)

Transfer Learning Detour

Before diving into SSL, I tried transfer learning with ImageNet-pretrained EfficientNet-B0.

Result: 81.5% accuracy

This performed worse than my from-scratch ResNet. ImageNet features (edges, textures, objects) don’t transfer perfectly to spectrograms. Good to know.

Self-Supervised Learning Experiments

This is where things got interesting. I wanted to understand why and how SSL works, not just use pretrained models.

SimCLR (Contrastive Learning)

The idea: learn representations by pulling augmented views of the same audio together while pushing different audios apart.

Audio → Mel Spectrogram → ResNet Encoder → Projection Head → Contrastive Loss

Key components:

NT-Xent Loss: Normalized Temperature-scaled Cross Entropy
Projection Head: 512 → 512 → 128 (discarded after pretraining)
Temperature: 0.5

Result: 82.5% accuracy

But here’s the catch: the contrastive task itself reached 98% accuracy during pretraining.

The model was solving a nearly trivial task.

Environmental sounds in ESC-50 are already highly separable in the spectral domain (dog bark vs chainsaw vs rain, etc.). With relatively mild augmentations, the identity of each sound barely changes. That means the encoder can solve the contrastive objective using coarse cues (energy bands, temporal envelope) instead of learning robust, invariant representations.

In other words, the model learned:

“these sounds are different”

instead of:

“these two transformed versions of the same sound are meaningfully the same”

Lesson: Contrastive learning only works when the task is hard enough. You need augmentations and negatives that force the model to learn invariances, not shortcuts.

Even with this limitation, the learned representation still reached 82.5% accuracy, which is competitive with standard supervised CNN baselines on ESC-50.

Masked Prediction (BEATs-style)

Inspired by BERT and BEATs: hide patches of the spectrogram and predict discrete tokens.

Note: This was a simplified, educational implementation to understand the BEATs framework, not a full reproduction. The real BEATs includes iterative tokenizer refinement, larger models (90M+ params), and training on millions of samples. My goal was to grasp the core concepts: patch embeddings, masking, and discrete token prediction.

Spectrogram → Patch Embedding → Encoder → Predict Masked Tokens

The approach:

Split spectrogram into patches (16×16)
Randomly mask 40% of patches
Predict the cluster ID of masked patches

The tokenizer (codebook):

K-means clustering on spectrogram patches
Each patch gets assigned to nearest centroid
Model predicts which cluster the masked patch belongs to

# Tokenizing a patch
distances = ||patch - codebook||²  # Distance to each centroid
token = argmin(distances)          # Closest centroid = token ID

Result: 74.5% accuracy (with CNN encoder)

Underperformed the supervised baseline. Hmm.

Transformer-BEATs

I tried a full Transformer encoder instead of CNN.

Architecture:

6 Transformer layers
256 embedding dimension
8 attention heads
~5M parameters

Result: 40% accuracy (nearly random, FAAAHH!!)

This was humbling. Transformers need massive amounts of data. With only 1,600 training samples, it couldn’t learn meaningful patterns. CNNs have stronger inductive biases that help with limited data.

Call An Ambulance But Not For Me GIFfrom Call An Ambulance But Not For Me GIFs

The BEATs Revelation

After all my experiments, I tried Microsoft’s BEATs model, pretrained on AudioSet (2 million+ clips).

I tested two approaches:

Frozen encoder: Only train a new classifier head on top.
Result: 94.50% accuracy

Fine-tuned with differential learning rates:

param_groups = [
    {'params': model.encoder.parameters(), 'lr': 1e-5},    # Slow for pretrained
    {'params': model.classifier.parameters(), 'lr': 1e-3}  # Fast for new head
]

Result: 95.25% accuracy 🎯

The frozen approach gets you 94.5% with minimal compute. The AudioSet pretraining is that good. Fine-tuning squeezes out another 0.75%, which matters if you’re chasing leaderboards but honestly? Frozen is probably fine for most use cases.

Even though I knew I probably wouldn’t make a dent against the pretrained model, I had to try it.

Things I Had to Figure Out

What are “tokens” in audio? In BEATs, tokens are discrete IDs representing audio patterns. K-means clustering groups similar spectrogram patches, and each patch gets a cluster ID. It’s like creating a vocabulary of audio building blocks.

Why iterate on the tokenizer? Clever trick: the tokenizer and model improve each other. Each iteration creates more meaningful tokens, forcing the model to learn finer distinctions.

Why two learning rates? Differential rates balance preservation and adaptation. Without this, you either destroy pretrained features (high LR) or the classifier never converges (low LR).

Final Results

Method	Accuracy	Approach
BEATs (fine-tuned)	95.25%	Pretrained + differential LR
BEATs (frozen)	94.50%	Pretrained, classifier only
ResNet-34 + TTA	83.50%	Supervised baseline
SimCLR	82.50%	Contrastive SSL
EfficientNet-B0	81.50%	ImageNet transfer
CNN-BEATs	74.50%	Masked prediction
Transformer-BEATs	40.00%	Tomfoolery 💀

The gap between my from-scratch SSL attempts and the pretrained BEATs tells the whole story.

Takeaways

Scale matters enormously for SSL. My weekend experiments couldn’t match 2M-sample pretraining, but they taught me how these methods work.
Contrastive learning needs hard negatives. Trivial pretext tasks don’t force useful representations.
Transformers are data-hungry. CNNs win on small datasets.
TTA is free accuracy. 3% boost for minimal effort.

This is why I love jumping between domains: contrastive learning from NLP, masked prediction from BERT, spectrogram tricks from signal processing. The dots connect in unexpected ways.

The real takeaway? Running these experiments taught me more than any tutorial ever could. Sometimes you have to build the thing yourself to understand why it works, and why it doesn’t.

What surprised me most is that self-supervised learning doesn’t automatically produce meaningful representations. It will happily learn shortcuts if the task allows it. Designing the right objective turns out to matter just as much as the model itself.

I'm Back

2026-02-16T00:00:00+00:00

It’s been a while. Almost nine years since my last post. Life happened, priorities shifted, and this blog sat quietly collecting dust.

A lot has changed. I have a family now, including a 13-month-old daughter who’s been teaching me more about persistence and curiosity than any codebase ever could.

But I’ve missed having a place to think out loud. So here I am, back at it.

No grand promises, just an intention: write more, share ideas, and see where it goes.

Let’s ship.

HomographyNet: Deep Image Homography Estimation

2017-07-21T00:00:00+00:00

Introduction

Today we are going to talk about a paper I read a month ago titled Deep Image Homography Estimation. It is a paper that presents a deep convolutional neural network for estimating the relative homography between a pair of images.

What is a Homography?

In projective geometry, a homography is an isomorphism of projective spaces, induced by an isomorphism of the vector spaces from which the projective spaces derive. - Wikipedia

If you understood that, go ahead and skip directly to the deep learning model discussion. For the rest of us, let’s go ahead and learn some terminology. If you can forget about that definition from Wikipedia for a second, let’s use some anologies to at least get a high-level idea.

Isomorphism?

If you decide to bake a cake; the process of taking the ingredients and following the recipe to create the cake could be seen as morphing. The recipe allowed you to morph/transition from ingredients to a cake. If you’re thinking wtf, just stick with me a bit longer. Now, imagine the same recipe could allow you to take a cake and morph/transition back to the ingredients if you follow it in reverse! In a nutshell, that is the definition of an isomorphism. A recipe that can allow you to transition between the two, loosely speaking.

Isomorphism in mathematics is morphism or a mapping (recipe) that can also give you the inverse, again loosely speaking.

Projective geometry?

I don’t have a clever analogy here, but I’ll give you a simple example to tie it all together. Imagine you’re driving and your dash cam snapped a picture of the road in front of you (let’s call this pic A); at the same time, imagine there was a drone right above you and it also took a picture of the road in front of you (let’s call this pic B). You can see that pic A and B are related, but how? They’re both pictures of the road in front you, only difference is the perspective! The big question….

Is there a recipe/isomorphism that can take you from A to B and vice versa?

There you go, the question you just asked is what a Homography tries to answer. Homography is an isomorphism of perspectives. A 2D homography between A and B would give you the projection transformation between the two images! It is a 3x3 matrix that descibes the affine transformation. Entire books are written on these concepts, but hopefully we now have the general idea to continue.

NOTE

There are some constraints about a homography I have not mentioned such as:

Usually both images are taken from the same camera.
Both images should be viewing the same plane.

Motivation

Where is homography used?

To name a few, the homography is an essential part of the following

Creating panoramas
Monocular SLAM
3D image reconstruction
Camera calibration
Augmented Reality
Autonomous Cars

How do you get the homography?

In traditional computer vision, the homography estimation process is done in two stages.

Corner estimation
Homography estimation

Due to the nature of this problem, these pipelines are only producing estimates. To make the estimates more robust, practitioners go as far as manually engineering corner-ish features, line-ish features etc as mentioned by the paper.

Robustness is introduced into the corner detection stage by returning a large and over-complete set of points, while robustness into the homography estimation step shows up as heavy use of RANSAC or robustification of the squared loss function. - Deep Image Homography Estimation

It is a very hard problem that is error-prone and requires heavy compute to get any sort of robustness. Here we are, finally ready to talk about the question this paper wants to answer.

Is there a single robust algorithm that, given a pair of images, simply returns the homography relating the pair?

HomographyNet: The Model

HomographyNet is a VGG style CNN which produces the homography relating two images. The model doesn’t require a two stage process and all the parameters are trained in an end-to-end fashion!

HomographyNet as descibed in the paper comes in two flavors, classification and regression. Based on the version you decide to use, they have their own pros and cons. Including different loss functions.

The regression network produces eight real-valued numbers and uses the Euclidean (L2) loss as the final layer. The classification network uses a quantization scheme and uses a softmax as the final layer. Since they stick a continuous range into finite bins, they end up with quantization error; hence, the classification network additionally also produces a confidence for each of the corners produced. The paper uses 21 quantization bins for each of the eight output dimensions, which results in a final layer of 168 output neurons. Each of the corner’s 2D grid of scores are interpreted as a distribution.

The architecture seems simple enough, but how is the model trained? Where is the labeled dataset coming from? Glad you asked!

As the paper states and I agree, the simplist way to parameterize a homography is with a 3x3 matrix and a fixed scale. In the 3x3 homography matrix, [H11:H21, H12:H22] are responsible for the rotation and [H13:H23] handle the translational offset. This way you can map each pixel at position [u,v,1] from the image against the homograpy like the figure below, to get the new projected transformation [u',v',1].

However! If you try to unroll the 3x3 matrix and use it as the label (ground truth), you’d end up mixing the rotational and translational components. Creating a loss function that balances the mixed components would have been an unnecessary hurdle. Hence, it was better to use well known loss functions (L2 and Cross-Entropy) and instead figure out a way to re-parameterize the homography matrix!

The 4-Point Homography Parameterization

The 4-Point parameterization is based on corner locations, removing the need to store rotational and translation terms in the label! This is not a new method by any means, but the use of it as a label was clever! Here is how it works.

Earlier we saw how each pixel [u,v,1] was transformed by the 3x3 matrix to produce [u',v',1]. Well if you have at least four pixels where you calculate delta_u = u' - u and delta_v = v' - v then it is possible to reconstruct the 3x3 homography matrix that was used! You could for example use the getPerspectiveTransform() method in OpenCV. From here on out we will call these four pixels (points) as corners.

Training Data Generation

Now that we have a way to represent the homography such that we can use well known loss functions, we can start talking about how the training data is generated.

As a sidenote, H^AB denotes the homorgraphy matrix between A and B

Creating the dataset was pretty straightforward, so I’ll only highlight some of the things that might get tricky. The steps as the describe in the above figure are:

Randomly crop a 128x128 patch at position p from the grayscale image I and call it patch A. Staying away from the edges!
Randomly perturb the four corners of patch A within the range [-rho, rho] and let’s call this new position p'. The paper used rho=32
Compute the homography of patch A using position p and p' and we’ll call this H^AB
Take the inverse of the homography (H^AB)^-1 which equals H^BA and apply that to image I, calling this new image I'. Crop a 128x128 patch at position p from image I' and call it patch B.
Finally, take patch A and B and stack them channel-wise. This will give you a 128x128x2 image that you’ll use as input to the models! The label would be the 4-point parameterization of H^AB.

They cleverly used this five step process on random images from the MS-COCO dataset to create 500,000 training examples. Pretty damn cool (excuse my english).

The Code

If you’d like to see the regression network coded up in Keras and the data generation process visualized, you’re in luck! Follow the links below to my Github repo.

Final thoughts

That pretty much highlights the major parts of the paper. I am hoping you now have an idea of what the paper was about and learned something new! Go read the paper because I didn’t talk about the results etc.

I’d like to think easy future improvements could be to swap out the heavy VGG network for squeezenet! Giving you the improvement of a smaller network. I’ll maybe experiment with this idea and see if I can match or improve on their results. As for possible uses today, I could see this network being used as a cascade classifier. The traditional methods are very compute heavy, so if we could maybe have this network in front to filter out the easy wins, we could cut down on compute cost.

Stay tuned for the next paper and please comment with any corrections or thoughts. That is all folks!

SqueezeDet: Deep Learning for Object Detection

2017-04-21T00:00:00+00:00

Why bother writing this post?

Often, examples you see around computer vision and deep learning is about classification. Those class of problems are asking what do you see in the image? Object detection is another class of problems that ask where in the image do you see it?

Classification answers what and Object Detection answers where.

Object detection has been making great advancement in recent years. The hello world of object detection would be using HOG features combined with a classifier like SVM and using sliding windows to make predictions at different patches of the image. This complex pipeline has a major drawback!

Cons:

Computationally expensive.
Multiple step pipeline.
Requires feature engineering.
Each step in the pipeline has parameters that need to be tuned individually, but can only be tested together. Resulting in a complex trial and error process that is not unified.
Not realtime.

Pros:

Easy to implement, relatively speaking…

Speed becomes a major concern when we are thinking of running these models on the edge (IoT, mobile, cars). For example, a car needs to detect where other cars, people and bikes are to name a few; I could go on… puppies, kittens… you get the idea. The major motivation for me is the need for speed given the constraints that edge computes have; we need compact models that can make quick predictions and are energy efficient.

The SqueezeDet Model

The latest in object detection is to use a convolutional neural network (CNN) that outputs a regression to predict the bounding boxes. This post is about SqueezeDet. I got interested because they used one of my favorite cnn, SqueezeNet! You can read my last post on SqueezeNet if you haven’t yet. To be fair SqueezeDet is pretty much just the YOLO model that uses a SqueezeNet.

Highlevel SqueezeDet

Inspired by YOLO, SqueezeDet is a single stage detection pipeline that does region proposal and classification by one single network. The cnn first extracts feature maps from the input image and feeds it to the ConvDet layer. ConvDet takes the feature maps, overlays them with a WxH grid and at each cell computes K pre-computed bounding boxes called anchors. Each bounding box has the following:

Four scalars (x, y, w, h)
A confidence score ( Pr(Object)xIOU )
C conditional classes

Hence SqueezeDet has a fixed output of WxHxK(4+1+C).

The final step is to use non max suppression aka NMS to filter the bounding boxes to make the final predictions.

The networks regresses and learns how to transform the highest probably bounding box for the prediction. Since there are bounding boxes being generated at each cells of the grid, the top N bounding boxes sorted by the confidence score are kept as the predictions.

The Composite Loss function

The figure above is the four part loss function that makes this entire model possible. Don’t get intimidated by it; let’s take it apart and see how it fits together. Each loss function is described below and highlighted:

Note: The yellow that bleed into the blue loss function is actually suppose to be blue, sorry!

Yellow: Regression of the scalars for the anchors
Green: The confidence score regression which uses IOU of ground and predicted bounding boxes.
Blue: Penalize anchors that are not responsible for detection by dropping their confidence score.
Pink: The cross entropy.

Using SqueezeDet

The authors of the paper did implement the model via TensorFlow! Go check it out on github https://github.com/BichenWuUCB/squeezeDet

Thresholding SqueezeDet

You have to tweak how confident or doubtful you want the model to be; the predictions are centered around K bounding boxes at each cell. So we have to use the top N bouding boxes, sorted by confidence score and then you can do additional thresholding on the class conditional probability score.

Here is an example of a recent project I did where I tweak the params:

Like the paper: N = 64

The image below shows mc.PLOT_PROB_THRESH = 0.1

The image below shows mc.PLOT_PROB_THRESH = 0.5

Final thoughts

Reading through the paper was a real grind. Some math notations were a bit wonky; the paper referenced a lot and it was a recursive process. I literally had to crunch through the entire history of object detection to understand this paper. At the very least I hope you were able to get a high level understanding of this paper. Comment below with any corrections or questions!

MainSqueeze: The 52 parameter model that drives in the Udacity simulator

2017-02-14T00:00:00+00:00

Introduction

What a time to be alive! The year is 2017, Donald Trump is president of the United States of America and autonomous vehicles are all the rage. Still at its infancy, the winning solution to dominate the mass production of autonomous vehicles are ongoing. The two main factions currently are the robotics approach and the end-to-end neural networks approach. Like the four seasons, the AI winter has come and gone. It’s Spring and this is the story of one man’s attempt to explore the pros and cons of the end-to-end neural networks faction in a controlled environment. The hope is to draw some conclusions that will help the greater community advance as a whole.

The Controlled Environment

The Udacity Simulator which is open sourced will be our controlled environment for this journey. It has two modes, training and autonomous mode. Training mode is for the human to drive and record/collect the driving. The result would be a directory of images from three cameras (left, center, right) and a driver log CSV file that records the image along with steering angle, speed etc. Autonomous mode requires a model that can send the simulator steering angle predictions.

The goals of this project are:

Use the simulator to collect data of good driving behavior.
Construct a convolution neural network in Keras that predicts steering angles from images.
Train and validate the model with a training and validation set.
Test that the model successfully drives around track one without leaving the road!
Draw conclusions for future work.

Data Exploration

If you think the Lewis and Clark expedition was tough, try exploring an unknown dataset. I am being dramatic. When you are exploring data; you want to keep thinking, what is the least amount of data you can sample which will represent your problem (population if you like stats). For this problem we will be using the provided Udacity dataset. Let’s dive in!

Our goal is steering angle prediction, so let’s take a look at what the dataset shows us! The plot below shows a few take aways:

Range is [-1,1]
Clustering around [-0.5, 0.5]

Let’s take a look at another angle, pun intended, of the steering data.

This histogram is the making of sampling nightmares; this is how alternate facts are created! Unbalanced data sampling would train our model to be very biased so we will clean this up.

Clean up steps:

Downsample the over represented examples
Upsample the under represented examples
Expose more varied examples to try and represent a uniform distribution.

1. Downsample

Steering angle zero is over represented, so drop 90% of the examples. Easy!

2. Upsample

Time to augment so we can start upsampling under represented examples. We start by flipping 40% of the examples that are not zero steering angle. So far so good!

3. Expose more varied examples

Variety is the spice of life; this is also true to training a well generalized model. For this problem we will introduce more steering angles by shifting the examples horizontally and adding or subtracting the appropriate angles corresponding to the shift we performed. There was no magic shift amount, you get this by experimenting. The result is below:

Great our dataset/sample is looking better. Next, let’s trim this to look more uniform. This was accomplished by:

Grabbing the bin ranges of the 100 bins.
Finding bins that have more than 400 examples.
Randomly sample and drop examples so we have no more than 400 examples.

How did I know 400 examples was enough? I simply kept reducing the number until I had reached a point where my model can still produced stable results. Below 400 my model started getting unstable results.

Model Architecture and Training Strategy

Main goals for us to think about while creating a model

Is it efficient for the task at hand?
If this was to be downloaded onto hardware in a car what would the power consumption and usability be like?
- There is an interesting paper called Deep Compression, which we don’t implement here but it is food for thought and shows that model can be tiny and still work!

I started with a modified comma.ai model and I had a successful model, but it needed 300,000 trainable parameters. Reading the Deep compression paper and a blog post titled Self-driving car in a simulator with a tiny neural network I quickly realized that we can do better. The blog post linked shows a 63 parameter model that uses tiny images and a small network to get a stable model. I wanted to experiment and see if I could make it smaller and still stable. The solution developed over a number of experiments is a modified SqueezeNet implementation. With a squeeze net you get three additional hyperparameters that are used to generate the fire module:

S1x1: Number of 1x1 kernels to use in the squeeze layer within the fire module
E1x1: Number of 1x1 kernels to use in the expand layer within the fire module
E3x3: Number of 3x3 kernels to use in the expand layer within the fire module

Fire Module Zoomed In

The fire module is the workhorse of squeezenet. Sqeezenet as described in the paper is around 700k trainable parameters. I went through a process, where I kept reducing the number of parameters while all other variables were kept constant. You get the theme here? We are at the frontier and this calls for empirical testing. Through some experiments I went from 10k parameters to 1005, then 329, then 159, then 63 and finally 52!!

The final 52 parameter squeezenet variant Model!!

This model combats overfitting by being super tiny and for kicks I added a small dropout layer. The model works on both tracks and has a six second epoch on my late 2012 Macbook air!! From the comma.ai model, it was evident that a validation loss of around 0.03 on 30% of this dataset results in a stable model that can handle the track at a throttle of around 0.2, which is a speed of around 20mph in the simulator. So, I didn’t bother worrying about the epoch hyperparameter. I simply created a custom early termination Keras callback that stopped the training when we hit our requirement.

One good rule of thumb I developed from this project is to try and reduce the number of variables you are tuning to gain better results faster.

Training Strategy

To get the model to drive in the simulator, you need:

Show the model how to drive straight
How to recover if it drifts off track
How to handle turns

The Udacity slack community was a huge help here; from their experience, I used the left and right camera images and adjusted the steering (+.25 for left, -.25 for right) angles to show the model how to correct steering back to center. Then I used the horizontal shifting to capture more angles. This was enough to get a stable model working on the fastest setting (means lowest resolution) on both tracks.

Reducing input image size was the next challenge. We do this by first cropping the top and bottom from the image that would be noise to the model. Then we resize the image to (64,64) and convert to HSV and only returning the S channel. This takes us from (160,320,3) to (64, 64, 1)!!

Before Cropping and Resizing.

After Cropping and Resizing.

Hyperparameters

Learning rate: 1e-1 (very aggressive!)
Batch size: 128 (tried 64, 128, 256, 1024)
Adam optimizer

I used Keras to do a validation split on 30% of the data and my custom early termination to stop the training when we we reach a validation loss of around 0.03!

</br>

Note: this loss plot is from a previous run without early termination

This model was tiny (52 params!) and with only ~20k images I was able to train it on a 2012 Macbook air. An epoch was about six seconds. The memory requirement were small, so I just loaded the entire dataset!

Pros

Can train without a GPU.
Enough to pass the challenge.
Smaller model meant I could experiment with more variables and different models to gain better intuition.
Learned that our current method of training with back prop is not efficient and that we can achieve a lot with a smaller network.
Get to use aggressive learning rate for faster convergence :D

Cons

Can not handle highest resolution setting.
Can not go over .22 throttle and still be stable.
As the network increases in size, hard to understand why the decisions are being made. I can see this being a huge problem for legal reasons. The end-to-end neural network faction is not looking so good here!

This bring me to the other rule of thumb, every problem will have tradeoffs. The question becomes what are the tradeoffs you are willing to make? This will depend on the business case you are solving!

The github repo link

P3_behavioral_cloning

Mez Gebre

Why I Stopped Trusting Validation Loss

The first one, four things that shouldn’t happen

The second one, the thing everyone believed was wrong

What they agree on

The part that’s uncomfortable

Three groups, three scorers, none audited

Where this leaves me

Three Ways to Think You've Validated a World Model

Prerequisite one: does the model even listen to you?

Prerequisite two: can anyone tell if a rollout is any good?

The near-miss that would have been a confidently wrong published result

The pattern that showed up three times

What honoring a pre-committed stopping rule actually costs

Where this leaves the project

Audio Classification: A Weekend of Experiments (FAAAAHH!)

The Dataset: ESC-50

The Supervised Baseline

ResNet-34 from Scratch

Test-Time Augmentation

Transfer Learning Detour

Self-Supervised Learning Experiments

SimCLR (Contrastive Learning)

Masked Prediction (BEATs-style)

Transformer-BEATs

The BEATs Revelation

Things I Had to Figure Out

Final Results

Takeaways

I'm Back

HomographyNet: Deep Image Homography Estimation

Introduction

What is a Homography?

Isomorphism?

Projective geometry?

NOTE

Motivation

Where is homography used?

How do you get the homography?

HomographyNet: The Model

The 4-Point Homography Parameterization

Training Data Generation

The Code

Final thoughts

SqueezeDet: Deep Learning for Object Detection

Why bother writing this post?

Cons:

Pros:

The SqueezeDet Model

Highlevel SqueezeDet

The Composite Loss function

Using SqueezeDet

Thresholding SqueezeDet

Final thoughts

MainSqueeze: The 52 parameter model that drives in the Udacity simulator

Introduction

The Controlled Environment

Data Exploration

Clean up steps:

1. Downsample

2. Upsample

3. Expose more varied examples

Model Architecture and Training Strategy

Main goals for us to think about while creating a model

Fire Module Zoomed In

The final 52 parameter squeezenet variant Model!!

Training Strategy

Before Cropping and Resizing.

After Cropping and Resizing.

Hyperparameters

Pros

Cons

The github repo link

References