We write down what broke

The validation curve went flat, and it stayed flat. Long enough that we started writing it up. A flat line looks like a ceiling, and a ceiling is a finding. We were already picking a title.

Then we went back and read the schedule.

The schedule was supposed to hold steady for most of the run and ease down over the last stretch. It had hit its floor long before that. The trainer counted steps in one unit, and the schedule had been written in another. So the easing curve finished about sixteen times too soon, and the model spent most of its life at the bottom of the curve, training as hard as it could … on a schedule that had already finished.

Seventeen of the twenty-two billion tokens in the run were trained at the floor. Most of the budget went there, into a question nobody had asked.

We stopped the run and fixed the unit mismatch. We added a check that refuses to start a run if the schedule does not land where it should. Then we relaunched from the last checkpoint on the corrected schedule. That flat stretch was the learning rate, pinned at its floor. With the steps that small, each batch changed the model by almost nothing, so the curve had nothing new to report.

We found it fast because the schedule, the config, and the curve were in the same journal file. Going back to look meant opening one file and reading down.

Three smaller ones

The rest of the miss side of the log is smaller. Each entry is a wrong conclusion we came close to keeping.

The throughput number was off by about seven times. The heartbeat wrote out a sustained rate, and we sized the next milestone from it. The estimate came out in hours. It took days. The field was measuring the wrong thing, and nothing checked it.

Then there was the service that would not stay stopped. It was set to restart on any failure. We stopped it to make a decision, and a plain kill brought it back half a minute later … resuming exactly where it had left off. The decision did not stick until we stopped it the proper way and disabled the restart.

And the reading that changed its mind. One of the curves we track looked like it turned at a certain point. We reran the cell with more steps, and the early reading came out as an artifact of running it too short. That first number is marked retired in the journal so it does not get reused by accident.

The wins will get their own pages when they are ready. This one is the misses. What is still open is running on the Small Model Research page.