experiment-llm/reports/degradation-analysis.md

Degradation analysis (iteration 1.5)

A controlled study: starting from the v1 baseline, change exactly one variable per run and observe the effect on generated text. Because only one thing differs per config, any change in output is attributable to that variable.

Samples: reports/compare.md (same 5 prompts, same RNG seed per prompt across all models, so differences reflect the model rather than sampling luck).

Scoreboard

config	change from v1	held-out val loss	perplexity	one-line read
v1	— (baseline)	1.52	4.6	coherent arcs, consistent characters
ctx	context 256→64	1.86	6.4	fluent locally, drifts over distance
short	20k→2k steps	2.13	8.4	clean but simple, thin structure
small	4.3M→0.38M params	2.40	11.0	loops, repeats, mis-picks words
data	555M→1M train tokens	5.78	322	memorizes (train ppl 1.1), fails to generalize

The first four rank by val loss in the same order the text quality ranks by eye — confirmation that held-out loss measures something real. data is the instructive exception (see below).

Per-variant findings

ctx (smaller context) — prediction confirmed exactly

Val loss (1.86) is close to v1, and locally the sentences are just as clean. The failure is purely long-range. In "In a big forest," the story opens with Tim the bird playing hide-and-seek with a dog named Max, then ends "They counted all the cars, one by one." — cars from nowhere, because with only 64 tokens of memory (~4–5 sentences) the forest opening has scrolled out of the attention window by the time the ending is written. Same signature in the spaghetti story: "Sue and Tim felt ashamed" where Tim was never introduced — his introduction fell off the back of the window.

Lesson: context length buys coherence over distance, not local grammar.

small vs short — same "worse," opposite causes

The most useful pair to study side by side; they fail in distinguishable ways.

• small (full training, low capacity) falls into repetition loops and word-slot errors — classic low-capacity signatures. The "In a big forest" sample is dominated by the word "Kitty" (used ~9 times, simultaneously a girl, a bear, and a cat); elsewhere "and and play", "friends friends", and "She loved to uniform because she was very smart" (uses "uniform" as a verb — right rhythm, wrong meaning). It has hit its ceiling and the ceiling is low.

• short (full capacity, barely trained) is the opposite: clean but thin. Openings like "Once upon a time, there was a little girl named Sue..." are perfectly grammatical — common patterns (story openings, frequent words) are learned first. What it hasn't learned yet is coherence and detail: stories are short and it slips ("Lily went to play... Lucy was very surprised" — name changes mid-paragraph). High ceiling, simply not yet climbed.

Lesson: "make it worse" is not one thing. Capacity-limited models loop and mis-select; undertrained models stay simple and shallow.

data (less data) — where the samples lie and the loss tells the truth

The standout result. The loss curve is a textbook overfitting U: val loss bottoms at ~2.48 around step 1,000, then climbs monotonically to 5.78 while train loss free-falls to 0.11 (train perplexity ~1.1 — the model has essentially memorized its 1M-token slice). A ~290× train/val perplexity gap.

But the samples don't look 290× worse. They read as locally fluent, because top-k sampling masks bad calibration and memorized n-grams still produce plausible spans. The damage shows only on close reading ("make the world green rhinoceros", "The little girl wrents of the bread", "Luriangle, let's do this" — broken tokens) and on novel prompts, where the model stitches memorized fragments into incoherent collages (an elephant and a wild cat wander into the kangaroo story).

Lesson: held-out loss reveals a failure that eyeballing samples would miss. Judging data by its text alone, you might rank it among the better runs — which is exactly why we measure validation loss instead of only reading output.

Takeaways

1. Held-out loss tracked perceived quality for the four generalizing models — and correctly flagged the one (data) whose good-looking samples hid a broken model. This is the case for quantitative eval.
2. Different degradations produce qualitatively different failure text, not just "more" or "less" bad: window-length → long-range drift; capacity → loops and word-slot errors; training time → shallow-but-clean; data → memorization with OOD collapse.
3. Sampling can mask model problems. top-k + in-distribution prompts flatter a weak model; out-of-distribution prompts and validation loss expose it.

Suggested follow-ups

• Sample data from its step-1,000 checkpoint (generalization peak, val 2.48) vs the final overfit checkpoint — a direct early-stopping demonstration. Requires numbered snapshot checkpoints (small addition to train.py).
• Run compare.py on an out-of-distribution prompt (e.g. "The stock market crashed because") — data and small should shatter hardest, making the memorization-vs-generalization split starker than in-distribution story openings do.