# Degradation analysis (iteration 1.5)

A controlled study: starting from the v1 baseline, change exactly **one**
variable per run and observe the effect on generated text. Because only one
thing differs per config, any change in output is attributable to that variable.

Samples: `reports/compare.md` (same 5 prompts, same RNG seed per prompt across
all models, so differences reflect the model rather than sampling luck).

## Scoreboard

| config | change from v1 | held-out val loss | perplexity | one-line read |
|---|---|---|---|---|
| **v1** | — (baseline) | 1.52 | 4.6 | coherent arcs, consistent characters |
| **ctx** | context 256→64 | 1.86 | 6.4 | fluent locally, drifts over distance |
| **short** | 20k→2k steps | 2.13 | 8.4 | clean but simple, thin structure |
| **small** | 4.3M→0.38M params | 2.40 | 11.0 | loops, repeats, mis-picks words |
| **data** | 555M→1M train tokens | **5.78** | **322** | memorizes (train ppl 1.1), fails to generalize |

The first four rank by val loss in the same order the text quality ranks by eye
— confirmation that held-out loss measures something real. `data` is the
instructive exception (see below).

## Per-variant findings

### ctx (smaller context) — prediction confirmed exactly
Val loss (1.86) is close to v1, and locally the sentences are just as clean.
The failure is purely long-range. In "In a big forest," the story opens with
Tim the bird playing hide-and-seek with a dog named Max, then ends *"They
counted all the cars, one by one."* — cars from nowhere, because with only 64
tokens of memory (~4–5 sentences) the forest opening has scrolled out of the
attention window by the time the ending is written. Same signature in the
spaghetti story: *"Sue and Tim felt ashamed"* where Tim was never introduced —
his introduction fell off the back of the window.

**Lesson:** context length buys *coherence over distance*, not local grammar.

### small vs short — same "worse," opposite causes
The most useful pair to study side by side; they fail in distinguishable ways.

- **small** (full training, low capacity) falls into **repetition loops** and
  **word-slot errors** — classic low-capacity signatures. The "In a big forest"
  sample is dominated by the word *"Kitty"* (used ~9 times, simultaneously a
  girl, a bear, and a cat); elsewhere *"and and play"*, *"friends friends"*, and
  *"She loved to uniform because she was very smart"* (uses "uniform" as a verb —
  right rhythm, wrong meaning). It has hit its ceiling and the ceiling is low.

- **short** (full capacity, barely trained) is the opposite: **clean but thin**.
  Openings like *"Once upon a time, there was a little girl named Sue..."* are
  perfectly grammatical — common patterns (story openings, frequent words) are
  learned first. What it hasn't learned yet is coherence and detail: stories are
  short and it slips (*"Lily went to play... Lucy was very surprised"* — name
  changes mid-paragraph). High ceiling, simply not yet climbed.

**Lesson:** "make it worse" is not one thing. Capacity-limited models *loop and
mis-select*; undertrained models stay *simple and shallow*.

### data (less data) — where the samples lie and the loss tells the truth
The standout result. The loss curve is a textbook overfitting U: val loss
bottoms at **~2.48 around step 1,000**, then climbs monotonically to **5.78**
while train loss free-falls to **0.11** (train perplexity ~1.1 — the model has
essentially memorized its 1M-token slice). A ~290× train/val perplexity gap.

But the samples don't *look* 290× worse. They read as locally fluent, because
top-k sampling masks bad calibration and memorized n-grams still produce
plausible spans. The damage shows only on close reading (*"make the world green
rhinoceros"*, *"The little girl wrents of the bread"*, *"Luriangle, let's do
this"* — broken tokens) and on novel prompts, where the model stitches memorized
fragments into incoherent collages (an elephant and a wild cat wander into the
kangaroo story).

**Lesson:** held-out loss reveals a failure that eyeballing samples would miss.
Judging `data` by its text alone, you might rank it among the *better* runs —
which is exactly why we measure validation loss instead of only reading output.

## Takeaways

1. **Held-out loss tracked perceived quality** for the four
   generalizing models — and correctly flagged the one (`data`) whose good-looking
   samples hid a broken model. This is the case *for* quantitative eval.
2. **Different degradations produce qualitatively different failure text**, not
   just "more" or "less" bad: window-length → long-range drift; capacity →
   loops and word-slot errors; training time → shallow-but-clean; data →
   memorization with OOD collapse.
3. **Sampling can mask model problems.** top-k + in-distribution prompts
   flatter a weak model; out-of-distribution prompts and validation loss expose it.

## Suggested follow-ups

- Sample `data` from its **step-1,000 checkpoint** (generalization peak, val
  2.48) vs the final overfit checkpoint — a direct early-stopping demonstration.
  Requires numbered snapshot checkpoints (small addition to `train.py`).
- Run `compare.py` on an **out-of-distribution prompt** (e.g. "The stock market
  crashed because") — `data` and `small` should shatter hardest, making the
  memorization-vs-generalization split starker than in-distribution story
  openings do.