Add degradation analysis + backing artifacts

- reports/degradation-analysis.md: interpretation of the one-parameter-at-a-time ablations (ctx/short/small/data vs v1), grounded in val loss + sample text. Key findings: held-out loss tracks quality for generalizing models; different degradations give qualitatively different failure text; data-starvation overfits (train ppl 1.1 / val ppl 322) with samples that hide the damage. - reports/compare.md: side-by-side samples across all configs - reports/loss-{small,short,ctx,data}.csv: variant training curves
2026-07-12 18:28:43 -04:00
parent 9fd56b6063
commit 921069c5db
6 changed files with 375 additions and 0 deletions
@@ -0,0 +1,155 @@
+# Degradation comparison
+
+## Prompt: Once upon a time,
+
+### v1 (baseline)
+
+> Once upon a time, a little girl named Lucy was playing in her room. She had a toy box with lots of glue. She liked to make things with the glue. One day, she found a pretty bird in her toy box.
+Lucy wanted to help the bird. She took the glue and made a necklace with the glue. She put the bird on the necklace and started to make it. The bird looked very pretty. It was a happy night when it touched the glue.
+Then, Lucy took the bird to the kitchen. She cut a cold ice cream for it. Then, something unexpected happened. The bird started to sing! Lucy was so surprised! The bird gave Lucy a big smile and flew away. Lucy felt happy and loved.
+
+### small (less capacity)
+
+> Once upon a time, a little girl named Lily was playing in a big castle. She loved to uniform because she was very smart. She liked to sing and sing songs.
+One day, Lily saw a big, shiny stone in a tree. She wanted to be flying around, so she decided to dance with her friends. Lily's family laughed and said, "Wow, Lily! What are you doing?" Lily said, "Let's race, please!"
+Lily and Max played games all day, and and play, like Lily would like her friend, Ben, and Ben liked bubbles. They would run and jump fast and chase each other! The funny jokes were ever wagging and barking loudly.
+The end. Lily and him started to tremble. They ran faster than Jill and faster. They all laughed and felt happy. They knew they could play with the ball. They played in the park and felt happy too.
+
+### short (less training)
+
+> Once upon a time, there was a little girl named Sue. Sue loved to play outside in the yard. One sunny day, she saw a big puddle. She wanted to jump in it.
+Sue went to her mom and saw her in the puddle. She put on her leash and waved her eyes. "Mom, look! I found a big puddle!" she said.
+"Wow, Sue, what's that!" said her mom. "I touched the puddle with water. Can you try again?" she asked. The big puddle was happy. Sue and her mom played with the big, colorful tip on the wet puddle. They laughed and had a fun day at the end.
+
+### ctx (smaller context)
+
+> Once upon a time, there was a little girl named Sue. Sue loved to eat spaghetti. One day, she found a big plate of spaghetti in a small kitchen. Sue was very happy and wanted to eat yogurt.
+Sue saw her mom was upset and asked, "Why are you sad?" Her mom said, "I don't know, but let's look for more spaghetti to make it less perfect." They went to look for more spaghetti and found one more spaghetti in the kitchen yard.
+As Sue was looking, her mom came into the room. "Oh no!" she said, "Now the spaghetti is not great, it is good." Sue and Tim felt ashamed because they did not want to share.
+Sue's mom came and saw the mess. "Sue, it's not nice to share and go to the table," she said. Sue and Tim learned that it is good to listen and to their moms. They promised to be more careful and not spoiled.
+
+### data (less data)
+
+> Once upon a time, a little girl named Amy had a toy box. She loved to play with her toys in her room. One day, she saw a new toy in the store. It was a big, shiny ball that she pretended it was as a toy.
+Amy wanted to give the ball to her friend, Tom, and they started to play a new toy. Tom saw Amy playing with the ball in his yard. He was very happy and said, "I don't want to share this toy with Amy." Amy looked at Tom again and said, "You can play together, Tom!"
+They played with the ball and red ball all day. Amy was very happy to share their toys with Amy friends. They played with the red ball together and had a great time. When it was time to go home, they had a great time.
+
+## Prompt: One day, a little girl named Lily
+
+### v1 (baseline)
+
+> One day, a little girl named Lily went to the park with her mom. They brought a seat with lots of fun things to play with. Lily saw a big tree with a swing. She wanted to play on the swing, but her mom saw her and said, "No, that's not a good idea. This is a bad idea for you."
+Lily felt sad, but she still had her idea. She went to find her mom and asked her many questions about the park. Then, she saw a squirrel! Lily asked the squirrel, "Do you want to play on the swing?" The squirrel liked the idea and started to play on the swing.
+Lily and her mom played on the swing all day. They laughed and had fun together. Lily was happy that she made a new friend and not so bad after all.
+
+### small (less capacity)
+
+> One day, a little girl named Lily went to the park. She saw a small girl named Sue. Sue wanted to be the only?" Sue said, "Yes, please!" So, Sue showed the little girl, Sue, came to help. She said, "You can help me use the bird, Sue."
+But, something unexpected happened. Sue's family laughed. The best bottle bottle had many fun toys. Sue and Tom became good friends. They played and laughed. In the end, they were all happy. Sue learned that helping others is not good to share.
+
+### short (less training)
+
+> One day, a little girl named Lily went to play with her ball. She saw her friend, Tim, and said, "Hi, I found your ball! I found a toy for you!"
+As she played, something unexpected happened. The ball started to shake and fall. Lucy was very surprised! She did not know what happened. The ball flew down and made a big, beautiful ball. They played and had a fun day together.
+
+### ctx (smaller context)
+
+> One day, a little girl named Lily went to the park with her mom. They brought a cheap toy to play with. Lily liked the toy very much. She liked the toy and wanted to play with it. She asked her mom, "Can I have a toy, please?"
+Her mom said, "yes are very nice, but first you have to clean them up." Lily did not like to clean up. She wanted to be a good girl, so she said, "Okay, Mom. I will clean my room for a minute."
+Lily worked hard to messy her room. She was very careful with the toys. She did not want to make her room clean and clean. But then, she started to clean her room. Her mom was happy and said, "Good job, Lily! You are so clean."
+Lily and her mom went home. They made her the room clean and pretty. Lily was happy that she helped her mom. The moral of the story is to always clean up mess and
+
+### data (less data)
+
+> One day, a little girl named Lily wanted to send a letter to a friend. She said, "I will send the letter to my friend?"
+The big bear came into the store. Lily passed the man and said, "I have trash to make you dizzy and did not know. She quickly turned the shelf to make everyone angry with her.
+The little girl wrents of the bread, but the Don't worry. It's really difficult for someone to do in time. Luriangle, let's do this as you get to the library nicely." The two friends happily, and they all had fun at the library.
+
+## Prompt: Tom was very happy because
+
+### v1 (baseline)
+
+> Tom was very happy because he got a new toy. He played with it all day long. The other kids saw the new toy and wanted to play with it too. So, they all had a fun day playing with the new toy.
+But one day, the new toy broke. The kids were sad and wanted to see his toy again. They thought of a plan to fix the new toy. They went to the park to find some glue to fix it.
+Then, something unexpected happened. A big wind came and took the new toy away. The kids were angry and sad. They didn't want to play with the new toy anymore. But, they knew they could play with it again after all.
+
+### small (less capacity)
+
+> Tom was very happy because he told Tom how he was. He knew he would tell his friends and unade. He would come to him and take a nap inside and let him look for him. He went to bed with him, with his friends. Tim was very happy because his family gave him a big hug.
+
+### short (less training)
+
+> Tom was very happy because he told his mom about the big, red ball. His mom said, "That is a good ball, but you have to be careful and not to be scared. You have to be careful with the big, round ball. You will be careful not bite."
+Tom did not listen to his mom. He did not want to play with his big red ball anymore. He was a perfect day. He learned that it is good to play. He was more careful not to throw things.
+
+### ctx (smaller context)
+
+> Tom was very happy because he had a big new friend. They played together all day long. The big dog was not sad anymore.
+
+### data (less data)
+
+> Tom was very happy because he had a new friend. They played together every day, and Tim never used the new bunny his special power to make the world green rhinoceros.
+
+## Prompt: The dog and the cat were
+
+### v1 (baseline)
+
+> The dog and the cat were very happy. They said, "We have a fun time with the tray!" The dog said, "Yes, let's do it!" So, they took turns rolling the tray and running around.
+Suddenly, a big, bad cat jumped out of the house! The dog was scared and ran away. The cat and the dog were safe and happy. They learned that sometimes, even when you don't give up, things can change us too.
+
+### small (less capacity)
+
+> The dog and the cat were very sad, but then, something unexpected happened. The cat ran away and found a big ball. The cat was sad but could not be kind. The cat had an idea. The bird tried the cat, but it was too big. So, the cat and the dog played together at the cat's house. The cat was happy and gave the cat a big hug. The dog laughed and played all day.
+
+### short (less training)
+
+> The dog and the cat were very happy.
+The bird and the cat started to feel good. The cat said, "I am proud of you, but we promise to always be safe." The cat agreed and helped the cat.
+The cat and the cat became good friends. They played together in the big park. And they learned that sharing is good.
+
+### ctx (smaller context)
+
+> The dog and the cat were very sad, and the bird could not fly. They had so much fun flying in the air. The cat was sad, but the dog had an idea. "Let's be friends, the cat, for both of us and play with the beautiful bird together!"
+So, the cat, the dog, and the bird played happily together in the park. They became best friends, and they were always happy.
+
+### data (less data)
+
+> The dog and the cat were very sad. They did not want to hurt the pink ball. The cat said, "I regretThen I cannot find a way to wait. I will not be sad or Sam. I am a gentle ok?"
+Sally and Tom nodded and decided to examine the orange ball. They looked at each other and felt warm and happy. They took the orange ball and played again. From that day on, they were the best of friends.
+
+## Prompt: In a big forest, there lived
+
+### v1 (baseline)
+
+> In a big forest, there lived a little bunny named Bobo. Bobo had a big problem. He needed a lime to eat. This lime was difficult for him to eat. He tried to eat it, but he was too small.
+One day, a nice bird named Betty saw Bobo. She wanted to help him. Betty flew down and said, "I have a lime. I have a lime. I will give you some milk and a lime."
+Bobo was happy with Betty's lime. They sat under a big tree and ate the water. Bobo liked the lime very much. He said, "Thank you, Betty, for helping me."
+After eating the lime, Bobo and Betty started to play together. They played hide and seek, and they had a lot of fun. They became good friends and always shared the lime.
+
+### small (less capacity)
+
+> In a big forest, there lived a sad cat named Kitty. Kitty had a big, soft pillow, and Kitty, and Kitty loved to sleep under the pillow.
+One day, Kitty and Kitty met a new friend, Kitty the bear, came over to play and they played together. Doggy said, "Wow, let's spin the pillow too!" Kitty and Kitty played more together.
+As Kitty played, a big cat, Kitty, and a dog talked to each other. To Kitty been talking about all, Kitty! Buddy and the cat became friends because she loved her messy pillow. They were all very happy to have made their day friends friends and the big, they all played together in the forest all day long.
+
+### short (less training)
+
+> In a big forest, there lived a little bird named Tim. Tim was a very intelligent bird. One day, he saw a big, red ball. He wanted to play with it.
+Tim went to the park to play with the ball. He saw a big tree with many leaves. He wanted to play with the ball, but he was too small. He tried to pick up the leaves, but he was too big.
+As Tim played, he saw a big tree. To his surprise, it was not a tree in the tree. Tim was very happy. He played with the ball all day, and he was very good at the big park.
+
+### ctx (smaller context)
+
+> In a big forest, there lived a little bird named Tim. Tim was a very intelligent bird. He could do many tricks all day long. One day, while playing with his friends, theyus met a big dog. The dog was big and had a lot of things.
+Tim and his friends decided to play a game of hide and seek. Tim took a deep breath and hid behind a big bush. The dog looked and looked, but Tim was not there to hide. He felt sad and sat down with his friend, a big dog named Max. They started to play hide and seek, and Tim liked to hide behind the bush.
+Tim and Max counted, "One, two, three." They were both very good at counting. Then, Tim and Max started to count, "One, two, three, four, five, six, seven. He started to count. The other cars were happy and started to count. They counted all the cars, one by one.
+When they finished,
+
+### data (less data)
+
+> In a big forest, there lived a kangaroo who lived in the forest. He liked to jump and play all day long.
+One day, the big kangaroo had a sack. He saw a wild cat stuck in a tree. He decided to thank his friend, a little bird. He opened the sack, and the big elephant wanted to protect it from the wild cat.
+But the wild cat saw lost its balance and fell into the sky. Tim and the cat were sad. They tried to get the sack, but they did not know. Then, a wise old owl saw them crying and asked, "Why are you both sad?"
+The owl told the bird about the sack and decided to poke its head up. The bird did not know, and the bird started to laugh.
+"Go away, squirrel," said the bird. "This is my best friend."
+In the end, the bird learned that being kind and helpful was not bad. The bird learned that friends can help each other animals when they need
@@ -0,0 +1,96 @@
+# Degradation analysis (iteration 1.5)
+
+A controlled study: starting from the v1 baseline, change exactly **one**
+variable per run and observe the effect on generated text. Because only one
+thing differs per config, any change in output is attributable to that variable.
+
+Samples: `reports/compare.md` (same 5 prompts, same RNG seed per prompt across
+all models, so differences reflect the model rather than sampling luck).
+
+## Scoreboard
+
+| config | change from v1 | held-out val loss | perplexity | one-line read |
+|---|---|---|---|---|
+| **v1** | — (baseline) | 1.52 | 4.6 | coherent arcs, consistent characters |
+| **ctx** | context 256→64 | 1.86 | 6.4 | fluent locally, drifts over distance |
+| **short** | 20k→2k steps | 2.13 | 8.4 | clean but simple, thin structure |
+| **small** | 4.3M→0.38M params | 2.40 | 11.0 | loops, repeats, mis-picks words |
+| **data** | 555M→1M train tokens | **5.78** | **322** | memorizes (train ppl 1.1), fails to generalize |
+
+The first four rank by val loss in the same order the text quality ranks by eye
+— confirmation that held-out loss measures something real. `data` is the
+instructive exception (see below).
+
+## Per-variant findings
+
+### ctx (smaller context) — prediction confirmed exactly
+Val loss (1.86) is close to v1, and locally the sentences are just as clean.
+The failure is purely long-range. In "In a big forest," the story opens with
+Tim the bird playing hide-and-seek with a dog named Max, then ends *"They
+counted all the cars, one by one."* — cars from nowhere, because with only 64
+tokens of memory (~4–5 sentences) the forest opening has scrolled out of the
+attention window by the time the ending is written. Same signature in the
+spaghetti story: *"Sue and Tim felt ashamed"* where Tim was never introduced —
+his introduction fell off the back of the window.
+
+**Lesson:** context length buys *coherence over distance*, not local grammar.
+
+### small vs short — same "worse," opposite causes
+The most useful pair to study side by side; they fail in distinguishable ways.
+
+- **small** (full training, low capacity) falls into **repetition loops** and
+  **word-slot errors** — classic low-capacity signatures. The "In a big forest"
+  sample is dominated by the word *"Kitty"* (used ~9 times, simultaneously a
+  girl, a bear, and a cat); elsewhere *"and and play"*, *"friends friends"*, and
+  *"She loved to uniform because she was very smart"* (uses "uniform" as a verb —
+  right rhythm, wrong meaning). It has hit its ceiling and the ceiling is low.
+
+- **short** (full capacity, barely trained) is the opposite: **clean but thin**.
+  Openings like *"Once upon a time, there was a little girl named Sue..."* are
+  perfectly grammatical — common patterns (story openings, frequent words) are
+  learned first. What it hasn't learned yet is coherence and detail: stories are
+  short and it slips (*"Lily went to play... Lucy was very surprised"* — name
+  changes mid-paragraph). High ceiling, simply not yet climbed.
+
+**Lesson:** "make it worse" is not one thing. Capacity-limited models *loop and
+mis-select*; undertrained models stay *simple and shallow*.
+
+### data (less data) — where the samples lie and the loss tells the truth
+The standout result. The loss curve is a textbook overfitting U: val loss
+bottoms at **~2.48 around step 1,000**, then climbs monotonically to **5.78**
+while train loss free-falls to **0.11** (train perplexity ~1.1 — the model has
+essentially memorized its 1M-token slice). A ~290× train/val perplexity gap.
+
+But the samples don't *look* 290× worse. They read as locally fluent, because
+top-k sampling masks bad calibration and memorized n-grams still produce
+plausible spans. The damage shows only on close reading (*"make the world green
+rhinoceros"*, *"The little girl wrents of the bread"*, *"Luriangle, let's do
+this"* — broken tokens) and on novel prompts, where the model stitches memorized
+fragments into incoherent collages (an elephant and a wild cat wander into the
+kangaroo story).
+
+**Lesson:** held-out loss reveals a failure that eyeballing samples would miss.
+Judging `data` by its text alone, you might rank it among the *better* runs —
+which is exactly why we measure validation loss instead of only reading output.
+
+## Takeaways
+
+1. **Held-out loss tracked perceived quality** for the four
+   generalizing models — and correctly flagged the one (`data`) whose good-looking
+   samples hid a broken model. This is the case *for* quantitative eval.
+2. **Different degradations produce qualitatively different failure text**, not
+   just "more" or "less" bad: window-length → long-range drift; capacity →
+   loops and word-slot errors; training time → shallow-but-clean; data →
+   memorization with OOD collapse.
+3. **Sampling can mask model problems.** top-k + in-distribution prompts
+   flatter a weak model; out-of-distribution prompts and validation loss expose it.
+
+## Suggested follow-ups
+
+- Sample `data` from its **step-1,000 checkpoint** (generalization peak, val
+  2.48) vs the final overfit checkpoint — a direct early-stopping demonstration.
+  Requires numbered snapshot checkpoints (small addition to `train.py`).
+- Run `compare.py` on an **out-of-distribution prompt** (e.g. "The stock market
+  crashed because") — `data` and `small` should shatter hardest, making the
+  memorization-vs-generalization split starker than in-distribution story
+  openings do.
@@ -0,0 +1,40 @@
+iter,train_loss,val_loss,lr
+500,3.1185,3.1282,0.000600
+1000,2.7515,2.7430,0.000598
+1500,2.5711,2.5666,0.000594
+2000,2.4273,2.4137,0.000589
+2500,2.3679,2.3545,0.000582
+3000,2.2945,2.2889,0.000574
+3500,2.2585,2.2699,0.000564
+4000,2.2313,2.2316,0.000552
+4500,2.1853,2.1899,0.000540
+5000,2.1778,2.1655,0.000525
+5500,2.1561,2.1486,0.000510
+6000,2.1237,2.1239,0.000494
+6500,2.1114,2.0983,0.000476
+7000,2.0989,2.0917,0.000458
+7500,2.0700,2.0796,0.000438
+8000,2.0694,2.0614,0.000418
+8500,2.0521,2.0414,0.000398
+9000,2.0460,2.0268,0.000377
+9500,2.0296,2.0165,0.000356
+10000,2.0116,2.0170,0.000334
+10500,2.0007,2.0038,0.000313
+11000,2.0024,1.9901,0.000292
+11500,1.9891,1.9879,0.000271
+12000,1.9751,1.9858,0.000250
+12500,1.9673,1.9474,0.000230
+13000,1.9512,1.9408,0.000210
+13500,1.9377,1.9407,0.000191
+14000,1.9227,1.9326,0.000173
+14500,1.9218,1.9179,0.000156
+15000,1.9142,1.9091,0.000141
+15500,1.9104,1.8999,0.000126
+16000,1.8979,1.8871,0.000113
+16500,1.8929,1.8983,0.000101
+17000,1.8862,1.8837,0.000090
+17500,1.8899,1.8808,0.000081
+18000,1.8819,1.8795,0.000073
+18500,1.8779,1.8705,0.000068
+19000,1.8675,1.8755,0.000063
+19500,1.8710,1.8623,0.000061
@@ -0,0 +1,40 @@
+iter,train_loss,val_loss,lr
+500,2.7063,2.8519,0.000600
+1000,2.0187,2.4752,0.000598
+1500,1.5892,2.4857,0.000594
+2000,1.2216,2.6957,0.000589
+2500,0.9258,2.9952,0.000582
+3000,0.7228,3.3175,0.000574
+3500,0.5745,3.6435,0.000564
+4000,0.4729,3.9053,0.000552
+4500,0.4051,4.1009,0.000540
+5000,0.3618,4.3420,0.000525
+5500,0.3297,4.5353,0.000510
+6000,0.3004,4.6517,0.000494
+6500,0.2772,4.8296,0.000476
+7000,0.2560,4.9157,0.000458
+7500,0.2453,5.0316,0.000438
+8000,0.2287,5.1291,0.000418
+8500,0.2181,5.1981,0.000398
+9000,0.2058,5.2845,0.000377
+9500,0.1965,5.3598,0.000356
+10000,0.1834,5.3955,0.000334
+10500,0.1776,5.4706,0.000313
+11000,0.1708,5.5157,0.000292
+11500,0.1647,5.5734,0.000271
+12000,0.1585,5.5878,0.000250
+12500,0.1506,5.6231,0.000230
+13000,0.1458,5.6917,0.000210
+13500,0.1392,5.6698,0.000191
+14000,0.1350,5.7050,0.000173
+14500,0.1313,5.7472,0.000156
+15000,0.1256,5.7613,0.000141
+15500,0.1240,5.7299,0.000126
+16000,0.1210,5.7794,0.000113
+16500,0.1175,5.7508,0.000101
+17000,0.1149,5.7778,0.000090
+17500,0.1132,5.7581,0.000081
+18000,0.1113,5.7880,0.000073
+18500,0.1102,5.7671,0.000068
+19000,0.1083,5.7651,0.000063
+19500,0.1069,5.7764,0.000061
@@ -0,0 +1,4 @@
+iter,train_loss,val_loss,lr
+500,2.8303,2.8186,0.000564
+1000,2.3438,2.3397,0.000377
+1500,2.1359,2.1259,0.000156
@@ -0,0 +1,40 @@
+iter,train_loss,val_loss,lr
+500,3.9623,3.9608,0.000600
+1000,3.4403,3.4320,0.000598
+1500,3.1455,3.1442,0.000594
+2000,2.9553,2.9633,0.000589
+2500,2.8516,2.8464,0.000582
+3000,2.7778,2.7718,0.000574
+3500,2.7201,2.7195,0.000564
+4000,2.6785,2.6591,0.000552
+4500,2.6386,2.6377,0.000540
+5000,2.6045,2.5929,0.000525
+5500,2.5917,2.5906,0.000510
+6000,2.5724,2.5575,0.000494
+6500,2.5456,2.5374,0.000476
+7000,2.5357,2.5335,0.000458
+7500,2.5279,2.5149,0.000438
+8000,2.5107,2.4987,0.000418
+8500,2.5012,2.4953,0.000398
+9000,2.5002,2.4832,0.000377
+9500,2.4812,2.4738,0.000356
+10000,2.4779,2.4697,0.000334
+10500,2.4721,2.4516,0.000313
+11000,2.4595,2.4584,0.000292
+11500,2.4524,2.4410,0.000271
+12000,2.4597,2.4465,0.000250
+12500,2.4451,2.4416,0.000230
+13000,2.4431,2.4348,0.000210
+13500,2.4292,2.4232,0.000191
+14000,2.4176,2.4196,0.000173
+14500,2.4216,2.4238,0.000156
+15000,2.4285,2.4162,0.000141
+15500,2.4089,2.4081,0.000126
+16000,2.4085,2.4228,0.000113
+16500,2.4114,2.4084,0.000101
+17000,2.4089,2.4052,0.000090
+17500,2.4007,2.3989,0.000081
+18000,2.4007,2.3943,0.000073
+18500,2.4035,2.3914,0.000068
+19000,2.4030,2.4014,0.000063
+19500,2.3995,2.3953,0.000061