rzen
9fd56b6063
Controlled study of how the model degrades. Four variants, each = v1 with a single field replaced (dataclasses.replace), so differences are attributable: - small: capacity down (2L/64d, ~0.38M params) - short: training time down (max_iters 20k->2k) - ctx: context window down (256->64) - data: data quantity down (max_train_tokens->1M; new TrainConfig knob + train.py slice) - scripts/compare.py: sample same prompt across all trained configs with a shared seed, print side by side, write reports/compare.md - tests/test_configs.py: enforces one-parameter-at-a-time (only intended fields differ from v1) + small param count (3 tests). Full suite: 29 passing.
18 lines
601 B
Python
18 lines
601 B
Python
"""Degradation experiment: same as v1, but a smaller context window.
|
|
|
|
Isolates *long-range coherence*. Only context_length changes from v1
|
|
(256 -> 64). Local fluency should survive; callbacks and plot threads that span
|
|
more than a few sentences should break sooner.
|
|
|
|
Honest caveat: shrinking context also lowers tokens/step (batch * context), so
|
|
this run sees fewer total tokens than v1 over the same number of steps. The
|
|
qualitative coherence effect is still the dominant, visible change.
|
|
"""
|
|
|
|
from dataclasses import replace
|
|
|
|
import v1
|
|
|
|
model = replace(v1.model, context_length=64)
|
|
train = v1.train
|