rzen/experiment-llm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
18 Commits 1 Branch 0 Tags
main
Commit Graph
5 Commits
Author SHA1 Message Date
rzen 9fd56b6063 Add degradation experiments: one-parameter-at-a-time ablations 
Controlled study of how the model degrades. Four variants, each = v1 with a
single field replaced (dataclasses.replace), so differences are attributable:
- small: capacity down (2L/64d, ~0.38M params)
- short: training time down (max_iters 20k->2k)
- ctx:   context window down (256->64)
- data:  data quantity down (max_train_tokens->1M; new TrainConfig knob + train.py slice)

- scripts/compare.py: sample same prompt across all trained configs with a
  shared seed, print side by side, write reports/compare.md
- tests/test_configs.py: enforces one-parameter-at-a-time (only intended fields
  differ from v1) + small param count (3 tests). Full suite: 29 passing.
 2026-07-12 12:58:48 -04:00
rzen 4947556c02 Phase 4: dependency-free ASCII loss-curve plotter 
- scripts/plot_loss.py: reads reports/loss-<config>.csv, renders train/val
  loss as a terminal ASCII chart, saves it to reports/loss-<config>.txt.
  Verified on a synthetic curve. Keeps deps at torch + numpy.
- README: documented plotting choice and the Phase 4 run commands.
 2026-07-12 11:32:34 -04:00
rzen c202bada6a Phase 2: token dataset + corpus encoder (code only, not yet run) 
- src/dataset.py: memmap uint16 token loader + random-crop batch sampler
  (x, y shifted-by-one, reproducible via generator)
- scripts/encode_corpus.py: stream stories -> encode -> uint16 .bin, one EOT
  token appended per story, flat memory. encode_file() factored out for testing.
- tests/test_dataset.py: batch shapes/shift/bounds/reproducibility +
  encoder round-trip/separators/uint16 on synthetic data (5 tests)
 2026-07-12 10:38:20 -04:00
rzen 2c5fc29592 Phase 1: from-scratch byte-level BPE tokenizer 
- src/tokenizer.py: byte-level BPE with GPT-2-style word pre-tokenization,
  protected <|endoftext|> special token, save/load as inspectable JSON.
  Simple recount-per-merge trainer over unique words.
- tests/test_tokenizer.py: round-trip, special-token integrity, unicode,
  lossless pre-tokenization, save/load, determinism (8 tests, no pytest dep)
- scripts/train_tokenizer.py: train on a corpus sample, report bytes/token
 2026-07-12 10:31:16 -04:00
rzen 5a7f2177bc Phase 0: project plan, pinned environment, data download + MPS smoke test 
- README with full pipeline plan (tokenizer -> data -> model -> training -> eval)
- pip/venv setup pinned to torch 2.13.0, numpy 2.5.1 (Python 3.13)
- scripts/download_data.py: one-time TinyStoriesV2 fetch with SHA-256 recording
- scripts/check_mps.py: verifies MPS backend, backward pass, GPU speedup
 2026-07-12 10:24:52 -04:00