 rzen
|
e708a5b886
|
Phase 3: decoder-only transformer (~4.3M params)
- src/model.py: GPT written module by module — RMSNorm, explicit causal
self-attention (scores/mask/softmax/weighted-sum), GELU MLP, pre-norm
residual blocks, learned positions, weight-tied head. GPT-2 style init.
- configs/v1.py: frozen ModelConfig dataclass (4L/256d/4h/256ctx/4096vocab)
- tests/test_model.py: exact param count (4,262,144), forward shapes, init
loss ~= ln(vocab), weight tying, causal-masking check (5 tests, forward-only)
- Verified forward pass runs on MPS.
|
2026-07-12 10:48:55 -04:00 |
|
|
rzen
|
c202bada6a
|
Phase 2: token dataset + corpus encoder (code only, not yet run)
- src/dataset.py: memmap uint16 token loader + random-crop batch sampler
(x, y shifted-by-one, reproducible via generator)
- scripts/encode_corpus.py: stream stories -> encode -> uint16 .bin, one EOT
token appended per story, flat memory. encode_file() factored out for testing.
- tests/test_dataset.py: batch shapes/shift/bounds/reproducibility +
encoder round-trip/separators/uint16 on synthetic data (5 tests)
|
2026-07-12 10:38:20 -04:00 |
|
|
rzen
|
2c5fc29592
|
Phase 1: from-scratch byte-level BPE tokenizer
- src/tokenizer.py: byte-level BPE with GPT-2-style word pre-tokenization,
protected <|endoftext|> special token, save/load as inspectable JSON.
Simple recount-per-merge trainer over unique words.
- tests/test_tokenizer.py: round-trip, special-token integrity, unicode,
lossless pre-tokenization, save/load, determinism (8 tests, no pytest dep)
- scripts/train_tokenizer.py: train on a corpus sample, report bytes/token
|
2026-07-12 10:31:16 -04:00 |
|