- artifacts/tokenizer.json: 3839 merges, trained on 50MB sample in 87s. 3.97 bytes/token on held-out text. - encode() caches word->ids so encoding the full corpus is mostly dict lookups (TinyStories has a small vocabulary).
- src/tokenizer.py: byte-level BPE with GPT-2-style word pre-tokenization, protected <|endoftext|> special token, save/load as inspectable JSON. Simple recount-per-merge trainer over unique words. - tests/test_tokenizer.py: round-trip, special-token integrity, unicode, lossless pre-tokenization, save/load, determinism (8 tests, no pytest dep) - scripts/train_tokenizer.py: train on a corpus sample, report bytes/token
rzen