rzen 3a489fef0a Phase 1: trained 4096-vocab tokenizer + per-word encode cache ...

- artifacts/tokenizer.json: 3839 merges, trained on 50MB sample in 87s.
  3.97 bytes/token on held-out text.
- encode() caches word->ids so encoding the full corpus is mostly dict
  lookups (TinyStories has a small vocabulary).

2026-07-12 10:33:24 -04:00

..

tokenizer.json

Phase 1: trained 4096-vocab tokenizer + per-word encode cache

2026-07-12 10:33:24 -04:00