Phase 1: trained 4096-vocab tokenizer + per-word encode cache

Browse Source

- artifacts/tokenizer.json: 3839 merges, trained on 50MB sample in 87s.
  3.97 bytes/token on held-out text.
- encode() caches word->ids so encoding the full corpus is mostly dict
  lookups (TinyStories has a small vocabulary).

...

This commit is contained in:

rzen

2026-07-12 10:33:24 -04:00

parent 2c5fc29592

commit 3a489fef0a

2 changed files with 9 additions and 1 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

artifacts/tokenizer.json

+1

File diff suppressed because one or more lines are too long