Tokenizers in your browser
A survey of the local tokenization landscape at huggingface
Everything runs 100% locally in your browser!
tokenizers v1
Maximum throughput
1.0.0-rc.2 · huggingface/tokenizers
tokenizers 0.23
Reference point
0.23.1 · huggingface/tokenizers
@huggingface/tokenizers
Tiny footprint
0.2.0 · huggingface/tokenizers.js
🔍
↵ load
or try
Tokenizer.from_pretrained()...
try
Side by side
Same tokenizer.json, three engines–
Encode speed, visually
One bounce = One encode pass over your text