gigatoken
Language model tokenization at GB/s
File Explorer
Download Latest Version (.zip)- config.toml
- CI.yml
- wheels.yml
- settings.json
- throughput_owt_train_gpt-2.svg
- mod.rs
- encode.rs
- encode_doc.rs
- encode_st.rs
- encode_st_sp.rs
- pretokenize.rs
- pretokenize_profile.rs
- pretokenize_profile_all.rs
- simdutf_transcode.rs
- unicode.rs
- measure.py
- plot_throughput.py
- results.py
- sweep.py
- families.json
- results.json
- encode_files.py
- hf_drop_in.py
- quickstart.py
- tiktoken_drop_in.py
- __init__.py
- hf.py
- hub.py
- sentencepiece.py
- tiktoken.py
- __init__.pyi
- __init__.py
- _cli.py
- _hf_compat.py
- _parallel.py
- _tiktoken_compat.py
- _tokenizer.py
- py.typed
- bit_patterns.py
- bit_patterns2.py
- data_view.py
- dclm_sample.jsonl
- dfa_states.py
- gather_free.py
- inspect_tokenizers.py
- luke_ref.py
- simd_unicode.py
- unicode.py
- unicode_classes.py
- unicode_gather.py
- mt_report.txt
- mt_report.txt
- mt_round4_findings.md
- mt_report.txt
- mt_round5_findings.md
- cold_l2.txt
- cold_pipel1.txt
- cold_pipel2.txt
- cold_rates.txt
- cold_thp_ipc.txt
- repro.sh
- warm_l2.txt
- warm_l23.txt
- warm_pipel1.txt
- warm_pipel2.txt
- warm_rates.txt
- .gitignore
- analyze.py
- analyze.sh
- analyze_mt.py
- campaign_report.md
- mt_round3_findings.md
- pmu_summary.py
- profile.sh
- report.md
- x86_port_plan.md
- zen5_st_profile.md
- build_release_cross_platform.py
- profile-cpu.fish
- bridge.rs
- cache.rs
- hub.rs
- matcher.rs
- mod.rs
- padding.rs
- pretokenize.rs
- sources.rs
- train.rs
- mod.rs
- pretoken_cache.rs
- sentencepiece.rs
- tiktoken.rs
- decompress.rs
- file_source.rs
- jsonl.rs
- mod.rs
- parquet.rs
- hf.rs
- hub.rs
- mod.rs
- tiktoken.rs
- cl100k.rs
- cl100k_family.rs
- deepseek_v3.rs
- kimi.rs
- mask.rs
- mod.rs
- nemotron.rs
- o200k.rs
- o200k_family.rs
- olmo3.rs
- qwen2.rs
- qwen3_5.rs
- r50k.rs
- avx512.rs
- combinator.rs
- mod.rs
- simd.rs
- state_machine.rs
- mod.rs
- options.rs
- pretoken.rs
- pretokenize_traits.rs
- unicode.rs
- batch.rs
- bpe_train.rs
- lib.rs
- main.rs
- test_hub.rs
- token.rs
- gpt2_tokenizer.json
- build_hf_tokenizer.py
- build_toker_tokenizer.py
- conftest.py
- test_chunked_doc.py
- test_hf_parity.py
- bench_file_source.py
- bench_train_encode.py
- conftest.py
- dclm_fixture.py
- hf_cache.py
- test_bpe_train.py
- test_bpe_train_compare.py
- test_cache_budget.py
- test_config_dispatch.py
- test_encode.py
- test_encode_dclm.py
- test_encode_files.py
- test_file_source.py
- test_from_hf.py
- test_from_tiktoken.py
- test_hf_compat.py
- test_kimi.py
- test_load_hf.py
- test_multiprocessing.py
- test_parquet_source.py
- test_pretokenizer_iterator.py
- test_sentencepiece.py
- test_tiktoken_compat.py
- test_tokenizer_usage.py
- test_use_with_tokenizers.py
- .gitignore
- .python-version
- Cargo.lock
- Cargo.toml
- CONTRIBUTING.md
- design_doc.md
- LICENSE
- pretokenizer_optimization_log.md
- pyproject.toml
- README.md
- rust-toolchain.toml
- uv.lock
# Installation Guide
1. Get the code
git clone https://github.com/marcelroed/gigatoken
Downloads the entire project code from GitHub to your computer.
cd gigatoken
Moves into the project folder you just downloaded.
2. Official Install Script
Easy RecommendedPrerequisites
- Python 3 Python is required to use pip.
pip install gigatoken
Installs the package published on PyPI directly β no need to clone the source.
After installing, open a new terminal and run the program's version command (e.g. --version) to confirm it worked.
Pulled directly from this repo's README.
3. Python
EasyPrerequisites
pip install gigatoken
Installs the package published on PyPI directly β no need to clone the source.
If it runs without errors and prints output in the terminal, it worked.
Pulled directly from this repo's README.
4. Rust
MediumPrerequisites
- Git Needed to download the project code from GitHub.
- Rust (rustup) Installing via rustup also installs cargo.
cargo build --release
Compiles the Rust project.
cargo run
Builds and then immediately runs the program.
If cargo build finishes without errors, it worked. The executable is created under target/.
// repository documentation
Was this content helpful?
(0 ratings)