# tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

### Repository Details
- **Repository:** [mistralai/tokenizers](https://github.com/mistralai/tokenizers)
- **GitHub Stars:** ⭐ 10,947
- **Primary Language:** Rust

*Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.*