Tokenizers Model Context Protocol (MCP) Server

GitHub

Fast State-of-the-Art Tokenizers optimized for Research and Production

10,947 stars Rust Rust 1 file · ~2,895 tokens #bert#gpt#language-model#natural-language-processing#natural-language-understanding#nlp#transformers

Tokenizers MCP Server

MCP server for customize/Tokenizers

Ready-to-Use Client Configurations 
{
  "mcpServers": {
    "Tokenizers": {
      "command": "npx",
      "args": [
        "-y",
        "@customize/Tokenizers"
      ]
    }
  }
}