huggingface-tokenizers

Install

Runs in your terminal. Adds the skill globally for Claude Code, Cursor, Codex and others; add -g -y to skip prompts.

What it does

HuggingFace Tokenizers - Fast Tokenization for NLP Fast, production-ready tokenizers with Rust performance and Python ease-of-use. When to use HuggingFace Tokenizers Use HuggingFace Tokenizers when: - Need extremely fast tokenization (<20s per GB of text) - Training custom tokenizers from scratch - Want alignment tracking (token → original text position) - Building production NLP pipelines - Need to tokenize large corpora efficiently Performance: - Speed: <20 seconds to tokenize 1GB on CPU - Implementation: Rust core with Python/Node.js bindings - Efficiency: 10-100× faster than pure Python…

Excerpt from the skill's own SKILL.md. Read the full file on GitHub before installing: skills run with your agent's permissions.

Categories

• Python

Related skills

• ascii-art

  NousResearch/hermes-agent

  Copy install command

  ASCII art: pyfiglet, cowsay, boxes, image-to-ascii.

  412 installs 240K
• comfyui

  NousResearch/hermes-agent

  Copy install command

  Generate images, video, and audio via diffusion workflows.

  382 installs 240KDeploy & DevOpsDocumentationProductivity
• touchdesigner-mcp

  NousResearch/hermes-agent

  Copy install command

  Control TouchDesigner via twozero MCP.

  316 installs 240KDataAI & agents
• baoyu-comic

  NousResearch/hermes-agent

  Copy install command

  Knowledge comics (知识漫画): educational, biography, tutorial.

  92 installs 240KDesign & UIAI & agentsWriting
• baoyu-article-illustrator

  NousResearch/hermes-agent

  Copy install command

  Article illustrations: type × style × palette consistency.

  89 installs 240KDesign & UIAI & agentsWriting
• minecraft-modpack-server

  NousResearch/hermes-agent

  Copy install command

  Host modded Minecraft servers (CurseForge, Modrinth).

  88 installs 240KSecurity