Orchestra-Research/AI-Research-SKILLs
evaluating-llms-harness
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
Before you install
- Last update
- Files in folder
- 5
- Scripts
- No script files found
- SKILL.md size
- 12 KB (about 2,974 tokens of context each time it loads)
- License
- MIT
- Source on GitHub
- Orchestra-Research/AI-Research-SKILLs
Skills are plain text instructions the agent follows, sometimes with scripts. Check the source, prefer repositories with many installs and stars, and read any script it ships.
Install
npx skills add Orchestra-Research/AI-Research-SKILLs@evaluating-llms-harnessRuns in a terminal with Node.js 18 or newer. Add -y to skip the prompts and -g to install for every project instead of this one.
Install for your agent
npx skills add Orchestra-Research/AI-Research-SKILLs@evaluating-llms-harness --agent claude-code -y- Project install writes to
.claude/skills/evaluating-llms-harness- Global install writes to
~/.claude/skills/evaluating-llms-harness
Start a new session afterwards so the agent picks the skill up. In Claude Code you can also call it directly with /evaluating-llms-harness.
What it does
lm-evaluation-harness - LLM Benchmarking Quick start lm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics. Installation: Evaluate any HuggingFace model: View available tasks: Common workflows Workflow 1: Standard benchmark evaluation Evaluate model on core benchmarks (MMLU, GSM8K, HumanEval). Copy this checklist: Step 1: Choose benchmark suite Core reasoning benchmarks: - MMLU (Massive Multitask Language Understanding) - 57 subjects, multiple choice - GSM8K - Grade school math word problems - HellaSwag - Common sense reasoning - TruthfulQA…
Excerpt from the skill's own SKILL.md. Read the full file on GitHub before installing: skills run with your agent's permissions.
Categories
Related skills
- faiss
Orchestra-Research/AI-Research-SKILLs
Facebook's library for efficient similarity search and clustering of dense vectors. Supports billions of vectors, GPU acceleration, and various index types (Flat, IVF, HNSW). Use for fast k-NN search, large-scale vector retrieval, or when you need pure similarity search without metadata. Best for high-performance applications.
- mlflow
Orchestra-Research/AI-Research-SKILLs
Track ML experiments, manage model registry with versioning, deploy models to production, and reproduce experiments with MLflow - framework-agnostic ML lifecycle platform
- serving-llms-vllm
Orchestra-Research/AI-Research-SKILLs
Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.
- tensorrt-llm
Orchestra-Research/AI-Research-SKILLs
Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.
- peft-fine-tuning
Orchestra-Research/AI-Research-SKILLs
Parameter-efficient fine-tuning for LLMs using LoRA, QLoRA, and 25+ methods. Use when fine-tuning large models (7B-70B) with limited GPU memory, when you need to train <1% of parameters with minimal accuracy loss, or for multi-adapter serving. HuggingFace's official library integrated with transformers ecosystem.
958 installs 14K - autoresearch
Orchestra-Research/AI-Research-SKILLs
Orchestrates end-to-end autonomous AI research projects using a two-loop architecture. The inner loop runs rapid experiment iterations with clear optimization targets. The outer loop synthesizes results, identifies patterns, and steers research direction. Routes to domain-specific skills for execution, supports continuous agent operation via Claude Code /loop and OpenClaw heartbeat, and produces research presentations and papers. Use when starting a research project, running autonomous experiments, or managing a multi-hypothesis research effort.