// Skill profile
NVIDIA/SkillEvaluator is an open-source evaluation framework for testing AI agent skills before they reach production. It provides a structured pipeline for benchmarking skill correctness, safety, and performance across configurable test suites. The framework enforces quality gates — pass/fail thresholds on accuracy, latency, token efficiency, and safety checks — so teams can automate the decision of whether a skill is ready to deploy. Designed to integrate into CI/CD pipelines, SkillEvaluator runs headless and outputs machine-readable reports. Currently at 1,850 GitHub stars with +280/day velocity as of November 2026. MIT licensed.
SkillEvaluator answers a question every team deploying agent skills faces: “Is this skill actually good enough to ship?” The framework provides structured evaluation pipelines that test skills against correctness benchmarks, safety checks, and performance thresholds before they reach production. Instead of manually spot-checking skill output, teams define test suites and quality gates that run automatically — in CI, before merge, or as a scheduled regression check.
The project comes from NVIDIA’s AI agent infrastructure team, the same group behind SkillSpector (the security scanner for agent skills). Where SkillSpector checks for vulnerabilities and malicious patterns, SkillEvaluator checks for quality — correctness, reliability, latency, and token efficiency. The two complement each other in a pre-deployment pipeline.
Correctness benchmarking: Define expected outputs for a set of test inputs and measure how often the skill produces correct results. Supports exact-match, fuzzy-match, and LLM-as-judge evaluation modes. Test suites are YAML-defined and version-controlled.
Quality gates: Set pass/fail thresholds on any metric — accuracy, p95 latency, token count, safety score. A skill that fails any gate blocks the deployment pipeline. Gates are configurable per environment (stricter for production, relaxed for staging).
Safety evaluation: Built-in checks for prompt injection resistance, hallucination rate, and output constraint adherence. Skills that handle user input get tested with adversarial inputs automatically.
Performance profiling: Measures token efficiency, latency distribution, and cost per invocation across a test suite. Useful for catching regressions when a skill is updated — did the new version get slower or more expensive?
CI/CD integration: Runs headless with JSON/JUnit output. Plugs into GitHub Actions, GitLab CI, or any pipeline that can run a CLI command and check exit codes.
SkillEvaluator requires well-defined test suites to be useful — the framework evaluates skills, but writing good test cases is still on the team. LLM-as-judge evaluation modes introduce their own variance, so critical gates should use deterministic checks where possible. The framework is model-agnostic but assumes skills follow the SKILL.md convention or expose a callable interface.
Teams shipping agent skills to production who need automated quality assurance. Platform teams building internal skill registries or marketplaces. Organizations with compliance requirements around AI tool validation. Developers who want regression testing for skills the same way they have it for application code.
AI agents that work well with SkillEvaluator.
Official Chrome DevTools MCP server — AI agents can debug, profile, inspect DOM, and analyze web performance.
GitHub's official MCP server — interact with repos, issues, PRs, code search, and notifications via AI agents.
Official AWS MCP servers — AI agents interact with S3, Lambda, EC2, CloudFormation, Bedrock, and more.