llm-evaluation

Name: llm-evaluation - Agent Skill
Rating: 5.0 (26449 reviews)
Author: wshobson

26,449

Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.

Download Folder View on GitHub

CLI Install

Recommended

Use the skills CLI to install this skill with one command. Auto-detects all installed AI assistants.

Method 1 - skills CLI

npx skills i wshobson/agents/plugins/llm-application-dev/skills/llm-evaluation

Method 2 - openskills (supports sync & update)

npx openskills install wshobson/agents

Auto-detects Claude Code, Cursor, Codex CLI, Gemini CLI, and more. One install, works everywhere.

Installation Path

Download and extract to one of the following locations:

~/.claude/skills/llm-evaluation/

SKILL.md

references

Skill Instructions

Back

Run with Cloud Agent

No setup needed. Let our cloud agents run this skill for you.

Select Provider

Select Model

Claude Sonnet 4.5

$0.20/task

Best for coding tasks

Environment setup included

LLM Evaluation

Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.

When to Use This Skill

Measuring LLM application performance systematically
Comparing different models or prompts
Detecting performance regressions before deployment
Validating improvements from prompt changes
Building confidence in production systems
Establishing baselines and tracking progress over time
Debugging unexpected model behavior

Core Evaluation Types

1. Automated Metrics

Fast, repeatable, scalable evaluation using computed scores.

Text Generation:

BLEU: N-gram overlap (translation)
ROUGE: Recall-oriented (summarization)
METEOR: Semantic similarity
BERTScore: Embedding-based similarity
Perplexity: Language model confidence

Classification:

Accuracy: Percentage correct
Precision/Recall/F1: Class-specific performance
Confusion Matrix: Error patterns
AUC-ROC: Ranking quality

Retrieval (RAG):

MRR: Mean Reciprocal Rank
NDCG: Normalized Discounted Cumulative Gain
Precision@K: Relevant in top K
Recall@K: Coverage in top K

2. Human Evaluation

Manual assessment for quality aspects difficult to automate.

Dimensions:

Accuracy: Factual correctness
Coherence: Logical flow
Relevance: Answers the question
Fluency: Natural language quality
Safety: No harmful content
Helpfulness: Useful to the user

3. LLM-as-Judge

Use stronger LLMs to evaluate weaker model outputs.

Approaches:

Pointwise: Score individual responses
Pairwise: Compare two responses
Reference-based: Compare to gold standard
Reference-free: Judge without ground truth

Quick Start

from dataclasses import dataclass
from typing import Callable
import numpy as np
 
@dataclass
class Metric

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

return Metric("accuracy", calculate_accuracy)

return Metric("bleu", calculate_bleu)

return Metric("bertscore", calculate_bertscore)

def custom(name: str, fn: Callable):

return Metric(name, fn)

class EvaluationSuite:

def __init__(self, metrics: list[Metric]):

self.metrics = metrics

async def evaluate(self, model, test_cases: list[dict]) -> dict:

results = {m.name: [] for m in self.metrics}

for test in test_cases:

prediction = await model.predict(test["input"])

for metric in self.metrics:

prediction=prediction,

reference=test.get("expected"),

context=test.get("context")

results[metric.name].append(score)

"metrics": {k: np.mean(v) for k, v in results.items()},

"raw_scores": results

suite = EvaluationSuite([

Metric.custom("groundedness", check_groundedness)

"input": "What is the capital of France?",

"expected": "Paris",

"context": "France is a country in Europe. Paris is its capital."

results = await suite.evaluate(model=your_model, test_cases=test_cases)