Skip to content

OpenAgent Eval

v0.1.0 · Local-first · Apache 2.0

The pytest of AI evaluation

Open-source, local-first framework for evaluating RAG systems and AI agents. A clean CLI, a typed Python SDK, and a pluggable metric & provider architecture — measure quality the way you test code.

Get Started GitHub
pip install openagent-eval
OpenAgent Eval pipeline: config to engine to report

Why OpenAgent Eval

  • Local-First Runs entirely on your machine. No dashboards or accounts required — your data never leaves your laptop.

  • CLI + SDK Drive evaluations from the command line with oaeval, or embed Engine directly in your Python test suite.

  • Framework Agnostic Works with any RAG implementation — LangChain, LlamaIndex, or fully custom pipelines.

  • Pluggable Swap LLMs, retrievers, embedders, and metrics through a clean provider/plugin architecture.

  • Comprehensive Metrics Retrieval, generation, performance, and cost metrics in one consistent interface.

  • Beautiful Reports Terminal, Markdown, HTML, and JSON reports with built-in failure analysis.

Quick Start

From install to insight in minutes

# Install
pip install openagent-eval

# Create a configuration file
oaeval init

# Run your first evaluation
oaeval run config.yaml

# Inspect the report
oaeval report latest

See the Quickstart for a full walkthrough, or jump straight to the CLI Reference.

Architecture

One pipeline, every stage pluggable

A Config describes your dataset, retriever, LLM, and the metrics to compute. The Engine loads the dataset, runs retrieval and generation, scores the results, and persists a report.

flowchart LR
    A[Config<br/>config.yaml] --> B[Engine]
    C[Dataset] --> B
    B --> D[Retriever<br/>Provider]
    B --> E[LLM<br/>Provider]
    D --> F[Context]
    E --> G[Answer]
    F --> H[Metrics]
    G --> H
    H --> I[ReportManager<br/>Terminal / MD / HTML / JSON]

Every stage is pluggable. Read more on the Architecture page.

Evaluation Metrics

Four categories, one consistent score

Metric names map to the built-in registry (openagent_eval.metrics.METRIC_REGISTRY):

  • Retrieval context_precision, context_recall, recall_at_k, precision_at_k, hit_rate, mrr, ndcg

  • Generation faithfulness, answer_relevancy, hallucination, semantic_similarity, exact_match, f1_score, bleu, rouge, bertscore

  • Performance latency

  • Cost token_count

Supported Providers

Bring your own, or use what ships

LLM Providers Retriever Providers Embedders
OpenAI, Google Gemini, Anthropic, Groq, OpenRouter, Ollama, Mock Chroma, Memory, BM25, FAISS, Qdrant, Pinecone, Weaviate, Elasticsearch, PGVector, HTTP, Mock Sentence-Transformers, Mock

Bring your own by implementing the provider base classes — see API Reference.

Python SDK

Embed evaluation in your test suite

import asyncio

from openagent_eval.config.models import Config
from openagent_eval.core.engine import Engine

config = Config(
    dataset={"path": "data/questions.json"},
    llm={"provider": "openai", "model": "gpt-4o-mini"},
    retriever={"provider": "chroma", "settings": {"collection_name": "my_collection"}},
)
engine = Engine(config)
report = asyncio.run(engine.run(dataset))
print(report.summary)

The SDK is fully documented in the API Reference and demonstrated in Examples.

CLI

Six commands cover the whole loop

Command Description
oaeval init Create a configuration file
oaeval run <config> Run an evaluation pipeline
oaeval report <id> View a stored report (latest for the most recent)
oaeval compare <a> <b> Compare two experiments
oaeval list List previous evaluations
oaeval doctor Check environment and dependencies

Full command documentation lives in CLI Reference.

Contributing

Built in the open, by the community

OpenAgent Eval is community-driven. Contributions of every size are welcome — from bug reports to new metrics and providers.

Community

Help shape the roadmap

Stay connected:

OpenAgent Eval — Apache 2.0 License. Built by the OpenAgent Eval Contributors.