Skip to content

FAQ

Frequently asked questions about OpenAgent Eval.

What is OpenAgent Eval?

An open-source, local-first framework for evaluating RAG systems and AI Agents. Our goal is to become the pytest of AI evaluation — a familiar, composable way to measure quality.

Do I need an API key?

Only if you use a hosted LLM provider (OpenAI, Gemini, Anthropic, Groq, OpenRouter). You can run fully locally with Ollama, or use the built-in mock providers for CI and dry-runs — no API key required.

Which LLM providers are supported?

OpenAI, Google Gemini, Anthropic, Groq, OpenRouter, Ollama, and a mock provider. See Architecture for the base classes and how to add your own.

Which retrievers are supported?

Chroma, Memory, BM25, FAISS, Qdrant, Pinecone, Weaviate, Elasticsearch, PGVector, HTTP, and a mock provider. Implement Retriever to add one.

Which embedders are supported?

Sentence-Transformers and a mock embedder. Local vector retrievers (Memory, FAISS, Qdrant, Pinecone, PGVector) use an embedder; server-side backends (Chroma, Weaviate) embed remotely.

Can I use OpenAgent Eval inside pytest?

Yes. Construct an Engine and call its async run() method with asyncio.run. See Examples.

How do I add a custom metric?

Subclass openagent_eval.metrics.base.BaseMetric, implement evaluate(**kwargs) -> MetricResult, and register it in METRIC_REGISTRY. A template lives at openagent_eval/plugins/examples/custom_metric.py.

Which report formats are available?

Terminal, Markdown, HTML, and JSON, plus a side-by-side comparison report for oaeval compare.

Does it send my data anywhere?

No. OpenAgent Eval is local-first. The only network calls are to the LLM/retriever providers you configure. We do not collect telemetry.

What Python versions are supported?

Python >= 3.11.

How do I deploy the documentation?

Push to main — GitHub Actions builds and deploys the site to GitHub Pages automatically. See Contributing for local preview.

Where do I get help?