Installation¶

This page covers installing OpenAgent Eval, both as a user and as a contributor.

Requirements¶

Requirement	Version
Python	>= 3.11
pip	latest
(Recommended) uv	>= 0.4

OpenAgent Eval is local-first: no external services or accounts are required.

Install from PyPI¶

The fastest way to get started is via pip:

pip install openagent-eval

Verify the installation:

oaeval --version
oaeval --help

You can also use uv:

uv pip install openagent-eval

Some retrievers and embedders require extra dependencies (e.g. chromadb, sentence-transformers, faiss-cpu, qdrant-client). Install the full set with:

pip install "openagent-eval[all]"

Install with uvx (no persistent environment)¶

uvx openagent-eval --help

Install from source (development)¶

Use this workflow if you want to contribute or run the latest unreleased changes.

# Clone the repository
git clone https://github.com/OpenAgentHQ/openagent-eval.git
cd openagent-eval

# Create a virtual environment and install dependencies
uv sync

# The CLI is now available inside the environment
uv run oaeval --help

Setting up API keys¶

OpenAgent Eval reads provider credentials from environment variables. Create a .env file in your project root (or export them in your shell):

# .env
OPENAI_API_KEY=sk-...
GOOGLE_API_KEY=...
ANTHROPIC_API_KEY=...
GROQ_API_KEY=...
OPENROUTER_API_KEY=...

No keys? Use Ollama

OpenAgent Eval supports fully local models through Ollama. Set provider: ollama in your config and no API key is required.

Next steps¶

Follow the Quickstart to run your first evaluation.
Browse the CLI Reference for every command.
Explore the API Reference to embed evaluations in your test suite.

Troubleshooting¶

If oaeval is not found after install, ensure the install location is on your PATH:

python -m openagent_eval --help

Running oaeval doctor checks your environment and dependency versions:

oaeval doctor