Skip to content

Installation

This page covers installing OpenAgent Eval, both as a user and as a contributor.

Requirements

Requirement Version
Python >= 3.11
pip latest
(Recommended) uv >= 0.4

OpenAgent Eval is local-first: no external services or accounts are required.

Install from PyPI

The fastest way to get started is via pip:

pip install openagent-eval

Verify the installation:

oaeval --version
oaeval --help

You can also use uv:

uv pip install openagent-eval

Some retrievers and embedders require extra dependencies (e.g. chromadb, sentence-transformers, faiss-cpu, qdrant-client). Install the full set with:

pip install "openagent-eval[all]"

Install with uvx (no persistent environment)

uvx openagent-eval --help

Install from source (development)

Use this workflow if you want to contribute or run the latest unreleased changes.

# Clone the repository
git clone https://github.com/OpenAgentHQ/openagent-eval.git
cd openagent-eval

# Create a virtual environment and install dependencies
uv sync

# The CLI is now available inside the environment
uv run oaeval --help

Setting up API keys

OpenAgent Eval reads provider credentials from environment variables. Export them in your shell before running oaeval — the CLI does not auto-load a .env file, so if you keep one, source it yourself first (e.g. set -a; . ./.env; set +a):

export OPENAI_API_KEY=sk-...
export GEMINI_API_KEY=...
export ANTHROPIC_API_KEY=...
export GROQ_API_KEY=...
export OPENROUTER_API_KEY=...

No keys? Use Ollama

OpenAgent Eval supports fully local models through Ollama. Set provider: ollama in your config and no API key is required.

See the Environment Variables reference for every variable OpenAgent Eval reads, including how keys are resolved and how they reach the process.

Next steps

Troubleshooting

If oaeval is not found after install, ensure the install location is on your PATH:

python -m openagent_eval --help

Running oaeval doctor checks your environment and dependency versions:

oaeval doctor