Roadmap

OpenAgent Eval is evolving from a RAG evaluation tool into a full AI agent evaluation framework. The roadmap below reflects our current plans — priorities may shift based on community feedback.

v0.3.0 — Current

• RAG evaluation pipeline
• CLI (oaeval) and Python SDK
• Plugin architecture
• Multiple report formats (terminal, markdown, html, json)
• Retrieval, generation, performance, and cost metrics
• 11 retriever providers (Chroma, Qdrant, Pinecone, Weaviate, FAISS, pgvector, Elasticsearch, BM25, HTTP, Memory, Mock)
• Corpus Health Auditor (contradiction, staleness, duplicates, coverage)
• Component Diagnosis (blame attribution, failure modes)
• Synthetic Test Data (question generation, adversarial cases)
• NLI-based metrics (DeBERTa faithfulness, relevancy scoring)
• Comprehensive documentation

v1.0 — Stable Release (planned)

• Generic LLM-as-Judge for custom criteria
• Pytest plugin for RAG evaluation
• Threshold-based test gating
• GitHub Actions workflow example
• Documentation site (this site) and GitHub Pages deployment

v2.0 — AI Agent Evaluation (planned)

• Tool-call evaluation
• Planning / reasoning evaluation
• Memory evaluation
• Multi-agent evaluation
• Dataset versioning and experiment tracking

v3.0 — Platform & Integration (future)

• CI/CD integration and native GitHub Action
• Cloud synchronization of evaluation runs
• Hosted evaluation dashboard
• Team collaboration and shared baselines

How we prioritize

1. Community requests raised in Discussions.
2. Contributor interest — the fastest path is an open PR.
3. Ecosystem gaps — providers and metrics the community needs most.

Get involved

• Propose a feature in Discussions.
• Pick up a planned item and open a PR — see Contributing.
• Ask questions in the FAQ.

Subject to change

This roadmap is indicative. Items may be reordered, merged, or deferred as the project matures.