Skip to content

Retriever Providers

Retriever providers fetch the documents/contexts that OpenAgent Eval measures for retrieval quality (context precision, recall, MRR, …). Every retriever returns Document objects with a relevance score normalized to [0.0, 1.0].

Which one should a beginner pick?

  • Just learning / no infra? Use mock — zero setup, offline.
  • Local, no external service? Use memory with the sentence_transformers embedder — pure NumPy, no server.
  • Already have a vector DB? Use its adapter (chroma, qdrant, pinecone, weaviate, faiss, pgvector).
  • Want a keyword baseline? Use bm25 — no embeddings needed.
  • Have a custom search API? Use http to point at any REST endpoint.

Comparison matrix

Provider Install extra Embedder needed? Server-side embed? Needs key?
Chroma chromadb ❌ ✅ ❌
Qdrant openagent-eval[qdrant] ✅ ❌ optional
Pinecone openagent-eval[pinecone] ✅ ❌ ✅
Weaviate openagent-eval[weaviate] optional ✅ optional
FAISS openagent-eval[faiss] ✅ ❌ ❌
PGVector openagent-eval[pgvector] ✅ ❌ optional
Elasticsearch openagent-eval[elasticsearch] knn mode only ❌ optional
BM25 openagent-eval[bm25] ❌ ❌ ❌
Memory (none — NumPy) ✅ ❌ ❌
HTTP (none — httpx) ❌ ❌ ❌
Mock (built-in) ❌ ❌ ❌

Common configuration

Retriever options live under settings:. Embedders (when required) go under embedder:.

config.yaml
retriever:
  provider: memory
  settings:
    documents_path: data/corpus.json
    k: 5
  embedder:
    provider: sentence_transformers
    model: all-MiniLM-L6-v2

Embedder requirement

memory, faiss, qdrant, pinecone, and pgvector require an embedder. elasticsearch requires one only in mode: knn. weaviate makes it optional (it can embed server-side). chroma, bm25, http, and mock do not need an embedder.

Score normalization

Different backends report relevance in incompatible scales (cosine distance, L2, inner product, raw BM25/Elasticsearch _score). All retrievers funnel raw scores through shared helpers so every Document.score lands in [0.0, 1.0] (higher = more relevant).

Next steps