BM25¶

BM25 is a classic keyword-based (lexical) retriever. It needs no embeddings and serves as the standard baseline for RAG evaluations — does your expensive vector retriever actually beat BM25?

When should you use this?¶

Use it as a baseline comparison against dense vector retrievers, or when your documents are keyword-heavy (code, part numbers, short factual answers) where semantic search adds little value.

Prerequisites¶

pip install rank-bm25

Step 1 — Install¶

pip install rank-bm25

Step 2 — Prepare documents¶

BM25 needs documents at construction time. Provide them as a JSON list:

documents.json

[
  {"content": "Python is a high-level programming language.", "id": "1"},
  {"content": "RAG combines retrieval with language model generation.", "id": "2"},
  {"content": "PostgreSQL supports vector search via the pgvector extension.", "id": "3"}
]

Step 3 — Configure¶

config.yaml

retriever:
  provider: bm25
  settings:
    documents_path: documents.json
    # k: 5               # default results to return
    # tokenizer: whitespace  # whitespace | simple

No embedder block — BM25 is pure keyword matching.

Step 4 — Run¶

oaeval run config.yaml

Python SDK example¶

eval_bm25.py

import asyncio

from openagent_eval.config.models import Config, RetrieverConfig
from openagent_eval.core.engine import Engine

config = Config(
    dataset={"path": "data/questions.json"},
    llm={"provider": "mock"},
    retriever=RetrieverConfig(
        provider="bm25",
        settings={"documents_path": "documents.json"},
    ),
    metrics={"retrieval": ["context_precision", "context_recall", "mrr"]},
)

engine = Engine(config)
report = asyncio.run(engine.run(dataset))
print(report.summary["metrics_summary"])

All configuration options¶

Option	Type	Default	Description
`documents`	`list[dict] \\| null`	`null`	Inline document list (`{"content": str}`).
`documents_path`	`str \\| null`	`null`	Path to JSON list or JSONL file.
`k`	`int`	`5`	Default number of results.
`tokenizer`	`str`	`whitespace`	`whitespace` or `simple` (lowercasing).

How scoring works¶

BM25 raw scores are min-max normalized into [0, 1] across the result batch. This makes scores comparable across different queries.

Troubleshooting¶

Empty results — Check that documents_path points to a valid file with at least one document.
ImportError: rank_bm25 — Install with pip install rank-bm25.
Weird scores — BM25 favors exact keyword matches. Try tokenizer: simple for case-insensitive matching.

Want semantic search? Use a vector retriever like Chroma or Memory.
Compare BM25 vs vector retrieval in the same evaluation by running both configs side by side.
Pair with an LLM from ../llm/index.md.