DocDrift — Documentation Q&A you can trust in CI

Platform

Everything between your docs and a trusted answer

A serving path with production safeguards, and a quality gate that treats answer quality like a test suite.

Drift gate in CI

Every change is scored against a committed baseline. Regressions fail the build.

5Ragas metrics per push

<0.05drift tolerance vs baseline

0silent regressions shipped

Agentic retrieval

The model decides when to search. Multi-hop questions get a second pass.

Layered guardrails

Injection filter in, grounding checks out. Every answer gets a verdict.

Citation audit

Citations verified against retrieval. Miscites auto-corrected.

Multi-LLM benchmark

Models scored head-to-head on your docs. The champion serves.

Full observability

Per-request traces, verdicts, latency — durable in Postgres.

Pipeline

From markdown to a gated deploy

Five stages. The last one is the point: quality regressions never reach production silently.

Ingest

Markdown chunked, embedded, indexed in Qdrant.

Retrieve

Dense search + MMR, optional BM25 hybrid and rerank.

Answer

Agent loop writes a grounded, source-cited answer.

Evaluate

Synthetic QA scored with five Ragas metrics.

05 · GATE

Drift gate

Regression vs baseline fails the build — bad answers never ship.

ci · llmops-eval.yml

# every push touching docs, prompts, or config
$ python pipeline.py
Running RAG evaluation… 5 metrics × synthetic QA set
faithfulness       0.91   ✓
answer_relevancy   0.88   ✓
answer_correctness 0.84   ✓
Drift check passed. baseline unchanged → deploy continues

Stack

Boring where it should be, sharp where it counts

Python service with pluggable model providers — local Ollama to any OpenAI-compatible endpoint — and optional Postgres/Redis/Qdrant Cloud for stateless deploys.

FastAPIQdrantRagas OllamaHF InferencePostgres RedisBM25Pydantic GitHub ActionsDocker

Docs

Up and running in three commands

Ingest your markdown, ask a question, or call the API. Everything runs locally with zero cloud setup.

Point DocDrift at a folder of markdown, then ask questions grounded in it — from the CLI or the browser.

1Install & configure

# Python 3.11
pip install -r requirements.txt
cp .env.example .env

2Ingest your docs

python -m src.ingestion.cli --all

3Ask a question

python -m src.agentic.cli "How long do auth tokens last?"

Prefer the browser?

Run the service and open the playground — same pipeline, live.

uvicorn src.api.app:app --port 8000   # then open http://localhost:8000

Every answer flows through agentic retrieval, guardrails, and the citation audit. Ask over HTTP:

curl -s http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"question": "What auth does v2 use?"}'

Response

{
  "answer": "Auth v2 uses OAuth2 + JWT. [Source: auth_service_v2.md]",
  "guardrails": { "grounded": true, "has_citation": true },
  "citation_audit": { "verdict": "accurate" },
  "tools_used": ["search_docs"]
}

Endpoints

POST /query	Ask a question — answer, guardrail verdict, citation audit, trace.
POST /query/stream	Same, streamed as Server-Sent Events (step → token → done).
GET /eval	Latest Ragas scores vs the committed drift baseline.
GET /models	Multi-LLM benchmark: champion + every model's scores.
GET /metrics	Aggregate latency, error rate, guardrail pass-rate, cache hits.
GET /health	Liveness + dependency checks (Qdrant, Redis, config).

Set DOCDRIFT_API_KEY to require an X-API-Key header on every route but /health.

One rule for every backend: env var set → external & durable; unset → local fallback. The app is stateless-safe and scales horizontally with these three.

QDRANT_URL QDRANT_API_KEY	Vector store. Cloud Qdrant persists across restarts; unset uses a local path.
DATABASE_URL	Postgres — durable traces, benchmark results, ingest state, feedback.
REDIS_URL	Shared rate-limit counter across instances (behind a load balancer).
OPENAI_BASE_URL OPENAI_API_KEY	Any OpenAI-compatible LLM — HF router, a local vLLM, or Ollama.
DOCDRIFT_API_KEY	API auth. Unset = open dev mode; set = X-API-Key required.

Full stack in one command

docker compose up -d   # api + Qdrant + Ollama

CI runs the drift gate on every push that touches docs, prompts, or config — a regression fails the build before it ships.

Documentation answers
you can trust in CI.