Every decision on payments-rag, with the alternative that lost and the price of the one that won. Writing the cost down is the part that keeps a decision honest. oncall-triage keeps its own 17 records in its repo ↗.
ADR-0023accepted2026-09-02Contextual retrieval: situate each chunk before embedding it
- chose
- One model call per page writes a short blurb for every chunk, and blurb plus chunk embed together. The stored text stays the verbatim passage, so citations do not change.
- instead of
- Rewriting the question at query time, which had just been measured and traded one hit for another.
- the cost
- A one-time re-index at roughly half a dollar for 484 chunks. Recall@5 went 0.60 to 0.80 with zero query-time cost, which is what the reranker could not do.
full record ↗
ADR-0022prediction was wrong2026-09-02Multi-query retrieval, measured first and left out of the live path
- chose
- Rephrase the question three ways in spec vocabulary, retrieve for each, fuse the lists. Exposed in the eval only.
- instead of
- Shipping it on the strength of the playbook that ranked it high on leverage and low on cost.
- the cost
- At default temperature recall was a dice roll, 0.50 to 0.70 on identical runs. Pinned to zero it was 0.60, the baseline, with one question gained and one lost. Recorded as the miss it was.
full record ↗
ADR-0021accepted2026-09-01Model matrix: freeze the pipeline, vary only the models
- chose
- One hand-rolled pipeline, six embedder and generator combinations, four topics, three judges from three vendors, every answer scored by all three and never averaged across judges.
- instead of
- Adding vendor SDKs, or letting the new corpora anywhere near the production index.
- the cost
- Three eval-only providers reached through OpenAI-compatible endpoints, corpus content fetched locally and never committed. Production untouched, by construction.
full record ↗
ADR-0020accepted2026-08-31Restructure so the folder tree says what the system does
- chose
- Packages named after the RAG loop itself: domain, indexing, retrieval, answering, adapters. The /ask use case extracted from the route so the CLI and any future entry point can reuse it.
- instead of
- Leaving the orchestrator loose at the package root, a complete use case inside an HTTP route, and PDF extraction welded into the indexer.
- the cost
- Old import paths are gone, so anything external that reached into the package must update. No behavior changed, which is the point.
full record ↗
ADR-0019accepted2026-08-27Measure the frameworks, in quarantine
- chose
- Haystack, LlamaIndex and LangChain/LangGraph comparators, built strictly inside the comparison harness, run isolated, scored by the same pipeline as everything else. Including the one framework this project famously rejected.
- instead of
- Arguing about frameworks from their documentation, or quietly leaving LangChain out because ADR-0004 already had an opinion about it.
- the cost
- Three more adapters to maintain, and a standing reminder that the production no-framework stance is a claim the comparison could still embarrass.
does not reopen 0004 · full record ↗
ADR-0018accepted2026-07-17Public deploy on Fly and Neon, with no login
- chose
- One small Fly machine that sleeps when idle, Postgres on Neon, everything public including the eval and usage views.
- instead of
- Staying local-only, or building accounts to gate the admin views.
- the cost
- The glass-box views are exposed to anyone. Abuse and spend are handled with rate limits and a hard daily cap instead of a login.
supersedes 0013 · full record ↗
ADR-0017accepted2026-07-11Angular front end over a FastAPI backend
- chose
- A real SPA plus an API layer, because a hosted deploy needs the API anyway and the target UX had outgrown the prototype.
- instead of
- Keeping the Streamlit prototype that had served its purpose.
- the cost
- More surface to maintain than a one-file UI.
supersedes 0010 · full record ↗
ADR-0016accepted2026-07-09Reranker built as an LLM cross-encoder, kept out of the live path
- chose
- Score each pair with a model call, run it in evals only, leave the interactive path on plain vector search.
- instead of
- A local cross-encoder model, which would drag a heavy ML stack in for a two-document corpus.
- the cost
- Seconds per query, which is exactly why it never reached the live path. It bought +0.10 recall and hit the fan-out ceiling.
full record ↗
ADR-0015accepted2026-07-06Package by concern, extract the LLM adapter
- chose
- Group modules so the folder tree reads as the pipeline, and give the model client its own adapter beside the embedding one.
- instead of
- A flat package where the layout said nothing about the architecture.
- the cost
- A round of import churn, paid once.
full record ↗
ADR-0014accepted2026-07-04Adopt a retrieval technique only if it moves the number
- chose
- Measure hybrid search and reranking against the recall baseline, keep vector as the default, adopt nothing that does not earn it.
- instead of
- Simply handing the model more chunks, which inflates cost and dilutes context without improving ranking.
- the cost
- Slower to adopt fashionable techniques. Two of them were built and then declined.
full record ↗
ADR-0013superseded2026-06Docker locally, no hosted service
- chose
- Reviewers clone the repo and run it, so there is no cost, no ops, and no secrets story to build.
- instead of
- A hosted demo URL.
- the cost
- No public demo. That calculus changed once articles started linking here, which is what superseded it.
superseded by 0018 · full record ↗
Golden set lives in the repo as YAML
- chose
- Ground truth as a diffable file, so changing it shows up in review.
- instead of
- A database or an external eval service.
- the cost
- Single author, no concurrent editing. In exchange, quietly editing the set to chase a number is visible.
full record ↗
ADR-0011accepted2026-06-15Code in its own repository
- chose
- Keep source separate from planning notes so the repo is standalone software.
- instead of
- One repo holding both code and the surrounding notes.
- the cost
- Two places to keep in step.
full record ↗
ADR-0010superseded2026-06Minimal Streamlit UI
- chose
- The smallest possible query box, to see what retrieval was actually returning.
- instead of
- Building a real front end before knowing what the app should show.
- the cost
- A design ceiling, reached once the app grew three views. Kept as history rather than deleted.
superseded by 0017 · full record ↗
ADR-0009prediction was wrong2026-07-01Strip boilerplate and chunk on sentence boundaries
- chose
- Remove the header and footer that repeat on every page, and pack whole sentences instead of fixed word windows.
- instead of
- Feeding raw page text straight to the embedder.
- the cost
- The hypothesis was that cleaner chunks would sharpen retrieval. Measured: no change at all, 0.34 to 0.34. Kept for cleanliness, and the wrong prediction is recorded rather than buried.
full record ↗
ADR-0008accepted2026-07-01A chunk never spans two pages
- chose
- Chunk within each page, so every citation points at exactly one page.
- instead of
- Concatenating the document first, which keeps ideas whole across a page break.
- the cost
- Context is lost at page seams. Accepted because the product is verifiable citations, and an ambiguous page number undoes the whole point.
full record ↗
The eval judge must be a different model from the one being graded
- chose
- One vendor answers, another grades, on a 0 to 100 scale with a critique.
- instead of
- Exact text match (fails on paraphrase), embedding similarity (too kind to wrong-but-similar answers), or the same model grading itself.
- the cost
- A second vendor dependency and a per-eval bill. The judge itself gets spot-checked by hand, because a judge is not automatically right.
full record ↗
Citations returned as structured JSON, not inline markers
- chose
- The model returns an answer plus a list of chunk ids, so every link is machine-checkable.
- instead of
- Citation markers written inline in the prose, which read more naturally.
- the cost
- A stricter prompt and a schema to conform to. In exchange a hallucinated citation fails loudly instead of rendering a wrong link.
full record ↗
ADR-0005accepted2026-06-15Claude Haiku answers in production
- chose
- The cheap tier, with the model name behind an env var so swapping it is config, not code.
- instead of
- The stronger, roughly three times pricier tier.
- the cost
- Likely lower accuracy on dense spec text. Accepted only because the eval can quantify the gap on demand rather than guessing at it.
forced by a model retirement mid-project · full record ↗
Raw API calls, no orchestration framework
- chose
- Hand-written orchestration against the vendor SDKs, so every prompt is legible.
- instead of
- LangChain or LangGraph, which would write less glue code.
- the cost
- More lines by hand. If orchestration ever grows genuinely complex this decision gets superseded in the open, not quietly violated.
full record ↗
One embedding model, pinned and guarded
- chose
- A single small embedding model, pinned in config, with a dimension guard that rejects a mismatch at insert time.
- instead of
- The larger model at several times the cost, or a local model that pulls in a heavy ML stack.
- the cost
- Changing the model later invalidates every stored vector and forces a full re-embed. The guard makes that failure loud instead of silent.
full record ↗
Vectors live in Postgres, not a dedicated vector database
- chose
- pgvector, so text, metadata and vector sit in one row and citations join trivially.
- instead of
- A specialised vector database, which means a new system to learn, run and pay for.
- the cost
- Good to roughly a million vectors, then this gets superseded. The retriever call is the seam that would change, and it was written to be that seam.
full record ↗
Build it in Python, not the strongest stack in the room
- chose
- Python, because every library in this space is Python-first.
- instead of
- .NET, where the work would have been faster to write but every dependency would fight the grain.
- the cost
- Working outside the primary stack, deliberately. The judgment transfers; the syntax was the cheap part.
full record ↗