Every decision on payments-rag, with the alternative that lost and the price of the one that won. Writing the cost down is the part that keeps a decision honest.
ADR-0018accepted2026-07-17Public deploy on Fly and Neon, with no login
- chose
- One small Fly machine that sleeps when idle, Postgres on Neon, everything public including the eval and usage views.
- instead of
- Staying local-only, or building accounts to gate the admin views.
- the cost
- The glass-box views are exposed to anyone. Abuse and spend are handled with rate limits and a hard daily cap instead of a login.
supersedes 0013 · full record ↗
ADR-0017accepted2026-07-11Angular front end over a FastAPI backend
- chose
- A real SPA plus an API layer, because a hosted deploy needs the API anyway and the target UX had outgrown the prototype.
- instead of
- Keeping the Streamlit prototype that had served its purpose.
- the cost
- More surface to maintain than a one-file UI.
supersedes 0010 · full record ↗
ADR-0016accepted2026-07-09Reranker built as an LLM cross-encoder, kept out of the live path
- chose
- Score each pair with a model call, run it in evals only, leave the interactive path on plain vector search.
- instead of
- A local cross-encoder model, which would drag a heavy ML stack in for a two-document corpus.
- the cost
- Seconds per query, which is exactly why it never reached the live path. It bought +0.10 recall and hit the fan-out ceiling.
full record ↗
ADR-0015accepted2026-07-06Package by concern, extract the LLM adapter
- chose
- Group modules so the folder tree reads as the pipeline, and give the model client its own adapter beside the embedding one.
- instead of
- A flat package where the layout said nothing about the architecture.
- the cost
- A round of import churn, paid once.
full record ↗
ADR-0014accepted2026-07-04Adopt a retrieval technique only if it moves the number
- chose
- Measure hybrid search and reranking against the recall baseline, keep vector as the default, adopt nothing that does not earn it.
- instead of
- Simply handing the model more chunks, which inflates cost and dilutes context without improving ranking.
- the cost
- Slower to adopt fashionable techniques. Two of them were built and then declined.
full record ↗
ADR-0013superseded2026-06Docker locally, no hosted service
- chose
- Reviewers clone the repo and run it, so there is no cost, no ops, and no secrets story to build.
- instead of
- A hosted demo URL.
- the cost
- No public demo. That calculus changed once articles started linking here, which is what superseded it.
superseded by 0018 · full record ↗
Golden set lives in the repo as YAML
- chose
- Ground truth as a diffable file, so changing it shows up in review.
- instead of
- A database or an external eval service.
- the cost
- Single author, no concurrent editing. In exchange, quietly editing the set to chase a number is visible.
full record ↗
ADR-0011accepted2026-06-15Code in its own repository
- chose
- Keep source separate from planning notes so the repo is standalone software.
- instead of
- One repo holding both code and the surrounding notes.
- the cost
- Two places to keep in step.
full record ↗
ADR-0010superseded2026-06Minimal Streamlit UI
- chose
- The smallest possible query box, to see what retrieval was actually returning.
- instead of
- Building a real front end before knowing what the app should show.
- the cost
- A design ceiling, reached once the app grew three views. Kept as history rather than deleted.
superseded by 0017 · full record ↗
ADR-0009prediction was wrong2026-07-01Strip boilerplate and chunk on sentence boundaries
- chose
- Remove the header and footer that repeat on every page, and pack whole sentences instead of fixed word windows.
- instead of
- Feeding raw page text straight to the embedder.
- the cost
- The hypothesis was that cleaner chunks would sharpen retrieval. Measured: no change at all, 0.34 to 0.34. Kept for cleanliness, and the wrong prediction is recorded rather than buried.
full record ↗
ADR-0008accepted2026-07-01A chunk never spans two pages
- chose
- Chunk within each page, so every citation points at exactly one page.
- instead of
- Concatenating the document first, which keeps ideas whole across a page break.
- the cost
- Context is lost at page seams. Accepted because the product is verifiable citations, and an ambiguous page number undoes the whole point.
full record ↗
The eval judge must be a different model from the one being graded
- chose
- One vendor answers, another grades, on a 0 to 100 scale with a critique.
- instead of
- Exact text match (fails on paraphrase), embedding similarity (too kind to wrong-but-similar answers), or the same model grading itself.
- the cost
- A second vendor dependency and a per-eval bill. The judge itself gets spot-checked by hand, because a judge is not automatically right.
full record ↗
Citations returned as structured JSON, not inline markers
- chose
- The model returns an answer plus a list of chunk ids, so every link is machine-checkable.
- instead of
- Citation markers written inline in the prose, which read more naturally.
- the cost
- A stricter prompt and a schema to conform to. In exchange a hallucinated citation fails loudly instead of rendering a wrong link.
full record ↗
ADR-0005accepted2026-06-15Claude Haiku answers in production
- chose
- The cheap tier, with the model name behind an env var so swapping it is config, not code.
- instead of
- The stronger, roughly three times pricier tier.
- the cost
- Likely lower accuracy on dense spec text. Accepted only because the eval can quantify the gap on demand rather than guessing at it.
forced by a model retirement mid-project · full record ↗
Raw API calls, no orchestration framework
- chose
- Hand-written orchestration against the vendor SDKs, so every prompt is legible.
- instead of
- LangChain or LangGraph, which would write less glue code.
- the cost
- More lines by hand. If orchestration ever grows genuinely complex this decision gets superseded in the open, not quietly violated.
full record ↗
One embedding model, pinned and guarded
- chose
- A single small embedding model, pinned in config, with a dimension guard that rejects a mismatch at insert time.
- instead of
- The larger model at several times the cost, or a local model that pulls in a heavy ML stack.
- the cost
- Changing the model later invalidates every stored vector and forces a full re-embed. The guard makes that failure loud instead of silent.
full record ↗
Vectors live in Postgres, not a dedicated vector database
- chose
- pgvector, so text, metadata and vector sit in one row and citations join trivially.
- instead of
- A specialised vector database, which means a new system to learn, run and pay for.
- the cost
- Good to roughly a million vectors, then this gets superseded. The retriever call is the seam that would change, and it was written to be that seam.
full record ↗
Build it in Python, not the strongest stack in the room
- chose
- Python, because every library in this space is Python-first.
- instead of
- .NET, where the work would have been faster to write but every dependency would fight the grain.
- the cost
- Working outside the primary stack, deliberately. The judgment transfers; the syntax was the cheap part.
full record ↗