case study · payments-rag · independent system, no commercial client
A RAG that cites the page, measured until the number moved
Ask the SEPA rulebooks a question in plain English and get a short answer with the page it came from. Built from scratch, without a framework, and then used as the bench for every retrieval technique that came after. Most of them lost.
context
Payments engineers read the European Payments Council rulebooks for SEPA Credit Transfer and SCT Inst. The documents are long, formal, and the answer to a practical question is usually one sentence on one page. The system indexes the two 2025 rulebooks and answers with that sentence's page, or says it cannot.
- Cite or refuse. Citations come back as data, never as markers in prose, so a made-up one fails loudly instead of rendering a wrong link. Chunks never span two pages for the same reason.
- Cheap tier by default. The small embedding model and the small generator, with the model name behind config so the quality question stays measurable instead of argued.
- No framework. Five fixed steps written by hand: embed, search, prompt, call, parse. Every step is legible, and every step can be timed on its own.
the measurement
Ten questions with known answers live in the repo as a YAML file, so editing them to flatter a number shows up in review. Two numbers come out of every run: recall@5 at the retriever, which asks whether the answer page reached the top five, and a 0 to 100 score at the answer, graded by a model from a different vendor than the one that wrote it.
The baseline was recall@5 0.60. The single answer the judge scored zero traced to one retrieval miss: the page ranked first when the question used the spec's own wording and ninth when it was asked casually. That trace set the direction for everything below.
what changed, in order
| date | change | the number | verdict |
|---|---|---|---|
| 2026-07-01 | Strip repeated headers and footers, chunk on sentence boundaries | 495 → 484 chunks · recall unchanged | kept for cleanliness; the prediction that it would help is recorded as wrong record ↗ |
| 2026-07-04 | Hybrid search: vector plus keyword, rank fusion | 0.60 → 0.60 | no lift on a two-document corpus; not adopted record ↗ |
| 2026-07-09 | Reranker, an LLM used as a cross-encoder | 0.60 → 0.70 · about a minute per query | promotes only what was already fetched; kept in the eval, out of the live path record ↗ |
| 2026-09-02 | Multi-query: rewrite the question three ways, fuse the results | 0.50 and 0.70 on identical runs · 0.60 with temperature pinned | a dice roll, then the baseline with one hit swapped for another; not shipped record ↗ |
| 2026-09-02 | Contextual retrieval: a short blurb per chunk, embedded with it | 0.60 → 0.80 · one-time re-index, about $0.5 | strictly additive, zero query-time cost; shipped record ↗ |
results
- Retrieval
- Recall@5 0.80, eight of ten questions with the answer page in the top five. Vector search itself is 17.3 ms of a 2,189 ms query; the embedding call is most of the rest.
- Answers
- Mean 96.3 of 100 from the cross-vendor judge, ten of ten above the pass bar of 70.
- Against five other systems
- Same ten questions, same scorer, six systems including OpenAI file search, NotebookLM, Haystack, LlamaIndex and LangChain. This one was the cheapest and the fastest, about three cents for the whole run at 2.68 s mean latency, and fourth on judge accuracy. Faithfulness 0.991.
- Across six model stacks
- A frozen pipeline with only the models swapped, 100 questions over four topics, three judges from three vendors. 66 of 80 answerable questions came out the same on every stack. All six refused every unanswerable question. Swapping the embedder moved scores more than swapping the generator. Whole experiment: $11.56.
handoff
This is an independent system with no commercial client. Nobody paid for it and nobody's operations depend on it. It exists to be measured, and the measurements are public.
What a team would inherit:
- The public repo, with the golden set and the eval as first-class code, not a notebook.
- 23 decision records, each naming the alternative that lost and what the choice costs. read them →
- A live demo behind rate limits and a daily spend cap. try it ↗
- Every number above traced to a run artifact in the repo, and a written record of the two predictions that were wrong.
four questions a recruiter asked, answered straight
- Were payments-rag and oncall-triage built for commercial clients?
- No. Both are independent projects, built and run by me, with no client behind them. That is what makes every number on this page publishable.
- Do they have real users, and at what scale?
- payments-rag is a public demo behind rate limits and a daily spend cap, with no accounts, so there is no user count to quote; readers of the article series try it. oncall-triage triages synthetic alerts from a fictional bank; its users are me and the people reviewing it. tab-librarian is on the Chrome Web Store and in my own daily use. Nobody's operations depend on any of them.
- Any commercial LLM or agent work before July 2026?
- No. Ten years of production software before that, none of it with language models. The AI work starts in July 2026 and everything it produced is linked from this site.
- Interested in a hands-on individual contributor role building production AI systems in Python?
- Yes. That is the work I have been doing since July and the work I want: building, measuring, and owning the system in production. The Python is three months old in production; the judgment under it is ten years old.
the write-ups, in order
- When 60% recall meant 90% accuracy: a RAG measurement story ↗6/10 → 9/10
- The illusion of improvement ↗495 → 484 chunks, 0.60 → 0.60
- Judge strictly to avoid hallucinations ↗refusal 0 · fabrication 0
- Comparing RAGs, Part 2: the benchmark ↗4th on accuracy, 1st on cost and speed
- Improving the scores of a RAG ↗recall@5 0.60 → 0.80
- Swapping every model in a RAG ↗66/80 outcomes identical across 6 model stacks