Lab

questions being tested · 3 open · 3 answered · a "no" counts as an answer

open · adk-learning · a week inside Google ADK, having already built the roles-and-mentor version by hand

What does a standardised agent framework give you that a hand-rolled one doesn't?

All seven days built and tested: single agent, pipeline, delegation, RAG grounding, cross-session memory, deploy with evaluation, governance capstone. The short version: the framework decided less than expected, and the hand-rolled version had already made most of the same choices. The two-part write-up is drafted, not yet published, so the verdict stays open here until it is.

the sprint repo ↗

answered · payments-rag · recall stuck at 0.60

What does hybrid search actually buy on a two-document corpus?

0.60 → 0.60

No lift at all. Big-corpus results from the literature did not transfer. Vector stayed the default.

the measurement write-up ↗

answered · payments-rag · the fashionable next move

How much of a retrieval miss can a reranker actually rescue?

0.60 → 0.70, ceiling

Only the ones already fetched. It promoted every reachable page and then hit a hard ceiling, at seconds per query. Built, measured, left out of the live path.

the measurement write-up ↗

answered · payments-rag · two evals disagreeing

Why do two evals of the same system disagree?

6/10 → 9/10

The metric. It checked for one labelled page while the answer sat on a different, equally correct one. Score the fact instead of the page and the two evals reconcile.

the measurement write-up ↗

open · payments-rag · open to anyone, no sign-in

How far can a public LLM demo be pushed before the budget bites?

Live behind rate limits and a hard daily cap. Nobody has found the ceiling yet, which is itself the uninteresting half of the answer.

try it ↗

open · payments-rag · every eval rests on it

How do you pick a golden set worth trusting?

Ten questions is a noisy ruler, and picking more of them is not obviously the fix. What makes a question worth including is the open part.

Things that outgrow a question become systems. see what they turned into →