← systems · level 2
oncall-triage
Three bank estates, one triage brain. An alert comes in; a verdict comes out: page, ack or monitor.
alert line — what happens when something breaks in an estate
- alertCloudWatch on AWS, Azure Monitor, or Prometheus over Kafka on KubernetesThree estates on purpose, each monitored its own way, because a triage system that has never seen an Alertmanager webhook or a consumer-lag alert is not credible.ADR-0006 ↗
- ingestone canonical shape, HMAC-signed, PII scrubbed, duplicates droppedThe brain runs on one cloud only; the other estates reach it over HTTPS with a signature. One LLM path to secure and pay for, and each cloud does something different.ADR-0001 ↗
- relayKafka alerts leave the cluster over plain outbound HTTPSA public Kafka endpoint meant a managed cluster and real money. A relay inside the cluster keeps the budget under ten dollars. The alert that says Kafka is down takes the other route, because it cannot travel over Kafka to say so.ADR-0015 ↗
- triagea three-role ADK agent on Claude Haiku reads the alert and the taught known issuesKnown issue: one terse line, no page. New: a researcher characterises it and a reporter writes the page recommendation. The model chooses the hand-off from what the store returned, not a fixed pipeline. Model id and prompt hash are stored on every verdict.ADR-0004 ↗
- verdict + consolepage, ack or monitor, one line of reasoning, and a place to teach the next oneA static console over a small API instead of Slack or e-mail integrations, so the verdict record is the integration point and the demo has no moving parts.ADR-0003 ↗
delivery line — how a change reaches the clouds
- pull requestTerraform plan posted on every PR
- C4 drift gatethe architecture model is checked against Terraform tags and Helm labelsA renamed or deleted container fails the PR that introduces the mismatch, with a table naming which one. Diagrams nobody trusts are the alternative.ADR-0016 ↗
- applyon merge, after approval, over OIDC onlyNo cloud credential exists in GitHub. Images are named by git SHA, so rollback is re-running the workflow with a previous one. A rollback drill was run for real and recorded.ADR-0008 ↗
- runbookevery likely cause cites the incident that taught itA runbook section with no citation is either mechanism-only or speculative. The failures that happened are the ones documented.ADR-0017 ↗
runs on
AWSlambda · sqs · dynamodb · the brainAzurefunctions · azure monitor · one estateKuberneteskind · helm · prometheus · strimzi kafkaClaude Haikuvia ADK and LiteLLMGitHub Actionsplan, apply, weekly estate demo
- Recorded, unattended: an AWS chaos alarm to a known-issue verdict in about 3 to 4 minutes; Azure in about 7, because its monitor evaluates slower.
- Kafka broker scaled to zero: the verdict about Kafka arrived over the other route. The system does not claim its Kafka transport survived a Kafka outage.
- Cloud bill under ten dollars a month, the model a few more, both budgeted and alarmed.