Chasing 56 firing alerts on the homelab
Value
3.1
■■■□ of 4
Time
1h 22m
active
Elapsed
2h 15m
wall-clock span
Commits
1
from this session
Cost
$11.85
model spend
Tokens
83k
uncached in, plus out
Tool calls
146
by the agent
Outcomes
10
8 left open
Summary
Written by the summariserFind the root cause of the 56 firing alert instances on Rob's m7kni Grafana stack, starting with the failed brewmdm prod database dump, and fix what was safe and approved.
Rob asked for root-cause investigation of the firing alerts, using gcx and Grafana Assistant, and the agent ran four read-only investigations in parallel. Most alerts turned out to be rule-generator defects or long-standing conditions rather than new outages, and the failed prod database dump was a known fix that had never reached the host. With Rob's approval, the agent re-ran the pgbackup role limited to brewmdm-db, which disabled the prod dump timer while leaving the stage dump working. The agent then pushed fixes to the cf2otel, graph2otel, Bumblebee, agentic-journal and tailscale2otel repositories and corrected several alert rules. Firing instances fell from 56 to about 7, though the Tampooni Defender sign-in, the tailscale2otel coverage threshold and a few unexplained alerts remain open.
What came out of it
- brewmdm-pg-dump@prod timer disabled on brewmdm-db via the pgbackup role (limited to the backup tag); the prod unit no longer shows failed, and the stage dump timer stays enabled and restorable.
- BME-0030 step for the disabled prod timer ticked with a note; the task note is committed locally but not pushed.
- cf2otel commit [commit] pushed: certificate rule excludes Cloudflare's backup_issued standby state, staleness is based on each collector's polling interval, and tunnel interval is corrected; CI published the rules and the new rule was read back from Grafana.
- graph2otel commits [commit] and [commit] shipped: Intune report exports are spread out and retry 429s, the Purview retention_labels collector and the expired Microsoft-managed credential alert are fixed, and the x/net bump restored green CI. Watchtower deployed [commit] to camden at 16:50Z, and throttle events dropped from about 46-57 per run to about 4 with no failed exports.
- Bumblebee catalog-stale threshold changed to 30h in chat-personal and the bumblebee-intune mirror; project scan limit raised from 15 to 30 minutes and deployed via the Intune installer (2026-10-09-projectcap-005); new BumblebeeHostSilent rule added for mbp14, with macmini excluded.
- Corrected Anthropic daily-spike and $100/day spend rules live in a new anthropic-alerts folder; bundled rules left untouched.
- agentic-journal commit [commit] deployed to camden and reported healthy: stuck redaction conversations stop retrying, the error-ratio alert counts only HTTP/gateway failures, and a separate invalid-output rule was added.
- tailscale2otel: release v5.2.0-rc.1 published; EKS image pin bumped in commit [commit]; advisory fixes (acknowledge key for the persistent flow store, read-only ConfigMap mount check) pushed in [commit]; auto-update rule now excludes non-updatable device tags; Tailscale auto-update enabled on camden.
- macmini confirmed by Rob as intentionally retired after a Intune portal wipe on 2026-09-30; it is excluded from the host-silence alert.
- Memory notes corrected for brewmdm (fix exists but was unapplied), tailscale2otel (runs in EKS, not camden), and a new note for the macmini wipe.
Last night my Grafana stack had 14 alert rules firing, covering 56 alert instances. The one that stood out was a failed nightly database dump for the brewmdm production environment. I asked my coding agent to investigate all of them at once, using Grafana's own data and its investigation tooling, with a separate read-only investigation for each group of causes. I wanted findings before anything changed.
The database dump turned out to be an old problem, not a new one. The production Postgres container was never deployed, so the dump job fails almost immediately every night, and it has never succeeded. The stage dump has worked all along, so no real data was at risk. A fix had been written the previous month to enable the timer only for environments that actually exist, but it had never been applied to the host. A reboot cleared the failed state, the next run failed again, and the alert fired. I approved re-running that role against the one database host. The prod dump timer is now disabled, and the stage backups are untouched and still restorable.
Most of the other alerts weren't outages either. Twenty-three certificate alerts were firing on the normal standby certificate that each Cloudflare zone keeps, which my rule treated as a failure. Several collectors that poll hourly were marked stale because one staleness threshold applied to all of them. Intune report exports were hitting Microsoft's rate limit, and the Bumblebee scan limit was too tight. I fixed the rules in their generators and pushed fixes to cf2otel, graph2otel, Bumblebee, agentic-journal and tailscale2otel. The tailscale2otel release candidate is now running in EKS.
Firing instances are down from 56 to 7. A few things are still open:
- Intune compliance stays below 90 percent until I finish Defender sign-in on one machine and retire three old ones.
- Tailscale auto-update coverage reads 0.76 against a 0.8 threshold. Some hosts can't update themselves, and two were unreachable over SSH.
- A few alerts should clear on their own over the coming days. Two new ones, an OPNsense collector alert and a Codex stream alert, haven't been investigated yet.
- Prod Postgres still doesn't exist, so that work stays parked.
Left open
- Prod Postgres has never been deployed, so BME-0030 stays parked; the prod dump is now disabled rather than working.
- Tampooni still fails Defender threat protection until Rob completes Defender sign-in on the phone; HAMRIG, LAPHAM and winsrv still need retiring, so Intune compliance stays below 90%.
- Auto-update coverage reads 0.76 against the 0.8 threshold; scotty, ai and homeassistant still count as without auto-update, and scotty and ai were unreachable over SSH.
- BumblebeeScanTimingOut (7-day window) and the bundled AnthropicDailyCostSpike are expected to clear later but were not confirmed clear.
- The bundled critical AnthropicHighCostThreshold rule is untouched and will page once lifetime spend passes about $1,000 (roughly 69 days away at the current rate).
- OPNsenseCollectorDegraded (new, info severity) and the Codex LB active stream interrupted alert were not investigated.
- Config warnings alert should clear on its next evaluation but was not confirmed; the NodeSystemdServiceFailed duplicates were not confirmed clear in Grafana.
- The tailscale2otel rollout result to EKS was reported but the rollout check output is not shown in the transcript.