Reliability and self-heal

Probes

Most probes must fail twice in a row before an incident opens; a stopped container or a corrupt database acts on the first failure. A passing probe clears the counter for every incident type it can raise.

The playbook

• Service or listener down: restart the container and wait for health; if the logs show an invalid chat token, escalate with the excerpt rather than rewriting a token silently.
• Listener stale twice within an hour: escalate, since that looks like a regression rather than a hiccup.
• Event stream or API down: reload the proxy configuration if it validates, restart the container, and if a deploy happened in the last 24 hours flag it as a rollback candidate.
• Disk pressure: vacuum the journal, rotate logs, prune guardian state.
• Certificate expiring: renew and reload the proxy.
• Anti-flap: the same incident recurring more than the configured number of times in 24 hours stops auto-heal and escalates.

Beyond restarts

• A daily digest summarises incidents and actions.
• A weekly synthetic end-to-end run exercises the real flows against staging.
• A logic watch runs the product’s own test suite against the deployed code in a read-only container and compares with the green baseline: a test that was green and is now red is a regression; a red test not in the baseline is a notification, not an action.
• An autonomous fix pipeline can take a confirmed regression through a scoped patch, local tests, a branch, CI, staging, the synthetic run, merge and production with a health gate and automatic rollback. Sensitive secrets are stripped from its environment and forbidden paths are refused.
• The guardian runs under a service supervisor with a watchdog, so a stalled loop is restarted, not just a crashed one.

Inside the product

• A durable bot outbox: approval messages, room notices and their removals are written to the database and flushed by the loop, so a decision made while the bot is down is still delivered.
• Webhooks retry with backoff and a circuit per endpoint.
• A nightly sweep reconciles the homeserver’s event list with the bot’s own message log and records every message that never made it in; a gap retires when the message turns up later, and a partial sweep is logged as incomplete rather than clean.
• A materialisation watchdog alerts admins when the newest cached score is more than 26 hours old or the window is under-covered.
• The AI failsafe switches providers when the active one stops answering.
• Every job that scores runs off the event loop so a long pass can never freeze the bot; this is guarded by a test.