Skip to content

Observability Standard

If we cannot see what a system is doing in production, we cannot operate it, debug it, or prove it works. This standard defines the minimum every deployed service must emit and how alerts work.

Every deployed service has

  • Structured logs - JSON (or the platform's structured format), one event per line.
  • Metrics - at least the four golden signals: request rate, errors, latency (p50/p95/p99), and saturation (CPU, memory, queue depth, connection pools).
  • Health endpoints - a liveness check (the process is up) and a readiness check (it can serve).
  • Tracing for services that call other services - propagate a trace ID across every hop (OpenTelemetry by default).
  • A documented home - where the logs, dashboards, and alerts live is written in docs/ARCHITECTURE.md (Observability section).

Logging rules

  • Every log line carries: timestamp (UTC, ISO 8601), level, service, environment, version (release tag), and the request or trace ID.
  • Levels mean something: ERROR needs a human, WARN is unexpected but handled, INFO records business events, DEBUG is off in production.
  • Never log secrets or personal data - no tokens, passwords, full card numbers, or raw request bodies containing them. Redact at the logger, not by convention (see the Data Privacy Standard and Secrets Management).
  • Log errors once, where they are handled, with the context needed to act - not at every layer.

Service level objectives

Each production service defines, in docs/ARCHITECTURE.md:

Item Example
SLI - what we measure Share of /api/* requests answered successfully in under 500 ms
SLO - the target 99.5% over a rolling 30 days
Error budget - the allowed miss 0.5% - about 3.6 hours a month

When the error budget is spent, reliability work takes priority over new features until it recovers.

Alerts

  • Alert on symptoms users feel (SLO burn, error rate, unavailability), not on every cause.
  • Every alert is actionable and links to a runbook section saying what to check first.
  • Every alert has an owner (a team, never one person) and a severity.
  • An alert that fires and needs no action is fixed or deleted - noisy alerts train people to ignore them.

Releases are observable

  • Every deploy is visible on dashboards (an annotation or the version label on metrics).
  • After a production release, the releaser watches the golden signals for a defined window (15 minutes by default) (proposed - pending owner confirmation in #16) before calling it done; a regression triggers the release's rollback plan (Release Management).

Definition of done

A feature is not done until you can tell from the dashboards whether it is working.