Observability alerts

Nothing paged. Something still changed.

A monitor fires below the threshold that would wake anyone: p99 creeping, an error budget burning down, a signature that is not yet an outage. It sits in a channel until somebody has an afternoon. The workspace comes up with the correlation already done, so the afternoon is a decision rather than an investigation.

The shape

Trigger, context, workspace.

Everything between the event and the first turn is gathered by an automation, so the agent never starts from nothing.

Trigger

Datadog · monitor 88

checkout-api p99 +340ms

Since 4.18.2, forty hours ago. No page fired.

Gathered first

  • The monitor, its history, and when the current level actually began
  • Traces from the slow tail, grouped by what they have in common
  • Every deploy to this service and its dependencies inside the window
  • The same monitor on every other service sharing that dependency
  • Prior times this monitor fired, and what closed them

Composed of

latency-triage

mcp
datadog, sentry
tools
github
ttl
2h

An engineer opens a workspace that has already tied the regression to a change. When the monitor spans several services, the same template fans out one harness per service, each with its own deadline, and a quota that bounds the whole sweep.

What it displaces

Before and after.

A dashboard somebody opens, squints at, and closes

The correlation already done, with the traces that support it

A regression found six weeks later, during an unrelated incident

The deploy it started at, named while it is still cheap to revert

Fanning out across twelve services by hand, or not at all

One harness per service, each with a deadline it cannot exceed

Start with the context already gathered.

Create an account, compose a template, and start your first harness.