Choose an alert signal based on the failure mode you need to detect. Evaluate related signals together to distinguish customer impact from expected traffic changes.

Latency

Alert when latency remains above an agreed service target for a meaningful interval. Prefer a percentile when available because an average can hide a small group of very slow requests. Scope the condition to the affected service or operation.

Error rate

Use error rate to detect a rising proportion of failed operations. Combine a percentage threshold with minimum request volume so one failure during very low traffic does not create a misleading critical alert.

Throughput

Throughput alerts can detect unexpected traffic loss, spikes, or stalled processing. Use time-aware baselines for services with predictable daily or weekly patterns. A throughput change without higher latency or errors may reflect normal demand rather than a fault.

Validate and tune

Test warning and critical conditions separately, verify recovery behavior, and review alert history after rollout. Sampling affects metrics derived from sampled spans; independently generated metrics can have different totals. Use one documented metric source consistently for an objective and its alerts.