Start with the service
A device, host or database matters because it supports something people use. Start with the service and its dependency relationships, then identify the signals that can explain its behavior. Keep the distinction between a failed component, lost redundancy and actual service degradation.
- Name the service owner and the expected service window.
- Identify the network, compute, storage, database and external dependencies.
- Record where dependency information is observed, inferred or missing.
Know whether you can trust the signal
A timeout can reflect a device failure, a credential problem, a collector issue or an unavailable monitoring path. Telemetry needs source, timestamp, freshness and quality context before it becomes operational evidence.
- Distinguish healthy, degraded, unknown, stale and unavailable monitoring states.
- Keep raw observations and derived rates conceptually separate.
- Inspect corroborating and contradictory signals.
Connect the incident to its history
An investigation needs the topology and context that existed when the incident occurred. Today’s map may hide the failed path, moved workload or changed configuration that explains an earlier event.
- Review topology, flow, log, host and configuration evidence in a shared time window.
- Treat recent changes and similar incidents as candidates, not automatic causes.
- Keep earlier conclusions when new evidence changes the diagnosis.
Choose the next check deliberately
The next diagnostic should reduce a specific uncertainty. Bound the target, vantage point, duration and resource budget. Privileged changes need their own permissions, approval and validation.
- Compare observations from more than one vantage point when appropriate.
- Capture an endpoint diagnostic snapshot only within its agreed resource and privacy limits.
- Preserve the result, including when it disproves the current explanation.
Close with evidence you can reuse
A post-incident review should explain the sequence, impact, evidence, chosen actions and remaining uncertainty. A root-cause label alone does not make an incident useful to the next engineer.
- Record recovery validation and evidence provenance.
- Link corrective actions, runbooks and known workarounds.
- Measure operational improvement on actual incident data rather than assumed percentages.
Use these workflows to connect technical capability to the outcome your team needs. Keep the service, time window, evidence and ownership consistent throughout the investigation.