Server I/O congestion after a Tailscale update
Tracing an unresponsive weather service back to disk contention on a small ECS instance.
Summary
The Weather Agent became slow and intermittently unresponsive on a small ECS instance. Application-level checks did not immediately explain the slowdown; the useful signal was host disk activity.
Impact
Requests took much longer than usual and the service appeared unavailable for short periods.
Timeline
- The service first presented as an application slowdown.
- Process and host signals were checked to separate application work from resource pressure.
- Disk I/O contention coincided with an automatic Tailscale update.
- Normal responsiveness returned after the competing work completed.
Root cause
The small instance had little I/O headroom. Update activity and the application competed for the same constrained disk, making healthy processes look stalled.
Resolution
The immediate resource contention was allowed to clear, and future diagnosis was changed to include host-level I/O signals before treating the application as the root cause.
What I learned
An application can be healthy and still be unusable when the host underneath it is saturated. On a small instance, disk latency belongs near the start of the debugging checklist.
Follow-up
- Keep operating-system updates visible in the incident timeline.
- Check I/O wait and disk latency alongside CPU and memory.
- Leave enough resource headroom for maintenance work.