Filtered in-app event lists are lagging

Incident Report for Mechanic

Resolved

Filtered event lists are now back to normal, with an average lag of 10-15 seconds.
Posted Dec 18, 2024 - 23:37 UTC

Monitoring

Filtered event lists are down to a ~15min lag. Home stretch.
Posted Dec 18, 2024 - 23:29 UTC

Identified

Right now, *filtered* event lists are lagging by approximately 2.5 hours.

However, events *are* being processed in realtime, with an approximate system Round Trip Time (RTT) of 6 seconds. (See https://status.mechanic.dev/#system-metrics for realtime metrics on this.)

This incident appears to only impact in-app filtered event lists, and does not appear to impact actual event processing.
Posted Dec 18, 2024 - 16:54 UTC

Investigating

We're seeing that filtered in-app event lists (e.g. event searches, events filtered by task id, etc) are lagging, producing results that do not reflect the realtime status of the actual underlying events.
Posted Dec 18, 2024 - 15:16 UTC
This incident affected: Mechanic.