Incident Report

Summary

On June 16, a large burst of GCP webhook log volume overwhelmed our GCP webhook receiver. Our webhook receiver handles HTTP requests, writes the contents to S3 for durability, and then those events are processed in a queue. The increase in load caused 503 SlowDown errors while writing to S3 because they were written one at a time instead of in batches, and after several retries those events failed to write to S3. The increase in data being written to S3 caused our queue to backup and GCP webhook data to be delayed. During this time, some amount of GCP Webhook Data was lost due to write failures to S3.

Timeline

• June 16, around 7:22 PM UTC: GCP webhook sources identified as delayed. Root cause investigation begins. No other sources were affected.
• June 16, during incident: Some high-volume GCP webhook sources were manually disabled by the RunReveal team to relieve back pressure on the receiver.
• June 16, around 8:54 PM UTC: Fix implemented. GCP webhook receiver isolated into its own scaled up pool, and batching added for S3 writes.
• June 16, around 11:30 PM UTC: Incident resolved. Backlog fully processed.

Post incident: Internal review finds a bug where marking a source disabled didn't actually stop ingestion. Affected sources notified once discovered.

Root Cause

There were two related but distinct issues. One relates to delayed ingestion; the other relates to lost data.

Delay: OOM kills and queue backup.

The webhook receiver's Go process hit its memory limit under additional request volume. The queue backup that followed was a side effect of the process restarting.

Data loss: PutObject failures.

Webhook events are treated as durability critical: because they arrive as one shot pushes that RunReveal can't pull and retry, they're written directly to S3 as a buffer, intended to protect against loss from a receiver crash. However, the write path to S3 was per event, not batched. When the receiver was scaled up to burn down the backlog, the surge of individual PutObject calls hit S3 rate limits and started failing with 503 errors. Once retries on a given event were exhausted, that event was dropped rather than persisted. This means the durability mechanism itself was compromised during the exact window it was meant to protect against, and some data loss for GCP webhook sources during that period is likely. The scope is not currently known. Batching was implemented to resolve the PutObject failures going forward. A separate, independently discovered bug meant that disabling a source as a mitigation step didn't actually stop ingestion. Events during that period were subject to the same PutObject failure and drop risk described above.

Impact

GCP webhook sources across workspaces experienced delayed ingestion for roughly 4 hours (about 7:22 PM to 11:30 PM UTC). No other sources were affected. Some data loss for GCP webhook sources during the PutObject failure window is likely, as events that exhausted retries were dropped rather than persisted to S3. The number of affected events and which sources were impacted is currently unknown and requires further investigation. Affected sources also incorrectly showed as disabled without prior notice due to a separate bug, which caused concern about data loss or unauthorized access; that concern is now understood to have had a basis in fact, unlike the initial assessment.

This incident was resolved and the backlog was fully processed.

A fix has been implemented and progress is being made to burndown the backlog of GCP webhook sources.

Our GCP Webhooks Sources are delayed. We've identified the root cause and are working on a solution. We'll keep this page updated once we've implemented steps to remediate the issue.

No other sources should be affected at this time.