Summary

On October 17, 2025, we experienced processing delays affecting several source types. Full recovery was achieved by October 18, 2025 at 02:56:50 UTC.

Impact

Affected Sources:

• CloudTrail: Delays of 1-2 hours
• Cloudflare (HTTP, WAF): Delays of 1-2 hours
• GitHub: Delays of 1-2 hours
• Other sources: Minimal lag

Data Integrity:

• No data loss occurred
• Some sources may have experienced duplicate events during the affected timeframe
• All events were ultimately processed and delivered

Timeline (UTC)

• 2025-10-17 06:16:00: Processing delays began
• 2025-10-17 06:21:00: Monitoring alerts triggered
• 2025-10-17 06:21:00+: Investigation began
• 2025-10-17 08:12:50: Major backlogs cleared
• 2025-10-17 23:20:50: Root cause identified and mitigation applied
• 2025-10-18 02:56:50: Full recovery, all sources caught up

Root Cause

A configuration issue related to processing high-latency data sources caused resource contention in our event processing pipeline. This affected processing capacity for all sources sharing the same infrastructure.

Resolution

The issue was resolved by adjusting source configurations and will be prevented in the future through:

Action Items

Short-term:

• Implement monitoring of s3 object processing time
• Add alerting for early detection of similar issues

Long-term:

• Architectural improvements to prevent cross-source performance impacts
• Capacity planning for geographically distributed data sources

Customer Impact

If you experienced delays in your log data during this timeframe, all data has now been processed and is available for analysis. If you notice any duplicate events from October 17, 2025, this is expected and can be deduplicated using event IDs if necessary.

For questions or concerns, please get in touch: contact@runreveal.com