Data Ingestion Delayed for SaaS Pipeline

We've now returned to a nominal state and are caught up on ingestion and detections.

We'll publish a full post-mortem out as soon as we've pieced together all the information internally with the details of the chain of events that happened and what we'll do to help mitigate this in the future.

For now, we'll be monitoring it closely throughout the weekend. Please stay tuned and subscribe to the blog to receive the update for the post mortem when we publish it.

We identified and remediated the issue with the help of our partners at Clickhouse.

There were issues with the instances that we auto-scaled the database onto and as a result Clickhouse is following up internally and with AWS to do a root cause analysis.

We'll monitor the progress of our queues as they catch up and update customers with a postmortem of the incident and details of what happened once we know everything that happened.

To follow-up from the previous incident report, there was indeed issues with our SaaS RunReveal Clickhouse instance.

We're investigating with the Clickhouse team now. Data ingestion will be delayed for customers of the SaaS pipeline, and customers utilizing BYODB may notice some duplicates as we work to resolve the issue.