System Outage Due to Spike in Webhook Traffic

2023-02-10T00:51:06-06:00

Feb 10, 00:51 CST
Resolved - As no escalated incidents have been observed, moving this incident to Resolved. The rate limit introduced will stay in place. A solution to handle fetch log discrepancies is being worked upon.

Feb 9, 14:38 CST
Update - We observed an abnormal spike in incoming web-hook traffic which affected our database and platform services.

We’ve rate limited the offending provider’s web-hooks and stopped certain fetch log entries.

We will continue to monitor the traffic till 6 PM CST. While we’re in this phase, certain fetch log entries may contain discrepancies.

Feb 9, 12:52 CST
Monitoring - A fix has been implemented and we are monitoring the results.

Feb 9, 11:53 CST
Identified - The issue has been identified and a fix is being implemented.

Feb 9, 11:26 CST
Investigating - We are currently investigating this issue.