Event Details

On August 11th, 2020 between 22:51 and 23:17 UTC, the Email Protection Service in the US East region experienced sporadic mail delays causing deferred delivery queues to increase for a small period of time.

Root Cause & Actions

The delay was caused by a database termination due to power loss or underlying hardware failure in the CSP. This immediately resulted in a failover event of the database, as designed. When the database came online again deferred messages were processed automatically and queues were returned to normal levels.

The team is evaluating additional protections to reduce the duration of time messages are deferred should a similar event occur in the future.

Posted Aug 12, 2020 - 20:03 UTC

Resolved

The issue has been resolved and all systems are operating normally. A postmortem will be posted after the investigation is complete.

Posted Aug 11, 2020 - 23:17 UTC

Monitoring

The system is recovering and deferred message queues are declining. Expecting the issue to resolve shortly.

Posted Aug 11, 2020 - 23:08 UTC

Investigating

Our team has noticed higher than normal email processing times and are investigating. If you are experiencing significant delays and need immediate assistance please contact support@area1security.com.

Posted Aug 11, 2020 - 23:01 UTC

This incident affected: Email Protection Service.