Delayed email processing

Incident Report for Area 1 Security

Postmortem

Event Details:

On December 10th, 2021 between 21:46 UTC and 22:36 UTC, the mail processing system at Area 1 experienced sporadic mail delays in two processing regions, though all protection and email delivery remained intact and fully functional.

Root Cause & Actions

The delay was due to our link analysis subsystem evaluating too many links simultaneously. During the incident the average time spent in queue before messages were processed, normally less than 1 second, rose to an average of 8 minutes in the affected regions. All messages were successfully queued and we did not observe delays in messages being accepted and entering the queued state. No messages were lost during the event.

The Area 1 engineering team has developed and deployed added measures to ensure our link analysis system does not cause message processing delays in the future.

Posted Dec 20, 2021 - 14:27 UTC

Resolved

The queues have returned to normal and all systems are operating normally. A postmortem will be posted in the coming days.

Posted Dec 10, 2021 - 22:36 UTC

Monitoring

The Area 1 team has resolved the issue and is monitoring the queues being worked off. All emails are being successfully processed and customers remain protected.

Posted Dec 10, 2021 - 22:30 UTC

Identified

The Area 1 team has identified the issue and is working to mitigate it. The queues have begun to decrease. All emails are being successfully processed and customers remain protected.

Posted Dec 10, 2021 - 22:22 UTC

Investigating

The Area 1 team is investigating delays in processing emails. All emails are being successfully processed and customers remain protected.

Posted Dec 10, 2021 - 22:19 UTC

This incident affected: Email Protection Service.