Incident Summary

On 05/06/2022, at 11:06 UTC a subset of Area 1’s email traffic wasn’t delivered to their intended recipients. Senders received hard bounces for these messages during the incident window. The incident was detected at 12:57 UTC; and remediated at 13:09 UTC. All email traffic is being processed as expected and customers who may have been affected during the incident window are operating under normal conditions.

Impact

During the incident approximately 3.69% of email traffic was hard bounced to the original sender. Area 1 has shared information about those hard bounces with customers whose traffic was affected during the incident window.

Root Cause

A single Area 1 email processing node within our US-East operating region entered the service without the appropriate configuration of valid customer domains that needed to be processed and relayed. As a security measure Area 1’s email security service is operated as a ‘closed relay’ to ensure that traffic for only validly configured domains are accepted and processed. This allows us to protect our customer’s traffic against bounce attacks and Email DDOS attacks

Since the errant node had no domains configured, any messages routed to that node were hard bounced to the sender during the incident window.

Resolution

Upon determining the affected node, it was removed immediately from the processing cluster and the bounces stopped at 13:09 UTC. Other nodes in the system were processing email in the right manner and automatically took over any traffic being routed to Area 1.

Corrective Actions

Area 1 is adding health checks and additional monitoring to determine node health along with supplementing our current checks to prevent misconfigured nodes from entering the processing cluster.  In addition we are also adding additional proactive monitoring for hard bounces for any traffic addressed to customer domains in order to reduce the time an unhealthy host is operational within a cluster.‌

Should you have any questions or need additional information, please reach out to your Area 1 customer support or customer success representative at support@area1security.com