On 05/25/2023, at 16:39 UTC a subset of Area 1's email traffic using our mailstream-central region was incorrectly routed to a new service region. Emails deferred in this region between 16:39 UTC and 16:48 UTC were hard bounced to the original sender.
During the 9 minute incident window approximately 24% of the inline email traffic received for the mailstream-central region was hard bounced to the original sender. All messages using our mailstream-east and mailstream-west regions were unaffected by this incident.
An intended change to our Cloudflare Spectrum applications resulted in an unintended routing change, caused by an unanticipated effect on our cloud provider's global load balancer configuration. The hard bounces were caused by our attempt to forward the deferred mail in our mailstream-asia region to the mailstream-central region. An internal policy preventing cross-region forwarding resulted in immediate hard bounce of all the messages that were deferred during the incident window..
Upon discovery and our internal alerting the team changed the Cloudflare Spectrum application configuration to restore email flow to the mailstream-central region. Email returned to normal routing at 16:48 UTC.
We are investigating with our cloud provider the behavior we observed to determine if it is intended or unintended. We will also be changing our documentation of egress IPs to expand the range customers should have configured. The 134.195.26.0/23 will replace 134.195.26.0/24 in the United States section. Lastly, we are adding capability to allow for cross-region forwarding should a similar situation arise or additional flexibility is required should there be disruptions in any one region.