| Date / Time (Pacific Time) | Event | Action |
|---|---|---|
| 09/05/2019 - 11:30 am PST | Environment variable for production firewall rule assigned to the wrong network | While fixing a bug and applying a new firewall rule for a customer configuration the service change occurred. |
| 09/05/2019 - 11:51 am PST | Area 1 monitoring systems alerted | Area 1 personnel began investigation into reported issues and working on an action plan to get it addressed. |
| 09/05/2019 - 12:16 PM PST | Customer reports they are experiencing DNS issues | Area 1 personnel alerted and respond to customer queries. |
| 09/05/2019 - 12:30 pm PST | Issue resolved | All systems returned to normal state and behavior. |
At approximately 11:30 am PST on September 5th 2019, Area 1’s DNS service experienced a behavior where customers began seeing sporadic failures in DNS resolutions. In cases where appropriate fail-over was configured, DNS requests began falling back to customer configured fail-over nodes.
In the process of tightening Area 1’s security around firewall rule configurations, Area 1 operational personnel discovered that a specific cloud provider API does not behave as documented. As a result the bug fix to enhance the security of configuration rule changes inadvertently caused production firewall rules to be applied to the test environment. Consequently the intended fix removed the same rule from our production systems; thereby preventing customer ingress IPs to be blocked from resolving DNS queries.
Area 1 immediately took several actions to address the issue immediately, and to prevent future recurrence.