INCIDENT REPORT

Summary:

On Wednesday, 27th April 2022, Gmail experienced delayed email delivery globally for a duration of 1 hour 5 minutes. To our Gmail customers whose business was impacted, we sincerely apologize. This is not the level of quality and reliability we strive to offer you and we are taking immediate steps to improve the platform’s performance and availability. We have conducted an internal investigation and are taking steps to improve our service.

Root Cause:

Google engineers often make configuration changes to optimize system performance. Recently, one such change adversely affected how Gmail manages traffic. As a result, some customers experienced delays in receiving email. A subset of inbound emails in this period were also rejected by our APIs with a request for client systems to retry them.

Remediation and Prevention:

Google engineers rolled back the configuration change, which prevented any further impact. Services were restored for all users within 1 hour and 5 minutes. Google Cloud is committed to learning from incidents like this and continually improving the reliability and robustness of our services. To that end, we will be:

• Re-examining and enhancing the process through which configuration changes are rolled out.
• Examining and improving how traffic is routed through our internal systems when configuration changes are made.
• Rolling out future changes in stages instead of globally.

Detailed Description of Impact:

On Wednesday, 27th April starting 14:30 to 15:35 US/Pacific, Gmail delivery was unavailable or delayed for messages to and from some customers globally for a period of 1 hour 5 minutes. Some customers experienced delays in receiving email. A subset of inbound emails in this period were rejected by our APIs with a request for client systems to retry them. To optimize system performance there was a recent change which adversely affected how Gmail manages the traffic. Google engineering rolled back the change which prevented any further impact.

Mini Incident Report while full Incident Report is prepared

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 27 April 2022 14:30

Incident End: 27 April 2022 15:35*

Duration: 1 hour, 5 minutes*

Affected Services and Features:

Gmail

Regions/Zones: Global

Description:

Gmail experienced delayed email delivery globally. From preliminary analysis, the root cause is suspected to be a recent configuration change. The issue was resolved after the configuration changes was rolled back.

Customer Impact:

Impacted customers would have experienced the following:

• Impacted customers would have experienced up to one hour delay in their email delivery between 27 April 2022 14:30 PT andl 27 April 2022 15:35 PT.
• *Note: customers may have experienced emails arriving after 15:35 because of third party client specific retry logic outside of Google, meaning that even though systems became healthy at 15:35, emails would not arrive until later.