Google Workspace Status Dashboard

Summary

On Wednesday, 25 August 2021, Google Chat experienced elevated errors for users connecting to the service for a duration of 1 hour and 12 minutes. To our Google Workspace customers whose business communications were impacted during this outage, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Detailed Description of Impact

From 25 August 2021 23:53 to 26 August 2021 01:05. (US/Pacific), Google Chat experienced elevated errors for users connecting to the service, and affected customers were unable to send or receive new messages. This impacted 25% of users. Web clients were unable to establish a connection and may have seen a "Loading chat..." banner during the incident, while mobile clients were unimpacted.

Root Cause

As part of routine maintenance, new data centers are turned up and provisioned with backend services for Google Chat. This includes a session server component, which handles the registration of chat sessions from client requests. During this phase, traffic is specifically directed away from these new locations until verifications complete and all components are ready to serve new traffic. The root cause was due to a latent issue in this session server component. This component had been updated recently with protections intended to prevent registering new sessions within data centers that were not yet live in production. However, this protection erroneously was not lifted until the component had restarted, leading these tasks to unintentionally reject session registrations from valid traffic. This caused most connection attempts that reached this data center to fail, leading to either a successful retry via another data center or a 502 error response after retries timed out. This was not caught during the data center turn up testing, because most clients on reconnection are reconnected to the same backend data center and would not have needed to register a new session.

Remediation and Prevention

Google engineers were alerted to elevated client registration errors on Wednesday, 25 August 2021 22:41, and began investigating the scope of the errors. Once the impact was clearly isolated to these new data centers, traffic was redirected at 23:14 to mitigate, which was completed by Thursday, 26 August 2021 00:37. Some clients continued to attempt to connect to the old data centers until retry logic timed out, at which point all customer impact ended at 01:05.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. Multiple prevention steps are being taken, such as expanding our staging environments to cover this scenario, as well as improving our session monitoring and mitigation procedures. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 25 August 2021 22:53

Incident End: 26 August 2021 00:54

Duration: 2 hours, 1 minute

Affected Services and Features:

• Google Chat

Description:

Google Chat experienced elevated errors for users connecting to the service for a duration of 2 hours and 1 minute. From preliminary analysis, the root cause of the issue is due to a rollout that modified how sessions were handled, leading to failed connections.

Customer Impact:

• Affected customers were unable to send or receive new messages.
• The Web UI presented the banner "Loading chat..." indefinitely during the incident.
• Up to 35% of Web clients were unable to establish a connection.
• Up to 5% of Android/IOS mobile app clients were unable to establish a connection.