Google Workspace Status Dashboard

This page provides status information on the services that are part of Google Workspace. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://workspace.google.com/. For incidents related to Google Analytics, visit the Google Ads Status Dashboard.

Incident affecting Google Voice

Incident began at 2021-07-16 18:44 and ended at 2021-07-16 21:15 (times are in Coordinated Universal Time (UTC)).

Date Time Description
Jul 30, 2021 4:46 p.m. UTC

SUMMARY

On Friday, 16 July 2021 at 11:44 US/Pacific, Google Admin Console, Google Voice, Google Vault, and Google Endpoint Management experienced elevated error rates for administrative operations for a duration of 2 hours, 31 minutes. The issue was mitigated by 14:15 US/Pacific.

We understand this issue has affected our valued customers and users, and we apologize to those who were affected.

ROOT CAUSE

Google Admin Console has multiple backend components to facilitate administration of Google Workspace domains. One such component is a job that handles an administrator’s ability to browse and update settings.

An unexpected queries per second (QPS) increase in traffic caused the service responsible for handling admin settings to be overloaded, which led to throttling of requests. The sudden increase was caused by another service responsible for syncing user’s admin role data. Google has an internal load-shedding safety mechanism to appropriately throttle the problematic service; however, in this case, the safety mechanism unintentionally throttled all traffic instead of only the component causing the increased QPS.

REMEDIATION AND PREVENTION

Google engineers were alerted to the issue on 16 July 2021 at 11:44 US/Pacific and immediately started an investigation. Efforts to scale up the number of servers began at 12:42 to deal with the increase in traffic. Once the nature of the problem was determined, the origin of the increase in QPS was manually throttled, fully mitigating the issue by 14:15.

To prevent recurrence and reduce the impact of similar events, we are taking the following actions:

Repair load-shedding mechanisms to appropriately throttle traffic in order to limit the scope of impact for issues of this type.

Improve internal processes to throttle single clients sending a high number of queries unexpectedly.

DETAILED DESCRIPTION OF IMPACT

Google Admin Console: Admins experienced intermittent elevated HTTP 500 errors when trying to browse or update settings, including Gmail and client-side encryption settings. Identity sync changes may have been delayed until the incident was resolved.

Google Voice: ​​Admins were unable to access most of the private branch exchange (PBX) features in the Admin Console and experienced elevated latency in processing subscription change events.

Google Endpoint Management: Admins experienced elevated HTTP 500 errors when trying to browse or update security related settings.

Google Vault: Retention and Hold flows experienced elevated errors.
Jul 19, 2021 6:45 p.m. UTC An incident report has been posted here.
Jul 16, 2021 9:33 p.m. UTC The problem with Google Voice has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

This issue affected Administrator ability to browse and update settings, but end users were not affected.
Jul 16, 2021 8:22 p.m. UTC We're investigating reports of an issue with Google Voice. We will provide more information shortly. The affected users are able to access Google Voice, but are seeing error messages, high latency, and/or other unexpected behavior.