Google Workspace Status Dashboard

Google Workspace Status Dashboard
Incidents
Incident at Mar 28, 2022, 10:56 a.m. UTC

This page provides status information on the services that are part of Google Workspace. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://workspace.google.com/. For incidents related to Google Analytics, visit the Google Ads Status Dashboard.

Available
Service information
Service disruption
Service outage

Incident affecting Google Chat

Incident began at 2022-03-28 10:56 and ended at 2022-03-28 23:00 (times are in Coordinated Universal Time (UTC)).

Date	Time	Description
Apr 6, 2022	4:04 p.m. UTC	Summary On 28 March 2022 03:56 US/Pacific, Google Chat experienced degraded functionality when creating new direct chats and new chat spaces using the Create space feature for a duration of 12 hours, 4 minutes. Approximately 0.32% of users experienced errors creating new direct chats, and 1.26% of users experienced errors creating new chat spaces. Some customers were able to create new spaces on retry for the duration of this incident. We sincerely apologize to our customers using Google Chat who were affected by this issue. We have conducted an internal investigation and are taking steps to improve our service. Root Cause The root cause of the issue was identified to be a rollout on March 25, 2022 15:19 US/Pacific to the database service used by the Google Chat back-end. Google routinely deploys service upgrades to improve the performance and reliability of our applications. In this case, the rollout contained a bug which caused the eventual exhaustion of open connection channels for certain server requests to the database service. This exhaustion of open connection channels caused an increased delay for Create space requests and creating new direct chats. Remediation and Prevention Once the root cause investigation identified the specific database service rollout affecting the Google Chat application, our engineers initiated a rollback and restarted select database tasks to expedite recovery, resulting in significant reduction in error rates. We are working on additional improvements to capture real-time client impact and error detection. We sincerely apologize for the length and severity of this incident. We are taking immediate steps to prevent a recurrence and improve reliability in the future.
Mar 29, 2022	8:33 p.m. UTC	We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using - https://cloud.google.com/support or by using help article - https://support.google.com/a/answer/1047213. (All Times US/Pacific) Incident Start: 28 March 2022 03:56 Incident End: 28 March 2022 16:00 Duration: 12 hours, 4 minutes Affected Services and Features: Google Chat - Create Space feature and Create direct chat messages Regions/Zones: Global Description: Google Chat users experienced issues with the “Create space” feature and experienced degraded performance when creating new spaces and new direct messages chats. Some Google Chat users were unable to create new space for 12 hours, 4 minutes. From preliminary analysis, the root cause of the issue was a rollout to the database service. The issue caused delays in response times that increased the rate of deadline expiration for server requests to the database related to creating new spaces. Customer Impact: Some Google Chat customers saw degraded performance and were unable to create new spaces. Some users also experienced issues with creating new direct message chats.
Mar 28, 2022	3:44 p.m. UTC	The problem with Google Chat should be resolved for the vast majority of affected users. We will continue to work towards restoring service for the remaining affected users, but no further updates will be added to this dashboard.
Mar 28, 2022	2:52 p.m. UTC	Google Chat service has already been restored for some users, and we expect a resolution for all users in the near future. Please note this time frame is an estimate and may change. Create Space function is failing. Re-trying to Create Space should work.
Mar 28, 2022	2:24 p.m. UTC	We are experiencing an issue with Create Space feature beginning around 03:00 US/Pacific. Symptoms: Create Space feature is failing. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2022-03-28 08:00 US/Pacific with current details. We apologize to all who are affected by the disruption.

Times are listed in Coordinated Universal Time (UTC)