Google Workspace Status Dashboard

This page provides status information on the services that are part of Google Workspace. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://workspace.google.com/. For incidents related to Google Analytics, visit the Google Ads Status Dashboard.

Incident affecting Google Workspace Support

Incident began at 2022-09-16 17:23 and ended at 2022-09-17 00:27 (times are in Coordinated Universal Time (UTC)).

Date Time Description
Sep 23, 2022 11:02 p.m. UTC

Full Incident Report

SUMMARY:

On Friday, 16 September at 10:23 US/Pacific, Cloud Support Systems, which utilizes a 3rd party product, experienced row-lock errors while updating cases. This preventedSupport agents from being able to update or make changes to some Support cases. Additionally, the send email to customers action was unavailable for some Support cases. Some customers may have seen an error when updating cases from the customer portal. As a workaround, customers and Support agents could successfully update their cases by retrying the action. The issue was mitigated at 17:27 US/Pacific and service was fully restored for both the Support agents and customers. The total duration of the service disruption was 7 hours and 4 minutes.

ROOT CAUSE:

Cloud Support Systems experienced a large backlog of case field update actions pending in the SLO timed actions queue. These timed actions were running in large batches, with updates running in the queue every few minutes. This caused the cases to be locked down from any other update actions while that job was running.

This backlog was caused by the following:

  • There was an increase in the limit of cases being processed every hour by the SLO time queue job.
  • There was a rollback issue with the 3rd party product while processing time based tasks which caused multiple processing requests of the same tasks.
  • There was a configuration in the case trigger which created a backlog on large batches which triggered the above bug with the 3rd party product.

REMEDIATION AND PREVENTION:

Our engineers were alerted to the issue by an internal support agent's case on Monday, 12 September 2022. Engineering began investigating and working on a solution. On Tuesday, September 13 there was one report of external impact. The issue progressed and impact became visible externally on Friday, 16 September 2022 at 10:23 US/Pacific, and the row lock issue was confirmed to have widespread external impact. Once the nature and scope of the issue became clear, Google engineers filed a case with the 3rd party product support to address the issue with Cloud Systems. The backlog actions were purged manually using the 3rd party setup UI which brought the backlog in the queue down by 80%, the issue was completely resolved for the end user on Friday, 16 September at 17:27 US/Pacific.

Google is committed to preventing recurrence by implementing the following action items:

  • Implement a code enhancement with the 3rd party vendor to confirm the error handling code which triggered the issue is fixed.
Sep 23, 2022 10:55 p.m. UTC

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 12 September 2022 21:02

Incident End: 16 September 2022 17:22

Duration: 3 days, 20 hours, 20 minutes

Affected Services and Features:

Google Cloud Support systems

Regions/Zones: Global

Description:

Google Cloud Support Systems experienced an intermittent issue where case updates from the portal were failing for a brief period. Retrying the same update may have succeeded in some cases. From preliminary analysis, the root cause of the issue is a large backlog of update actions that built up in the entitlement process.

Customer Impact:

  • Less than 0.1% of Google Cloud and Google Workspace customers experienced delays in contacting, providing and getting updates to and from Google Support Agents. The issue impacted support’s ability to send updates to customers via the case management system. Following are the scenarios in which errors were received.

    • While sending emails to customers, the send action is unavailable due to errors.
    • Error while changing case ownership.
    • Error while posting comments on the case.

  • Some customers experienced an error “There was an error adding your comment. Please try again” while updating cases from the support portal.
Sep 17, 2022 12:33 a.m. UTC The problem with Google Workspace Support has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.
Sep 16, 2022 10:53 p.m. UTC We expect to resolve the problem affecting a significant subset of users of Google Workspace Support at Sep 17, 2022, 1:30 a.m. UTC. Please note that this time frame is an estimate and may change.

Issue is affecting all case updates and interrupting the following functions:

  • Unable to send emails/ save drafts from case detail page.
  • Support team unable to change case ownership.
  • Customers are unable to update affected cases from the portal.
Sep 16, 2022 6:13 p.m. UTC We're experiencing an issue affecting less than an estimated 30% of the Google Workspace Support user base. We will provide an update by Sep 16, 2022, 8:00 p.m. UTC detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change. The affected users are able to access Google Workspace Support, but are seeing error messages, high latency, and/or other unexpected behavior.

Issue is affecting all vector cases updates and interrupting the following functions:

  • Unable to send emails/ save drafts from case detail page.
  • Support team unable to change case ownership.
  • Customers are unable to update affected cases from the portal.

Retry after some time after the error is seen