Google Workspace Status Dashboard

This page provides status information on the services that are part of Google Workspace. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://workspace.google.com/. For incidents related to Google Analytics, visit the Google Ads Status Dashboard.

Incident affecting Google Voice

Incident began at 2022-07-06 23:12 and ended at 2022-07-07 00:07 (times are in Coordinated Universal Time (UTC)).

Date Time Description
Jul 13, 2022 12:34 a.m. UTC

INCIDENT REPORT

SUMMARY:

On Wednesday, 6 July 2022 16:12 US/Pacific, Google Voice experienced an issue with call routing for a duration of 55 minutes. For the duration of the incident, incoming and outgoing voice calls from Google Voice could not be completed, including any redialing attempts. The issue was mitigated on 6 July 2022 17:07.

We understand that this issue has impacted our valued customers and users, and we apologize to those who were affected. We have conducted a detailed investigation into this incident and are taking steps to improve the service.

ROOT CAUSE:

Google Voice utilizes a backend call router configured to determine the call flow and the product of the call. Based on the internal determination the call routing is completed successfully. A code change deployed to improve the call routing logic caused a misclassification of the incoming and outbound calls. The misclassification of the calls was caused due to a missing criteria field that the code change could not parse successfully. Further investigation into the incident revealed that the call router returned internal error while processing the call and eventually disconnected the calls with a Service Unavailable error.

REMEDIATION AND PREVENTION:

Once the incident was detected, Google engineers began remediation immediately, according to the timeline below (all times in PDT):

  • On Wednesday, July 6, at 4:12 PM, the incident was detected.
  • On Wednesday, July 6, at 4:14 PM, the incident response was initiated by an engineer who conducted an impact assessment.
  • On Wednesday, July 6, at 4:22 PM, the incident was escalated to the wider team and root cause investigation began.
  • On Wednesday, July 6, at 5:03 PM, a misconfiguration in the GTP system was identified, and the configuration was rolled back.
  • On Wednesday, July 6, at 5:07 PM, the configuration was successfully restored to the latest known good version.
  • On Wednesday, July 6, at 5:10 PM, internal verification indicated that the issue was resolved.
  • On Wednesday, July 6, at 5:19 PM, the incident was marked as mitigated.

Google is taking the following steps to improve the user experience and prevent the issue in the future:

  • Improve the continuous deployment strategies to minimize the customer impact before rollout of updates.
  • Further improve validation and unit testing for call routing configuration changes before rollout.
  • Enforce code change blocks when dependencies are identified for high risk configuration changes and updates.
  • Enhance server-side detection logic for call routing configuration updates through continuous deployment.
  • Implement additional metric alerts and improve canary testing for client errors for call routing services.

Google is committed to improving our technology and operations to prevent similar service disruptions in the future. We would like to apologize for the length and severity of this incident. We are taking immediate steps to prevent a recurrence and improve reliability in the future.
Jul 7, 2022 5:16 p.m. UTC

We apologize for the inconvenience this service outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support by opening a case https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 6 July 2022 16:12

Incident End: 6 July 2022 17:07

Duration: 55 minutes

Affected Services and Features:

Google Voice - Calling feature affected

Regions/Zones: Global

Description:

Google Voice experienced failures in completing voice calls for all users. Failures lasted for 55 minutes starting at 6 July 2022 16:12 US/Pacific. From preliminary analysis, the root cause of the issue was identified as a change deployed to route specific type of VOIP calls which caused failure to route all voice calls. The issue was mitigated on 6 July 2022 17:07 US/Pacific after rolling back the identified change.

Customer Impact:

Users experienced failures for voice calls incoming and outgoing from Google Voice. Calls made by redialing also failed to complete.
Jul 7, 2022 12:27 a.m. UTC The problem with Google Voice has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.
Jul 7, 2022 12:14 a.m. UTC We're aware of a problem with Google Voice affecting a majority of users. We will provide an update by Jul 7, 2022, 1:00 a.m. UTC detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change.

Customers impacted by this issue may experience calls made through Google Voice to fail.

The messaging feature is not affected and will continue to work successfully.

We apologize to all who are affected by the disruption.

No workaround available at this time.