Incident Report

Summary

On 24 March 2023, Google Workspace Support Chat experienced issues displaying some support language options for 3 hours, 28 minutes. As a result, some customers may have been unable to choose their preferred support language when starting a chat session.

Root Cause

When Google Workspace customers choose chat support, the Support UI displays a list of languages the customer may choose from before the chat session begins.

On 24 March 2023 at 01:10 US/Pacific, engineers rolled out an update for the Support UI that migrated the language select dropdown to a new select component. This update introduced an issue where the rendering of the select component would cut off some language options when the number of available languages exceeded the select component’s container boundaries.

Remediation and Prevention

Google engineers were alerted to the issue via an internal report on 24 March 2023 at 03:27 US/Pacific and immediately started an investigation. Engineers identified the Support UI change as the cause of the issue at 04:13 US/Pacific and started a rollback of the update at 04:16 US/Pacific. The rollback finished at 04:38 US/Pacific, resolving the issue.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• Update automated testing for Support UI releases to better detect these types of potential UI issues.

Detailed Description of Impact

On 24 March 2023 from 01:10 to 04:38 US/Pacific, Google Workspace Support Chat experienced issues displaying some support language options.

Normally, all languages the customer is entitled to use should appear in the list of languages whether available or not. For the duration of this issue, Workspace support admins may have not seen their preferred language listed at all.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 24 March, 2023 01:10

Incident End: 24 March, 2023 04:38

Duration: 3 hours, 28 minutes

Affected Services and Features:

Google Workspace Support - Chat Support

Regions/Zones: Global

Description:

Google Workspace Support Chat did not display the option for the customer to select a subset of support languages (whether available or unavailable) globally for a duration of 3 hours and 28 minutes. As a result, the list of available languages changed per-session. From preliminary analysis, the root cause of the issue was due to a recent deployment. The issue was fully mitigated once a rollback completed.

Customer Impact:

Normally all languages the customer is entitled to use should appear in the list of languages (whether available or not). Workspace support admins would occasionally not see their preferred language listed at all.

Additional details:

As the language list appeared to change on reload, Workspace support admins could re-attempt by refreshing the support chat and would see a different list. This workaround was not confirmed to work.