Incident Report

Summary

On Monday, 12 December 2022 at 21:30 US/Pacific, some Google Meet users experienced issues joining livestreams globally for a duration of 12 hours and 20 minutes. The issue did not impact iOS users of Meet. To our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we have conducted an internal investigation and are taking immediate steps to improve the resiliency of our service.

Root Cause

Google Meet livestream allows Meet customers to stream meetings to a large number of viewers. It relies on the YouTube streaming infrastructure and the YouTube player to view livestreams in the browser.

The trigger of the incident was a recent software update, which introduced an issue that resulted in the video player failing to recognize Meet livestreams and failing to start live playbacks. The change was initially pushed to 1% of users, and after monitoring for a period of time with no issues observed, Google engineers followed established practices and pushed the change to 100% of users in production.

Errors that occurred during this period, however, were missed during the initial rollout, because our monitoring did not properly incorporate signals from Meet's player. Once the change was pushed to 100%, the issue's frequency increased. The errors that occurred due to the change also led to additional internal API traffic due to retry behaviors, which overloaded our servers.

Remediation and Prevention

Google engineers were notified of the issue through a customer support case on Tuesday, 13 December at 03:12 US/Pacific and immediately began an investigation. Once the scope of the issue became clear, the incident was escalated at 04:53. The incident was further escalated to the YouTube team at 07:44. At 09:20, the root cause was identified, and engineers initiated a rollback of the software update. The incident was fully mitigated at 09:50, once the rollback completed.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

• Implement a fix for the video player retry loop in the Meet player to prevent request overload.
• Implement monitoring and alerting to detect failures to connect to livestreams in Meet and reduce mitigation time for issues of this type.
• Investigate an alternative live streaming platform for Meet and remove dependency on YouTube.
• Enhance testing in the staging environment to identify and prevent regressions before rollout progresses.

Detailed Description of Impact

On Monday, 12 December 2022 from 21:30 to Tuesday, 13 December at 09:50 PT, some Google Meet users experienced issues joining livestreams globally.

Google Meet

• Android and Web Google Meet clients experienced errors, which resulted in a retry loop behavior for the Meet broadcast players.
• Some users would continue to see the error after the rollback completed, as a browser refresh would be required.

Mini Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 12 December 2022 21:30

Incident End: 13 December 2022 09:50

Duration: 12 hours, 20 minutes

Affected Services and Features: Google Meet

Regions/Zones: Global

Description:

Some Google Meet users experienced issues joining Live Streams globally for a duration of 12 hours and 20 minutes. The issue did not impact iOS users of Meet.

From preliminary analysis, the root cause of the issue was due to an experimental rollout of the video player, which stopped recognizing Meet Live Streams. The issue was fully mitigated once a rollback was completed.

Customer Impact:

Some customers were unable to join Meet Live Streams during the period of the outage.

The issue with Google Meet has been resolved for all affected users as of Tuesday, 2022-12-13 09:22 US/Pacific.

The issue did not impact iOS users of Meet. Users were unable to start a livestream on Google Meet. Livestreams of meetings created during the outage did not work. However, those that are created before the start of the issue (2022-12-12 23:00 PST) and created after mitigation time (2022-12-13 09:22 PST) should have worked.

We thank you for your patience while we worked on resolving the issue.