Google Workspace Status Dashboard

This page provides status information on the services that are part of Google Workspace. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://workspace.google.com/. For incidents related to Google Analytics, visit the Google Ads Status Dashboard.

Incident affecting Google Meet

Incident began at 2021-10-25 11:00 and ended at 2021-10-26 17:00 (times are in Coordinated Universal Time (UTC)).

Date Time Description
Nov 5, 2021 2:27 a.m. UTC

INCIDENT REPORT

Summary

The Google Meet Livestream feature experienced disruptions that caused intermittent degraded quality of experience for a small subset of viewers, starting 25 October 2021 0400 PT and ending 26 October 2021 1000 PT. Quality was degraded for a total duration of 4 hours (3 hours on 25 October and 1 hour on 26 October). During this time, no more than 15% of livestream viewers experienced higher rebuffer rates and latency in livestream video playback. We sincerely apologize for the disruption that may have affected your business-critical events. We have identified the cause of the issue and have taken steps to improve our service.

Root Cause

Our investigation into the incident revealed that the backend message delivery mechanism for Google Meet livestreams that assists with processing experienced an unexpected surge in workload that hit the configuration threshold. As a result, up to 15% of livestream viewers noticed higher rebuffer rates and degraded Meet livestreams.

Remediation and Prevention

The issue was diagnosed by monitoring some affected viewers that reported degraded performance. Google engineers mitigated the impact by reconfiguring the resources required for processing the affected delivery of data and system messages for higher quality of livestream playbacks.

Improvements identified:

  • Increased resource allocation for the backend message delivery system in the short term, and automatically detect message delivery overload in the long term.
  • Enhancements to monitoring systems to capture real time data on quality of livestreams for all volumes of traffic.
  • Alert logic updates to capture spikes in rebuffering rate proactively to help mitigate before any customer impact.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

Starting 25 October 2021 0400 PT and ending 26 October 2021 1000 PT, 15% of Google Meet livestream viewers experienced greater than 5% rebuffer rates for a total duration of 4 hours. Viewers with ongoing livestream events were experiencing rebuffering issues. The affected livestreams experienced greater than 5% rebuffering of total viewing time.
Oct 28, 2021 11:30 a.m. UTC The problem with Google Meet has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Preliminary Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 26 October 2021 11:35

Incident End: 27 October 2021 18:00

Duration: 1 day, 6 hours, 25 minutes

Affected Services and Features:

Google Meet - Livestreams

Regions/Zones: Global

Description:

Google Meet experienced issues for a subset of customers publishing live streams for 1 day, 6 hours, and 25 minutes. From preliminary analysis, the root cause was a configuration change in how livestream media is published to viewers. We will publish a full Incident Report incident once we have completed our internal investigation.

Customer Impact:

  • For affected customers' livestreams, a subset of viewers experienced rebuffering or intermittent frame/audio loss.
Oct 28, 2021 9:25 a.m. UTC Google Meet service has already been restored for some users, and we expect a resolution for all users in the near future. Please note this time frame is an estimate and may change.
Oct 28, 2021 6:39 a.m. UTC Google Meet service has already been restored for some users, and we expect a resolution for all users in the near future. Please note this time frame is an estimate and may change.
Oct 28, 2021 3:17 a.m. UTC Google Meet service has already been restored for some users, and we expect a resolution for all users in the near future. Please note this time frame is an estimate and may change.
Oct 28, 2021 12:56 a.m. UTC Our team is continuing to investigate this issue. We will provide an update by Oct 28, 2021, 3:00 a.m. UTC with more information about this problem. Thank you for your patience.
Oct 27, 2021 10:04 p.m. UTC Our team is continuing to investigate this issue. We will provide an update by Oct 28, 2021, 1:00 a.m. UTC with more information about this problem. Thank you for your patience.
Oct 27, 2021 7:04 p.m. UTC We are continuing to investigate this issue. We will provide an update by Oct 27, 2021, 10:00 p.m. UTC detailing when we expect to resolve the problem.
Oct 27, 2021 6:05 p.m. UTC

We're aware of a problem in which Google Meet users may experience latency and/or buffering while viewing livestream events. We will provide an update by Oct 27, 2021, 7:00 PM UTC.
Oct 27, 2021 4:58 p.m. UTC

Google Meet users may experience latency and/or buffering while viewing livestream events.