Incident Report

Summary

On Friday, 26 September 2025 at 08:40 US/Pacific, Google Meet users experienced elevated error rates globally for 1 hour, 1 minute. To our Google Meet customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

The root cause of the incident was a flaw in the error-handling logic within the experimentation module's configuration evaluation. This flaw created the conditions for resource contention, where too many processes attempted to access the same system resource at once.

The trigger was a specific change to the Meet experimentation configuration file, which caused an elevated error level during evaluation and led to the contention in the Meet backend. Our monitoring systems detected the issue right away, and engineers quickly identified the change as the source. They reverted the update, which resolved the issue.

Remediation and Prevention

Google engineers were alerted to the issue through the internal alerts at 08:49 US/Pacific on 26 September, 2025 and immediately started an investigation. They identified several recent changes as potential causes, and began systematically rolling back those changes.

At 09:25 US/Pacific, engineers reverted the Meet API Server binary to a previous version.

At 09:41 US/Pacific, engineers reverted and paused subsequent Meet backend API server configuration rollouts across all Google Meet production clusters thus mitigating the issue.

At 11:52 US/Pacific, engineers reverted the change that caused lock contention.

We sincerely apologize to our customers who were impacted by this disruption. This is not the level of quality and reliability we strive to offer, and we are taking the following steps to ensure this does not happen again. The Google Meet Engineering team is accelerating the deprecation of legacy code that caused lock contention.

Detailed Description of Impact

On Friday, 26 September 2025, from 08:40 to 09:41 US/Pacific, users were either unable to connect to Google Meet or experienced problems with establishing meetings. Users already in Google Meet sessions were likely not impacted. The issue affected up to 20 percent of Google Meet users globally.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 26 September, 2025 08:40

Incident End: 26 September, 2025 09:47

Duration: 1 hours, 7 minutes

Affected Services and Features:

Google Meet

Regions/Zones: Global

Description:

Google Meet users experienced elevated error rates globally for a duration of 1 hour, 7 minutes. The preliminary root cause was a recent rollout to a backend API server, which was mitigated after the rollback completed.

Google will complete a full Incident Report in the following days that will provide a full root cause.

Customer Impact:

Affected users experienced 504 errors when attempting to access Google Meet.

Summary:

Some users experienced errors when trying to access Google Meet.

Description:

The issue with Google Meet has been resolved for all affected users as of Friday, 2025-09-26 09:47 PDT.

A root cause analysis is in progress. Google will be publishing a preliminary Incident Report shortly and a full Incident Report in the coming days.

We thank you for your patience while we worked on resolving the issue.

Customer Symptoms:

Impacted users experienced errors when attempting to access Google Meet.

Summary:

Google Meet may be unavailable for users globally.

Description:

We are experiencing an issue with Google Meet beginning on Friday, 2025-09-26 08:49 PDT.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2025-09-26 11:00 PDT with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms:

Users are unable to join calls on Google Meet.

Workaround:

None at this time.