Incident Report

Incident Start: 26 September 2025 02:02

Incident End: 26 September 2025 08:02

Duration: 6 hours

Summary

Starting on Friday, 26 September 2025 02:02 US/Pacific, a power equipment failure in zone us-central1-a caused service disruptions. Google Workspace Admin Console experienced elevated latency and error rates. Service was fully restored by 08:02 US/Pacific.

Root Cause

The incident was triggered by a power equipment failure at one of the data centers in us-central1-a at 02:02 US/Pacific. This failure resulted in a power disruption to a subset of storage servers within the zone, leading to the observed impact across the listed services.

Detailed Description of Impact

Admin Console (Google Workspace)

A majority of Admin-exposed services, such as the Security Investigation Tool (SIT), Email Log Search (ELS), and Admin Quarantine, experienced issues where reporting pages appeared blank, charts failed to load, or a buffering circle was continuously displayed. Some services also returned ‘502’ (Bad Gateway) errors. The services were fully mitigated by 08:02 US/Pacific. The delayed recovery was caused by a dependency on Google BigQuery, which was also impacted by the incident.

Remediation and Prevention

Google engineers were alerted to the incident from an internal monitoring alert at 02:13 US/Pacific on 26 September 2025 and addressed the incident by transferring the affected servers to alternate power equipment, which was completed by 02:35 US/Pacific.

A majority of the impacted services recovered quickly upon the resumption of power. However, several products had extended time to recover due to their dependency on the impacted storage system.

To protect the data during a power event, a safety feature was auto activated. This feature temporarily placed affected storage servers into a protective, read-only mode to prevent any data from being written that might not be saved correctly in case of another power loss.

While this safety feature worked as intended to ensure data durability, it had an unintended side effect. Our underlying file system, which orchestrates data storage, needs to perform very small, internal "write" operations to coordinate its tasks, even when you are only reading data. Because a large number of storage servers were in the read-only state, the file system was unable to perform these essential internal writes. This caused parts of the file system to become temporarily unavailable, which in turn impacted services like Persistent Disk that rely on it.

We sincerely apologize to our customers whose services were impacted by this disruption. This is not the level of quality and reliability we strive to offer, and we are taking steps to ensure this does not happen again.

• Speeding storage system recovery: We are reverting the recent change, to the previous behavior. We will improve testing of this feature to ensure it does not introduce availability issues after power loss events before reintroducing it at a later date.
• Enhancing our monitoring: We are building a new dashboard to give us a real-time, comprehensive view of the power and battery status of all our storage servers. This will allow us to detect and respond to potential power-related issues much more quickly.
• Strengthening data-protection protocols: We are creating clearer guidelines for our engineers on when it is appropriate to override automated data-protection features. This will help ensure that we always prioritize the safety and durability of your data.
• Rigorously testing our safety features: We are implementing more thorough fault testing to confirm that our data-protection mechanisms work exactly as expected, especially after they have been activated and then reset.
• Improving dependent system recovery: We are implementing changes in BigQuery to improve the time to recovery of the BigQuery processing for recently ingested data after a partial or full cluster level storage system outage.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 26 September 2025 02:02

Incident End: 26 September 2025 08:00

Duration: 5 hours, 58 minutes

Affected Services and Features: Admin Console

Description: Starting at 02:02 on Friday, 26 September 2025, multiple Google services in a specific zone began experiencing elevated latency and error rates. A power equipment failure at one of the datacenters in the zone resulted in power disruption for a subset of storage servers. To restore services, the underlying power issue was immediately addressed by transferring the affected servers to an alternate power equipment by 02:35.

A majority of the impacted services were recovered quickly upon resumption of power. Due to manual intervention required to restore a small subset of the storage servers, a few services continued to experience partial unavailability and were fully recovered by 08:00.

Google will complete a full Incident Report in the following days that will provide a full root cause.

Customer Impact: Workspace Admin Console users may have observed ‘502’ errors (Bad Gateway) or continuous loading.

Description :

The issue with Admin Console has been resolved for all affected users as of Friday, 2025-09-26 07:56 PT.

The issue was triggered by a backend storage service issue. We are investigating the full root cause, and we will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we worked on resolving the issue.

Summary

Workspace Admin Console users might observe 502 errors (Bad Gateway) or continuous loading.

Description

We are experiencing an issue with Admin Console (Security Investigation Tool, Email Log Search, Admin Quarantine) and AdminSDK: Reports API beginning on Friday, 2025-09-26 02:13 PDT.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2025-09-26 09:00 PDT with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms

Workspace Admin Console users might observe 502 errors (Bad Gateway) or continuous loading.

Workaround

None at this time.

Summary

Workspace Admin Console users might observe 502 errors (Bad Gateway) or continuous loading.

Description

We are experiencing an issue with Admin Console (Security Investigation Tool, Email Log Search, Admin Quarantine) and AdminSDK: Reports API beginning on Friday, 2025-09-26 02:13 PDT.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2025-09-26 08:00 PDT with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms

Workspace Admin Console users might observe 502 errors (Bad Gateway) or continuous loading.

Workaround

None at this time.

Summary

Workspace Admin Console users might observe 502 errors (Bad Gateway) or continuous loading.

Description

We are experiencing an issue with Admin Console (Security Investigation Tool, Email Log Search, Admin Quarantine) and AdminSDK: Reports API beginning on Friday, 2025-09-26 02:13 PDT.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2025-09-26 07:00 PDT with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms

Workspace Admin Console users might observe 502 errors (Bad Gateway) or continuous loading.

Workaround

None at this time.

Summary

Workspace Admin Console users might observe 502 errors (Bad Gateway) or continuous loading.

Description

We are experiencing an issue with Admin Console (Security Investigation Tool, Email Log Search, Admin Quarantine) and AdminSDK: Reports API beginning on Friday, 2025-09-26 02:13 PDT.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2025-09-26 06:30 PDT with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms

Workspace Admin Console users might observe 502 errors (Bad Gateway) or continuous loading.

Workaround

None at this time.

Summary

Workspace Admin Console users might observe 502 errors (Bad Gateway) or continuous loading.

Description

We are experiencing an issue with Admin Console (Security Investigation Tool, Email Log Search, Admin Quarantine) and AdminSDK: Reports API beginning on Friday, 2025-09-26 02:13 PDT.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2025-09-26 06:00 PDT with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms

Workspace Admin Console users might observe 502 errors (Bad Gateway) or continuous loading.

Workaround

None at this time.

Summary

Workspace Admin Console users might observe 502 errors (Bad Gateway) or continuous loading.

Description

We are experiencing an issue with Admin Console (Security Investigation Tool, Email Log Search, Admin Quarantine) and AdminSDK: Reports API beginning on Friday, 2025-09-26 02:13 PDT.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2025-09-26 05:30 PDT with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms

Workspace Admin Console users might observe 502 errors (Bad Gateway) or continuous loading.

Workaround

None at this time.