Incident Report

Summary

On 29 October 2024 some Gmail web users experienced slow loading, unresponsive or crashed web pages for a duration of 22 hours, 46 minutes

To our customers whose businesses were impacted during this outage, we sincerely apologize. This is not the level of quality and reliability we strive to offer you.

Root Cause

Browser extensions, such as third-party ones, have significant freedom to run their own JavaScript code on web pages they're active on. This means they can modify the page's appearance and content, interact with remote servers, and potentially even disrupt the page's functionality.

While extensions operate in their own isolated environment, they can technically access information from the main web page. However, the reverse isn't true, limiting the website's control over the extension's actions. Essentially, extensions have a wide range of capabilities with minimal oversight, potentially leading to unintended consequences or conflicts with the web page.

A recent update to a third-party browser extension resulted in some Gmail web users who have these extensions installed experiencing performance problems, such as increased latency or unresponsiveness.

Remediation and Prevention

Google engineers were alerted to the issue by internal monitoring on 30 October 2024 at 07:04 US/Pacific and immediately started an investigation. The issue was fully mitigated at 11:44 US/Pacific after the third-party mitigated the issue from their end.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

• Implement testing for popular browser extensions to better detect and quickly mitigate similar issues.
• Improve internal documentation/processes to enable faster response and mitigation time for this type of issue.

Detailed Description of Impact

Gmail

Customers using the Gmail Web UI may have experienced increased latency or unresponsiveness when also using certain third-party extensions.

Additional Information for Customers:

This issue only affected the Gmail Web UI. Gmail iOS/Android apps could be used as a workaround during the issue duration.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 29 October, 2024 13:00

Incident End: 30 October, 2024 11:46

Duration: 22 hours, 46 minutes

Affected Services and Features:

Gmail

Regions/Zones: Global

Description:

Some Gmail users experienced latency, unresponsiveness, and inability to load the Web UI for a duration of 22 hours, 46 minutes. From preliminary analysis, the root cause of the issue was an incompatibility with a third-party add-on following an update to Gmail front end servers.

Customer Impact:

Customers using the Gmail Web UI may have experienced increased latency or unresponsiveness when also using certain 3rd party addons.

Additional details:

This issue only affected the Gmail Web UI. Gmail iOS/Android apps could be used as a workaround.

Engineers mitigated the issue by rolling back the recent update to Gmail front end servers.

The issue with Gmail has been resolved for all affected users as of Wednesday, 2024-10-30 11:46 US/Pacific.

We will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we worked on resolving the issue.

Gmail users may experience slowness in Gmail UI, including lag and non-responsiveness during loading and sending operations.

We are experiencing an issue with Gmail beginning at Wednesday, 2024-10-30 05:30 US/Pacific.

Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2024-10-30 12:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.