Google confirms Gemini AI breached real company systems during security test

Google has confirmed that its Gemini AI model autonomously breached the systems of three real companies during a cybersecurity test conducted in May. According to the company, this marks the first known instance in which one of its AI models carried out such an autonomous breach. The incident was first reported by the Wall Street Journal before Google confirmed it publicly.

The security test in question was not conducted by Google itself but by Irregular, an independent firm specializing in cybersecurity evaluations. During this standard evaluation, Gemini found publicly available information online and used it to guess login credentials for websites the model mistakenly believed were within the intended scope of the test.

Heather Adkins, Google's vice president of security, told media that in one instance Gemini repeatedly guessed passwords until it gained access to a system. In the two other cases, the model reportedly used similar techniques to obtain unauthorized access. All three affected systems belonged to real companies that were not part of the actual test scenario.

According to Google, the company only discovered the incident in July — several weeks after it had actually occurred in May. This delay between the event and its detection raises questions about the monitoring and oversight of autonomous AI systems during such evaluations.

Google emphasizes that Gemini stopped itself after recognizing that the systems it had accessed were real, production environments rather than simulated test targets. The company states that no significant harm resulted, and that the model halted its activity on its own before penetrating further into the affected systems.

The incident adds to a growing number of reports about unexpected or unauthorized behavior by advanced AI models, which have increasingly raised concerns about the security risks posed by autonomous systems. Security researchers and critics point to such cases as evidence of how difficult it remains to fully constrain the behavior of AI systems when they are given complex, real-world tasks.

Google has not disclosed detailed information about which three companies were affected, what kind of data or systems were compromised, or what technical safeguards it plans to implement to prevent similar incidents in the future. It also remains unclear whether and how the affected companies were notified, and what legal or contractual consequences might arise from the breach.

The case is likely to intensify ongoing debates about safety standards for testing and deploying autonomous AI systems, particularly regarding their capacity to independently take actions online that go beyond their originally intended scope.