New Trick Exposes Hidden Thoughts of AI and Revives Distillation War

A computer trick allows the 'thinking' of cutting-edge AI models to be laid bare, revealing possible covert distillations and data leaks. Researchers demonstrate that smaller versions of the models can expose their hidden reasoning.

• Researchers discover vulnerability in APIs of OpenAI, Anthropic, and Google that exposes the internal reasoning of AI models.
• The method allows distillation of information from closed models and recovery of passwords and API keys, although it has already been patched.
• Distillation becomes a geopolitical issue as China and the U.S. compete for supremacy in AI.

A new computer trick allows the internal reasoning of the most advanced artificial intelligence models to be exposed, revealing secrets that tech companies try to protect. According to a Wired article, recent research has uncovered a vulnerability in the APIs of OpenAI, Anthropic, and Google that not only facilitates large-scale distillation but can also leak passwords and API keys.

The method, developed by scientists from the University of Tübingen, the Max Planck Institute, MATS Research, and the company Snyk, allows the extraction of the hidden "thoughts" of cutting-edge models. Although the vulnerability has already been patched, the finding provides evidence—albeit not conclusive—that some Chinese models may have been trained by distilling reasoning information from U.S. models.

The Trick that Exposes AI Reasoning

Advanced AI models solve difficult problems by breaking them down into parts that they analyze in an artificial "chain of thought." Companies keep that reasoning secret but send an encrypted version to the user's computer to transfer part of the computation.

The researchers' attack is based on the fact that most companies offer models of different sizes. Smaller models are weaker but also less aligned, meaning they are more willing to reveal their internal thoughts.

Feeding the encrypted reasoning traces to a smaller version of the same model can reveal the hidden reasoning, explains Alexander Panfilov, a computer scientist at the University of Tübingen. "All the major frontier model providers we tested share this vulnerability," he warns.

Panfilov and his colleagues tested the models of OpenAI, Anthropic, and Google through their APIs and found the same issue. They also demonstrated that the method could extract secret information, such as API keys and passwords embedded in the reasoning traces.

"The idea of exchanging messages to a weaker model variant that has the same decryption key but weaker alignment is brilliant," comments Florian Tramer, a cybersecurity scientist at ETH Zurich. "It is definitely becoming a problem," he adds.

The Controversy of Distillation

Distillation is a well-established technique for efficiently copying the capabilities of existing models to new ones. It is especially common in the development of open-weight or fully downloadable models.

Recently, distillation has become a controversial topic due to claims that Chinese AI companies use it to copy the best U.S. models. In February, OpenAI told U.S. lawmakers that DeepSeek appeared to have copied one of its models to build the R1.

In June, Anthropic told lawmakers that Alibaba had systematically distilled its models to build its own, called Qwen. There is no indication that they have used this specific technique, but researchers believe that their method would allow for more information to be distilled from closed models than previously thought.

Mark Zuckerberg, CEO of Meta, stated that distillation is an important principle of how the open-source ecosystem functions. He warned that restricting this practice would put the United States at a disadvantage.

On the other hand, Kyle Miller, a researcher at the Center for Security and Emerging Technologies, says it is unclear how much distillation actually helps China. "If the ability of Chinese labs to distill were eliminated, the competitive landscape would not change drastically," he claims.

Researchers fed each model 90 questions to check if the open-weight models were distilling from the closed ones. By providing the first words of the captured reasoning traces, they sometimes found that the open models generated remarkably similar responses.

Evidence of Similarity and Reactions

The Chinese open-weight model Kimi K3, from Moonshot AI, produced an output surprisingly similar to the hidden reasoning traces of Claude Opus 4.8 and GPT 5.6 Sol for certain prompts. Despite the similarities, researchers point out that the work cannot causally establish distillation.

Two other open-weight models, DeepSeek from China and Inkling from the American company Thinking Machines, did not show this similarity. Moonshot AI and Z.ai did not respond to requests for comments.

There had been prior speculation on Chinese social media about the possibility of discovering and using hidden reasoning traces for distillation. Yarin Gal, a computer scientist at the University of Oxford, says that distillation is not only widespread but has also helped AI advance more rapidly.

Companies have already taken action: Panfilov and his co-authors alerted OpenAI, Anthropic, and Google, and each has adjusted its API to mitigate the issue. Michael Aciman, a spokesperson for Anthropic, stated that they value independent research and have begun developing short-term mitigations.

Google and OpenAI declined to comment. Panfilov adds that fully correcting distillation would require a fundamental review of how APIs function.

Geopolitical and Security Implications

Distillation has become a matter of geopolitical importance as the United States and China compete for supremacy in AI. Hardline hawks on China claim that the country gains a strategic advantage by distilling U.S. technology.

Others argue that distillation is a widely used way to rapidly boost the capabilities of an AI model. Zuckerberg defends this practice from an open-source perspective.

Kyle Miller points out that it only improves the capabilities of existing models to a limited degree and that Chinese companies seem to have the expertise necessary to build cutting-edge models from scratch. "No one here in the U.S. knows how much distillation is benefiting Chinese labs," he says.

Researchers also highlight that the method could be used to recover personal information, although that vulnerability has already been addressed. However, some reasoning traces may still be discoverable using the same method.

Panfilov and his collaborators claim that using their method would allow for the distillation of more information from closed models than previously thought. This could have significant implications for intellectual property in the AI industry.

This content is provided for general informational purposes only and doesn't constitute financial, investment, legal, or tax advice. Any events, rewards, online promotions, or related information mentioned herein should not be considered a recommendation, solicitation, or invitation to purchase, sell, trade, or otherwise deal in any crypto assets. Crypto assets are highly volatile and may result in loss. The availability of WEEX services, products, and related events may vary by region. You are responsible for ensuring that your participation is in accordance with applicable local laws and regulations.