What is a distillation attack? Explained for business owners

What is distillation in AI?

Distillation is a training method where a small AI model learns from a larger one. In their 2015 paper, Hinton, Vinyals and Dean describe it as compressing the knowledge of a large model, or a group of models, into a single smaller model that is cheaper to run. The student model is trained on what the larger model outputs, not just on the original data.

Distillation is ordinary, legitimate engineering. Companies use it to make fast, cheap versions of their own models. Anthropic says the same in its statement on distillation attacks: distillation is a legitimate method, and the problem is its misuse.

What makes a distillation attack different?

A distillation attack uses someone else’s model as the teacher without permission. The attacker sends the target model a very large set of prompts, records the answers, and uses them as training data for a different model. The OWASP GenAI project calls this functional replication in its entry on Unbounded Consumption: the target model generates synthetic training data, and the attacker fine-tunes another model into a functional equivalent.

The related term is model extraction, meaning an attempt to recreate part of a model by collecting enough of its outputs. Distillation is one way to do it. See the glossary entries for distillation attack and model extraction.

How do attackers get API access?

Attackers need an account that can send requests at scale, and they obtain one without paying their own way. Anthropic’s report describes three AI labs that made more than 16 million exchanges with Claude through about 24,000 fraudulent accounts, in breach of its terms and regional access limits. It also describes commercial proxy services that resell access, including one network that ran more than 20,000 fraudulent accounts at once and mixed distillation traffic with ordinary requests.

A leaked API credential gives an attacker the same capability with no account to fake. That is an inference rather than a finding from those reports, but the mechanism is plain: whoever holds a working key can send requests, and the vendor sees only the key.

How big can a distillation campaign get?

Campaigns can reach millions of requests, and attackers adapt quickly. Anthropic reports that the three campaigns it described totalled more than 16 million exchanges. It detected one of them while it was still running. After Anthropic released a new model, that campaign reportedly moved nearly half its traffic to the new model within 24 hours.

Anthropic also says attackers aim at particular abilities, such as coding, tool use and reasoning, not at the model as a whole. The practical lesson for a business is that copying traffic can look like heavy but legitimate use. Nothing in a single request gives it away. The pattern across many requests and accounts does.

Who is most exposed to distillation?

Two kinds of business are most exposed, and the risks differ.

Your situation	What an attacker wants	Main risk
You hold keys for a vendor’s AI service	A working key that costs them nothing	A large bill, and a breach of the vendor’s terms in your name
You sell AI features through your own API	Large volumes of your model’s answers	A competitor copying what you built

A team that only uses AI tools internally still holds credentials, so the first row applies. A product company can face both rows at once.

Why does the bill land on you?

The key’s owner pays for everything sent with it. API providers charge per request or per token, and the vendor cannot tell your staff’s traffic from a thief’s. OWASP names the cost side as “Denial of Wallet”: high volumes of operations that exploit the pay-per-use pricing of AI services and create an unsustainable financial burden.

How large the bill can grow depends on your limits:

Vendor	What limits the bill	Source
Anthropic	Monthly cap by tier: $500 (Start), $1,000 (Build), $200,000 (Scale), plus any lower limit you set	Rate limits
OpenAI	Optional hard monthly limit per organisation or project; alerts alone do not stop traffic	Spend limits

If you have set no limit of your own, the tier cap or your payment method is the only ceiling.

Do AI vendors ban this in their terms?

Anthropic’s terms do. Section D.4 of its Commercial Terms bars customers from using the services to build a competing product, including training competing AI models, and from reverse engineering or copying the services. The terms cover attempts as well as completed acts. A business that lets a contractor or an automation use its key for that purpose is the one in breach.

Other vendors word their terms differently, so read your own vendor’s terms before you let anyone use a shared key.

What should a business do about it?

Two groups face different risks, and each has its own guide.

• You use AI tools and hold API keys. Protect the keys and cap spend. Start with what to do in the first hour after a leak, then fix storage with how to store AI API credentials safely.
• You sell an AI feature through your own API. Watch for copying and slow it down. See how to spot and limit distillation on your own AI API.

Anthropic’s report lists the kinds of signals it used to detect campaigns: synchronised traffic across accounts, shared payment methods and coordinated timing. Small businesses rarely have that visibility, which is why per-key limits and caps carry most of the load.