How does a distillation attack work?
Distillation is a standard training method, described in a 2015 paper by Hinton, Vinyals and Dean, in which a smaller model learns from the outputs of a larger one. In an attack, the teacher is a model the attacker does not own. The attacker sends it a huge number of prompts, saves the answers, and trains their own model on them.
Anthropic’s report on distillation attacks says distillation itself is legitimate and that the abuse lies in using it without permission. It describes campaigns that ran through fraudulent accounts and proxy services that resell access.
Why does a distillation attack matter for a small business?
A business can be hit from two sides. If your API key is stolen, the thief’s requests are billed to you. If you sell an AI feature through your own API, a customer could be copying it.
The full explanation is in what is a distillation attack. Related terms: model extraction and API credential.
Sources
- Hinton, Vinyals and Dean: Distilling the Knowledge in a Neural Network (arXiv)(arxiv.org)
- Anthropic: Detecting and preventing distillation attacks(anthropic.com)
