Technology News • • 5-8 minutes

Prompt Leaks: Two Million Canaries Expose the AI Supply Chain

Diego Cortés
Diego Cortés
Full Stack Developer & SEO Specialist
Share:
Prompt Leaks: Two Million Canaries Expose the AI Supply Chain
Image generated with AI

Your prompt does not travel straight from your app to the model. It goes through a router, a provider and, sometimes, a reseller nobody picked. Two million canaries prove that somewhere along the way, someone reads it.

The Experiment: Two Million Prompts With Bait

On November 19, HD Moore — creator of Metasploit and now chief research officer at Dragos — published "Three breaches, endless leaks: Your AI supply chain needs canaries". Over five months, from July to November 2026, his team sent nearly two million prompts to commercial AI services. Each carried a unique secret: an AWS key tied to a canary account with full logging, an unrepeatable DNS name, a URL no legitimate system would visit, an SSH password, a reference to a private git repository. One secret per prompt and per provider. If it showed up somewhere else, that place had seen the prompt.

What a Canary Is and Why It Works

A canary is bait that exists only to be touched. It protects nothing — it exposes. If someone opens that URL or uses that key, you know the data passed through their hands, without relying on the provider's word. The team started with Thinkst Canarytokens and ended up building its own service to generate millions of distinct secrets.

Finding 1: Three Hosting Providers Compromised

The bait prompts were processed alongside real customer requests, so those were almost certainly copied too. The earliest incident dates to mid-July, and none of the three knew it had been breached until Dragos told them.

The Big Breach Was Copied at the Front Door

The largest hit a couple hundred AWS keys, all sent in short windows in late October. Every request in those windows was stolen. The detail that matters: some came from requests the provider rejected before a model ever ran. If they were stolen anyway, the attacker was copying at the entrance, not deep in the stack. In the other two breaches, part of what was taken had been marked Zero Data Retention.

What the Attacker Did With the Keys

First they tested each key, in some cases within hours. Then they listed S3 buckets and checked Secrets Manager and Parameter Store. Finally they tried to enable Bedrock models, create Bedrock API keys and add their own IAM users. Tens of thousands of AWS API calls made with keys that existed only to be detected.

Finding 2: Reading Prompts Without Exploiting Anything

Several resellers on an inference marketplace could view and retain customer prompts using the platform's standard logging. No vulnerability, no control bypass, nothing that could be patched. The mechanism fits in a sentence: list a popular model at the lowest price, take the traffic, pay the difference and collect the data from the logs. A suspiciously low price is a signal about the path, not a bargain.

Finding 3: The Privacy Flag Does Not Travel

Requests sent to US-based providers ended up forwarded to Chinese AI organizations, unique URLs were visited months after a single request, and one seeded link surfaced in an internal company chat in China. The clearest case: a Chinese lab fetched the secret URLs from more than half of the prompts sent to its models between July and October, including close to two thirds of those marked as no-data-collection. The crawler usually showed up two or three days later.

Zero Data Retention Only Binds the First Hop

ZDR is a contractual promise, not a technical control: it says the prompt and the response are not stored, and the router should forward those requests only to providers with the same setting. The report's line works as a summary: a privacy flag only binds the first hop. Every hop sees the full prompt and the full response.

OpenRouter's November 22 Response

The router sitting in the middle of much of that traffic scopes the finding first: the reports did not show any breach or unauthorized access of OpenRouter. What they did show is everything above, and the company acted on its providers.

Three Providers Out and 125 Endpoints Stripped of ZDR

OpenRouter off-boarded three providers, removed 125 endpoints from Zero Data Retention eligibility — those flagged by researchers plus others pulled as a precaution — and notified potentially affected customers. Three tasks remain underway: turning the testing method into a continuous check, verifying network paths to catch forwarding to third parties, and requiring assurances before restoring ZDR eligibility.

What Novita, Phala and Together AI Each Say

OpenRouter links three provider notices, and they should not be confused with the three compromised or the three removed: they are separate events and the company does not cross the lists. Novita acknowledges that requests to MiniMax models were forwarded to MiniMax's own API between May 28 and November 11. Phala warns that request content for one model must be treated as exposed in an October–November window. Together AI reports that an unauthorized third party downloaded stored prompt samples from June and July.

What This Means for Your Application

None of this is a CVE you close by bumping a library version. It is a problem of routing, contracts and verification, and you fix it in that order.

Pick the Upstream Provider Yourself

If you use a router, pin the destination provider instead of accepting the default pool, turn off automatic fallback to reseller marketplaces and check that the privacy setting actually changes where the request lands — not just a label in your dashboard. To control who reaches your site, the usual robots.txt, nginx and agent-signature rules are still the first filter.

Seed Canaries and Rotate Anything That Touched a Prompt

A free Thinkst Canarytoken is enough to start: drop a decoy URL into the context you send to production and alert yourself if anyone opens it. Then apply the uncomfortable rule: any credential that travelled inside a prompt must be treated as exposed and rotated. Secret filtering helps, but a lot of sensitive data — source code, internal names, plans — has no API-key shape at all.

Self-Hosting: When It Pays Off and What It Costs

Self-hosting is the only setup where nobody else reads the prompt. The report's orders of magnitude are enough to decide with numbers: a 512 GB Mac Studio from $22,299, a DGX Spark at $6,950, a DGX Station around $105,000 and an eight-GPU B300 server from $570,000. For the most sensitive work it makes sense; as a general answer, it does not. If you go that way, llama.cpp 1.0 with a stable API and Ollama for running models on your own machine are two reasonable starting points.

Conclusion: Privacy Is a Routing Decision

The lesson is not "all prompts leak" but something less comfortable: some leak, and from the outside you cannot tell which. Your app's privacy is not defined by the provider you picked in a dashboard, but by the whole chain the data travelled through. If this is your kind of problem, we keep taking AI infrastructure apart on this blog: the San Francisco declaration on the AI supply chain is the natural next chapter.

Categories