How to block (or allow) AI agents on your site: robots.txt, nginx and signed agents
An AI agent does not ask for permission: either you write the rules down or your site decides by default. These are the four layers available today to block AI agents without breaking the traffic you actually want, with code and with what each layer cannot do.
The United booking case put the issue on front pages, but the underlying decision is the same for a blog, a store or an API: who may act on your site, and with what proof. Most sites still have nothing written about it.
What counts as an agent on your server
From the server side, three different things arrive looking much alike, and the technical answer changes depending on which one it is.
The first is the training crawler: it reads your content to feed a model. It does not care about your business or your session; it just wants the text. The second is the AI search bot, which wants to cite your page inside an answer: block it and you disappear from that surface. The third is the agent that acts: it enters with a session, fills forms, changes a booking, pays. That one is not a reader, it is an automated user.
Read also
The working rule is simple: the reader gets a policy, the searcher gets identified, and the actor gets an API. Mixing all three into a single block is the most common mistake, and it is the one that ends up breaking search traffic.
The user agents that matter in 2026
These are the ones worth knowing by name, company and job:
- GPTBot (OpenAI): training.
- OAI-SearchBot (OpenAI): indexing for search answers.
- ChatGPT-User (OpenAI): a visit triggered by a person from the chat.
- ClaudeBot and anthropic-ai (Anthropic): training.
- Claude-SearchBot (Anthropic): search.
- Google-Extended (Google): use of content for Gemini training and grounding.
- PerplexityBot (Perplexity): indexing for its search engine.
- CCBot (Common Crawl): an open corpus many models use later.
- Bytespider (ByteDance): heavy crawling, historically loose with limits.
One detail that gets misread constantly: blocking Google-Extended does not remove your site from Google or change Googlebot indexing. It only affects the use of your content for training and for grounding Gemini answers. They are two separate permissions, managed separately.
Layer 1: robots.txt, the rule nobody has to obey
robots.txt is the RFC 9309 standard and it works by voluntary agreement. A bot can ignore it with no technical consequence: there is no authentication and no penalty. It exists to declare intent and for serious providers to respect it, not to prevent access.
It is still the first layer because it is the only one everyone can see, SEO auditors included. And it allows something valuable: separating training from search. Blocking GPTBot while allowing OAI-SearchBot is a perfectly valid and fairly common choice.
# Training out, AI search in
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ByteSpider
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
Each User-agent line opens a group with its own rules, so do not mix directives for different bots inside the same group. The file lives at the domain root, is served as text/plain and must return 200: a 404 or a redirect leaves it with no effect.
How to check your rules actually do something
First, that the file is served the way you expect:
curl -sI https://yoursite.com/robots.txt
curl -s https://yoursite.com/robots.txt | head -20
Second, that a cache, a CDN or a framework rewrite is not hiding it. If the 200 comes back with your homepage HTML, you have a routing problem, not a content one. And third, the real test: read the logs. If the bot keeps coming back at the same rate after the rule, the rule is decorative.
Layer 2: nginx and the user agent that lies
Here you actually cut traffic, with one warning: the user agent is a comment, not a credential. Anyone can write "GPTBot" in their client, and any bot can write Chrome's user agent. It filters what is declared and it stops naive scrapers; it does not tell actors apart.
The block goes in the http context, not inside server. That is the classic mistake: map is declared once in http {}.
http {
map $http_user_agent $agent_block {
default 0;
"~*GPTBot" 1;
"~*ClaudeBot" 1;
"~*CCBot" 1;
"~*Bytespider" 1;
}
limit_req_zone $binary_remote_addr zone=agents:10m rate=10r/s;
}
And in the site's server block, the consequence:
server {
server_name yoursite.com;
if ($agent_block) {
return 403;
}
location / {
limit_req zone=agents burst=20 nodelay;
try_files $uri $uri/ /index.php?$query_string;
}
}
The block, line by line
map compares the user agent against patterns and returns a value: ~* means a case-insensitive regular expression match. The variable $agent_block ends up as 1 for the listed bots and 0 for everything else.
if ($agent_block) { return 403; } is an acceptable exception to the well-known rule against if in nginx: return is one of the directives that behaves well inside a conditional block. What you should not put there is proxy_pass or rewrite.
limit_req_zone defines a shared zone keyed by IP with a limit of ten requests per second; limit_req applies it with a burst of twenty and no queue. That is the part that actually protects the server when someone decides to ignore the 403, which is why it is worth having even if you block nobody.
The cost of over-blocking is real: a pattern that is too broad takes out legitimate browsers, monitoring tools or the AI search traffic that was sending you visits. So is the cost of under-blocking: bandwidth, database load and the bill.
Layer 3: Laravel 13 middleware and rate limiting
The application is the only place where you know whether there is a session behind the request. That makes it the best wall: an agent acting on a user's behalf needs that session, and that is where you can require a login, two-factor authentication or specific permissions.
The middleware cuts on the declared user agent. Same logic as nginx, but with context: you can log the attempt, return a different JSON response or let public routes through.
<?php
namespace App\Http\Middleware;
use Closure;
use Illuminate\Http\Request;
use Symfony\Component\HttpFoundation\Response;
class BlockDeclaredAgents
{
private const BLOCKED = ['GPTBot', 'ClaudeBot', 'CCBot', 'Bytespider'];
public function handle(Request $request, Closure $next): Response
{
$agent = $request->userAgent() ?? '';
foreach (self::BLOCKED as $needle) {
if (stripos($agent, $needle) !== false) {
abort(403, 'Automated access not allowed');
}
}
return $next($request);
}
}
Register it in bootstrap/app.php, inside withMiddleware(), using $middleware->append(...) if you want it on every request or $middleware->alias([...]) to apply it only where it belongs.
And for volume, the framework's limiter:
use Illuminate\Cache\RateLimiting\Limit;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\RateLimiter;
public function boot(): void
{
RateLimiter::for('agents', function (Request $request) {
return Limit::perMinute(30)
->by($request->ip())
->response(fn () => response('Too many requests', 429));
});
}
With that in place, the route is protected with Route::middleware('throttle:agents'). If you already limit per user, this is the IP complement, not the replacement. And one thing not to do: blocking whole datacenter IP ranges. You break legitimate services, you break yourself behind a VPN, and the bot is out through another provider in ten minutes.
Code: middleware and RateLimiter working together
The recommended order is your own middleware first, the limiter after. That way a declared bot gets a cheap 403 without touching the database, and excessive anonymous traffic falls into the limiter's 429. If you want to go deeper on the limiter, the blog already has a guide to rate limiting in Laravel 13 with more per-route and per-user examples.
Layer 4: signed agents (Web Bot Auth)
None of the previous three layers answers the important question: is this agent who it says it is? Web Bot Auth tries with cryptographic signatures over HTTP Message Signatures, RFC 9421, and an Ed25519 key pair.
The agent signs every request and sends the Signature-Input and Signature headers; optionally, Signature-Agent tells the site where its public key is published so it can be verified. The draft comes from Cloudflare (draft-cloudflare-web-bot-auth) and the company already treats signed agents as verified inside its network. On your own server, what you can do today is verify the signature and flag the request as identified, without depending on a third party: in PHP, the sodium extension provides sodium_crypto_sign_verify_detached(), which is exactly the operation you need once you have the agent's public key.
This blocks nobody. It changes the conversation: you move from "what it claims to be" to "what it can prove". A signed agent with a verified key deserves a different limit, and one that calls itself GPTBot without a signature deserves the same treatment as a stranger.
Measure before you block
Two commands over the access log tell you more than any list:
awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20
grep -c "GPTBot" access.log
The first groups requests by user agent (in combined format, the agent is the field after the quoted referer). The second counts how many times a given bot showed up. With that you answer three questions before touching anything: how much traffic each agent brings, which routes it touches, and whether that traffic converts or just consumes.
The decision changes with the type of content. On ad-supported pages, a training crawler pays you nothing and still costs bandwidth. On a catalog, an AI search bot may be your best discovery channel. In a private area, the answer is not a server rule: it is a login. And on an API, the answer is a key, not a user agent.
Conclusion
robots.txt is the policy, the server is the door, the application is the doorman and the signature is the ID card. Each layer solves one part and none solves everything, which is why they add up instead of replacing each other. If you take one thing away, take this: an agent is no longer an experiment, it is a client with rules, and the rules get written before it arrives.
We keep going in this direction on the blog: how tools connect to a model with MCP, how a runaway agent gets switched off with the CrowdStrike kill switch, and what server traffic looks like with HTTP/3 on nginx.
