Rate Limiting Meta's Crawlers (meta-externalads) With Nginx

Last weekend the load on one of the servers I manage went through the roof. The culprit turned out to be meta-externalads/1.1 - Meta’s ads crawler - hammering a large multilingual WordPress site at around 100 requests per minute, mostly against rarely-visited pages that weren’t in the page cache. Each request was a full PHP and MySQL render, and the server was not happy about it.

The obvious fix is to block the user agent, but there’s a catch: the site runs Meta ads. This crawler is the thing that verifies ad landing pages, so blocking it outright risks breaking the ad campaigns. What you actually want is to let it crawl, just slowly.

Here’s how to do that with plain nginx - no Cloudflare, no paid WAF, and no per-IP rules that a distributed crawler walks straight through.

Why Meta is suddenly hammering everyone

This isn’t an isolated incident. Meta is aggressively scraping the web at the moment, and the reason appears to be that it is building its own AI search engine to power Meta AI, reducing its dependence on Google and Bing - something Meta’s own developers have allegedly confirmed. Its meta-external* crawler family is doing the collecting: meta-externalagent gathers content for AI training and the search index, and even the ads crawler seems to have joined the land grab, crawling far beyond the handful of landing pages it actually needs to verify.

Unlike Googlebot, this crawling sends nothing back your way - no search traffic, no referrals. So there’s no reason to let it consume your server resources at whatever rate it fancies.

Why per-IP rate limiting doesn’t work here

The standard nginx rate-limiting example keys on the client IP:

limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/m;

That’s fine for a single abusive client. But when I counted the IPs behind the crawler traffic, there were 458 distinct addresses in one day’s log, all from Meta’s 2a03:2880::/32 IPv6 space. With per-IP limiting, each address gets its own allowance - 458 IPs at 10 requests/minute each is more traffic than the original spike. IPv6 makes this worse: every /128 counts as a separate client, and large operators have effectively unlimited addresses.

The trick is to key the rate limit on something all those IPs share: the user agent.

The config

1. Define the zone (http context)

This goes at http level. On most distros, anything in /etc/nginx/conf.d/*.conf is included there automatically, so create a file like /etc/nginx/conf.d/meta-ratelimit.conf:

map $http_user_agent $limit_meta {
    default            "";
    "~*meta-external"  "metabot";
}

limit_req_zone $limit_meta zone=metabot:1m rate=30r/m;

Two things are doing the work here:

The map gives every request from a meta-external* user agent the same constant key ("metabot"), so the whole crawler shares a single 30-requests-per-minute bucket no matter how many IPs it arrives from. Bring 458 IPs or 4,580 - same bucket.
Everyone else gets an empty string as their key, and nginx skips rate limiting entirely for empty keys. Normal visitors are untouched, at zero cost.

The ~*meta-external pattern catches the whole family - meta-externalads (ads crawler), meta-externalagent (the AI training and search-index crawler) and meta-externalfetcher - while leaving facebookexternalhit (the link-preview fetcher) alone, since that one only fires when someone actually shares a link.

2. Apply the limit (server context)

In the server { } block of the site you want to protect:

limit_req zone=metabot burst=20;
limit_req_status 429;

burst=20 queues short spikes and drips them through at the zone rate instead of instantly rejecting them, which is friendlier to a legitimate crawler. limit_req_status 429 matters more than it looks: nginx defaults to 503, but 429 Too Many Requests is the signal Meta’s crawlers are documented to back off from. Send the right status and the crawler genuinely slows down over the following days, rather than just retrying.

3. Watch out for the inheritance quirk

limit_req directives don’t accumulate across levels. If a server block defines any limit_req of its own (common if you use something like the Nginx Ultimate Bad Bot Blocker, which injects its own limits), it completely replaces anything inherited from http level rather than adding to it. So if your vhost already has rate limiting from another include, the limit_req zone=metabot ... line must sit in the same server context alongside it - defining it only at http level will silently do nothing for that site.

4. Restart, don’t reload

This one cost me an hour. After changing the config I reloaded nginx, tailed the log… and nothing changed. Changed it again - still nothing.

The reason: nginx -s reload is graceful. Old worker processes keep serving every connection they already hold until those connections close - and Meta’s crawler rides persistent HTTP/2 connections that can stay open for a very long time. So the crawler was pinned to old workers still enforcing the old config, indefinitely. Every change I made was correct; none of it had taken effect for the traffic I was watching.

nginx -t && systemctl restart nginx

A full restart severs the held connections and forces the crawler to reconnect to workers running the new config. Worth remembering any time you change user-agent maps or rate limits and a long-lived bot “ignores” them.

Verifying it works

Tail the access log filtered to the crawler:

tail -f /var/log/nginx/access.log | grep meta-external

What you want to see is 200s trickling through at roughly the zone rate (30r/m is about one every two seconds) with 429s absorbing everything above it:

2a03:2880:16ff:16:: ... "GET /page-one" 200 ...
2a03:2880:10ff:18:: ... "GET /page-two" 200 ...
2a03:2880:18ff:b::  ... "GET /page-three" 429 178 ...
2a03:2880:16ff:29:: ... "GET /page-four" 200 ...

Different IPs, one shared rhythm - that’s the constant-key bucket doing its job.

For a before/after picture, count requests per hour:

grep "meta-externalads" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1-2 | tr -d '[' | sort | uniq -c

In my case that went from nearly 6,000 requests in the peak hour to double digits.

Tuning

rate=30r/m is a deliberate starting point, not a magic number. The ads crawler only needs enough throughput to verify landing pages - it does not need to traverse your entire site at speed. Keep an eye on Meta Ads Manager for a couple of days after deploying; if nothing complains about landing-page crawl issues, the rate is generous enough (and you could probably halve it). If something does flag, bump the rate up - it’s one number in one file.

A robots.txt entry is worth adding as belt and braces, though compliance reports are mixed - the nginx limit is the part that’s actually enforced:

User-agent: meta-externalads
Crawl-delay: 10

The same pattern for any distributed crawler

Nothing here is Meta-specific. The recipe - map a user agent to a constant key, rate-limit on that key, return 429 - works for any well-behaved-but-greedy crawler that arrives from many IPs: AI training bots, SEO tools, aggressive feed fetchers. Add another line to the map with its own key and zone, and each crawler gets its own dial:

map $http_user_agent $limit_meta {
    default            "";
    "~*meta-external"  "metabot";
}

map $http_user_agent $limit_gpt {
    default            "";
    "~*GPTBot"         "gptbot";
}

limit_req_zone $limit_meta zone=metabot:1m rate=30r/m;
limit_req_zone $limit_gpt  zone=gptbot:1m  rate=10r/m;

For outright junk with no legitimate purpose, skip the ceremony and just block it - that’s what tools like the Bad Bot Blocker are for, and I’ve written before about blocking fake Googlebot traffic the same way. The rate-limit treatment is for crawlers that have a real job to do on your site - you just get to decide how fast they do it.