Letting Verified AI Crawlers Through the Nginx Bad Bot Blocker

I noticed this in an access log on a server running the Nginx Ultimate Bad Bot Blocker:

216.73.216.108 - - [31/Aug/2026:00:01:43 +0000] "GET /blog/category/servers/ HTTP/2.0" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])"

That is ClaudeBot, Anthropic’s crawler, being dropped with a 444. Which was odd, because I had explicitly whitelisted it. The blocker ships with the AI crawlers blocked by default, so I had added override entries to blacklist-user-agents.conf for the ones I want to let in:

"~*(?:\b)OAI-SearchBot(?:\b)" 0;
"~*(?:\b)ChatGPT-User(?:\b)" 0;
"~*(?:\b)ClaudeBot(?:\b)" 0;
"~*(?:\b)Claude\-User(?:\b)" 0;

Those entries were correct, in the right file, and nginx had been reloaded. The crawler was still getting dropped. The reason turned out to be a layer I had added myself, and the fix ended up being a small companion project to my fake Googlebot blocker.

TL;DR: the whole thing is on GitHub at robwent/nginx-verified-ai-bots. It verifies ClaudeBot and GPTBot (and friends) against their operators’ published IP ranges and lets the genuine ones past the blocker, without touching the blocker’s own files.

Why the User-Agent whitelist wasn’t enough

The Bad Bot Blocker makes two independent decisions per request, and either one can return the 444:

The User-Agent check. A map on $http_user_agent sets $bad_bot. Your blacklist-user-agents.conf is included at the top of that map, and because map tests regexes in order and stops at the first match, a 0 entry there beats the blocker’s own 3 (block) entry further down. This is the layer my whitelist controls, and it was working.
The IP check. A geo block sets $validate_client from blacklist-ips.conf and whitelist-ips.conf. A UA whitelist has no effect on this at all.

The IP layer is where my AbuseIPDB integration plugs in. It pulls the AbuseIPDB blacklist every few hours and includes it in blacklist-ips.conf. And 216.73.216.108 is on AbuseIPDB with 100% confidence - presumably because ClaudeBot crawls a lot of sites and plenty of people report it. So the UA passed, the IP failed, and the request was dropped.

The obvious fix is to add the address to whitelist-ips.conf with a 0. That works, for that one address. But the crawler rotates through a pool, and every new address is a fresh 444 until you notice it in the logs and add it. I had already done this dance a couple of times.

The less obvious fix - whitelist the whole range - doesn’t work either, and this is the bit worth understanding. geo resolves by longest-prefix match: the most specific entry wins regardless of where it appears. A 216.73.216.0/22 0; in the whitelist loses to a 216.73.216.108 1; from AbuseIPDB every time, because a /32 is more specific than a /22. You cannot out-whitelist a per-IP blocklist with CIDRs.

Anthropic and OpenAI now publish their crawler IPs

Until fairly recently Anthropic’s position was that it didn’t publish IP ranges because its bots ran from shared cloud provider space. That changed in April 2026. Anthropic now publishes a JSON list of the prefixes its crawlers use, and if a request comes from one of them, it is genuinely from Anthropic:

Anthropic (ClaudeBot, Claude-User, Claude-SearchBot - one combined list): bots.json

It is easy to miss. The Claude API docs have an IP addresses page that lists a single range (160.79.104.0/21), and that is the egress range for API and Console traffic, not the crawlers. The crawler feed is linked from Anthropic’s crawler help article rather than shown inline. 216.73.216.0/22 is the first entry in it.

OpenAI has published per-bot lists for a while:

GPTBot: gptbot.json
OAI-SearchBot: searchbot.json
ChatGPT-User: chatgpt-user.json

All four files use the same shape as Google’s crawler lists - a prefixes array of ipv4Prefix / ipv6Prefix entries:

{
  "creationTime": "2026-08-18T23:56:36Z",
  "prefixes": [
    { "ipv4Prefix": "216.73.216.0/22" },
    { "ipv4Prefix": "34.162.230.222/32" },
    { "ipv4Prefix": "40.124.101.48/28" }
  ]
}

Which means the exact approach from the fake Googlebot post applies, just with the decision turned around. There, a request that claims Googlebot but isn’t from a Google IP gets dropped. Here, a request that claims ClaudeBot and is from an Anthropic IP gets waved past the blocker. Same geo + map building blocks, same updater script, and it sits alongside the blocker rather than modifying it.

There is a nice side effect. Because the pass is now conditional on the IP, the User-Agent whitelist entries can go. Genuine ClaudeBot passes on IP verification. Anything spoofing the ClaudeBot UA from a random address falls through to the blocker’s default 3 and gets dropped - which the UA whitelist was letting straight in.

The nginx approach

Three building blocks in the http{} context, one small include per server block.

A geo block per operator flags whether the client IP is in that operator’s published ranges. A map on the User-Agent decides which operator the request claims to be from. A composite map combines them: $verified_ai_bot is 1 only when the claim and the IP agree.

The enforcement is where it differs from the Googlebot version. Instead of a return, it is a set. The blocker’s blockbots.conf does if ($validate_client) { return 444; } and if ($bad_bot = 3) { return 444; }. Both variables are defined by geo / map, which nginx marks as changeable, so a server-level if that runs before the blocker’s include can simply zero them:

if ($verified_ai_bot) {
    set $validate_client 0;
    set $bad_bot 0;
}

This is the same mechanism the blocker’s own SUPER WHITELIST uses, but gated on a verified IP rather than a UA string anyone can type.

The files

There are three config files and one script. The config files reference each other by absolute path, so the locations matter. This is the layout the files ship with; if you put them anywhere else, update the include paths in verified-ai-bots.conf and INSTALL_DIR in the script to match.

File	Location	Purpose
`verified-ai-bots.conf`	`/etc/nginx/verified-ai-bots/`	Definitions - `geo` blocks and `map`s. Included once in `http{}`.
`verified-ai-bot-allow.conf`	`/etc/nginx/verified-ai-bots/`	The bypass. Included in each `server{}` block, before the blocker.
`fake-ai-bot-block.conf`	`/etc/nginx/verified-ai-bots/`	Optional. Drops spoofed AI crawler UAs on servers without the blocker.
`update-verified-ai-bot-ips.sh`	`/opt/scripts/`	Generates the IP lists and reloads nginx. Run from cron.
`anthropic-ips.conf`, `openai-ips.conf`	`/etc/nginx/verified-ai-bots/`	Generated by the script. Do not edit by hand.

Everything under /etc/nginx/verified-ai-bots/ is nginx config; the script lives outside the nginx tree so a stray include glob can never pick it up. That mirrors the /etc/nginx/fake-googlebot/ + /opt/scripts/ layout from the Googlebot project, so if you already run that, the two sit side by side.

First, verified-ai-bots.conf - the definitions, included once in http{}:

# verified-ai-bots.conf
#
# Verifies AI crawlers against their operators' published IP ranges.
# Include ONCE in the http{} context.
#
# Sets two variables:
#   $verified_ai_bot = 1  UA claims an Anthropic/OpenAI crawler AND the IP is
#                         in that operator's published ranges.
#   $fake_ai_bot     = 1  UA claims an Anthropic/OpenAI crawler but the IP is
#                         NOT in that operator's ranges.
# Everything else (human traffic, other bots) leaves both at 0.

# 1 if the client IP is in Anthropic's published crawler ranges
# (ClaudeBot, Claude-User, Claude-SearchBot share one list).
geo $is_anthropic_ip {
    default 0;
    include /etc/nginx/verified-ai-bots/anthropic-ips.conf;
}

# 1 if the client IP is in OpenAI's published crawler ranges.
# GPTBot, OAI-SearchBot and ChatGPT-User publish separate lists; they are
# merged here, so a GPTBot UA from a ChatGPT-User address still verifies.
geo $is_openai_ip {
    default 0;
    include /etc/nginx/verified-ai-bots/openai-ips.conf;
}

# Which operator (if any) the User-Agent claims to belong to. Case-insensitive.
map $http_user_agent $claimed_ai_bot {
    default              "";
    "~*claudebot"        anthropic;
    "~*claude-user"      anthropic;
    "~*claude-searchbot" anthropic;
    "~*gptbot"           openai;
    "~*oai-searchbot"    openai;
    "~*chatgpt-user"     openai;
}

# Composite decisions. Key is "claim:isAnthropicIP:isOpenAIIP".

# Claim matches the IP's operator -> verified
map "$claimed_ai_bot:$is_anthropic_ip:$is_openai_ip" $verified_ai_bot {
    default 0;
    "anthropic:1:0" 1;
    "anthropic:1:1" 1;
    "openai:0:1"    1;
    "openai:1:1"    1;
}

# Claim does not match the IP's operator -> fake
map "$claimed_ai_bot:$is_anthropic_ip:$is_openai_ip" $fake_ai_bot {
    default 0;
    "anthropic:0:0" 1;
    "anthropic:0:1" 1;
    "openai:0:0"    1;
    "openai:1:0"    1;
}

Second, verified-ai-bot-allow.conf - the bypass, included in each server block before the blocker:

# verified-ai-bot-allow.conf
#
# Lets verified AI crawlers past the Nginx Ultimate Bad Bot Blocker.
# Include in each server{} block BEFORE the blocker's blockbots.conf include.
#
# Zeroes the blocker's IP check ($validate_client - covers blacklist-ips.conf
# and any AbuseIPDB include) and its User-Agent check ($bad_bot) for requests
# that carry a genuine crawler UA from that operator's published ranges.
# Spoofed UAs from other addresses are untouched and still hit the blocker.
#
# Depends on $verified_ai_bot from verified-ai-bots.conf.

if ($verified_ai_bot) {
    set $validate_client 0;
    set $bad_bot 0;
}

Third, fake-ai-bot-block.conf - optional. You don’t need it if you run the Bad Bot Blocker, which already drops these UAs by default, but it makes the project useful on its own for servers without it. Include it in each server block you want protected:

# fake-ai-bot-block.conf
#
# OPTIONAL. Drops requests that spoof an Anthropic/OpenAI crawler UA from an
# address outside that operator's published ranges. Not needed if you run the
# Bad Bot Blocker (it already blocks these UAs by default); useful on servers
# without it. Include in each server{} block you want protected.
#
# Depends on $fake_ai_bot from verified-ai-bots.conf.

if ($fake_ai_bot) {
    return 444;
}

The same Cloudflare note from the Googlebot post applies if you use it: behind a proxy a 444 surfaces as a 520, so change it to 403 if you would rather keep your origin error stats clean.

Keeping the lists up to date

The two *-ips.conf files are generated by the updater script. It is the Googlebot one with two changes: each provider can have several source URLs (OpenAI’s three files are merged and deduplicated into one list), and a provider’s list is only replaced if every one of its sources fetched and parsed. A partial merge would silently drop a whole crawler’s ranges, which is exactly the failure this is meant to prevent. As before, nginx is only reloaded if a list actually changed and nginx -t passes.

#!/usr/bin/env bash
#
# update-verified-ai-bot-ips.sh
#
# Fetches the official Anthropic and OpenAI crawler IP ranges and writes them
# as nginx `geo` value files, consumed by verified-ai-bots.conf.
#
# Usage: ./update-verified-ai-bot-ips.sh [install_dir]
#   install_dir   Default: /etc/nginx/verified-ai-bots
#                 Must match the include paths inside verified-ai-bots.conf.
#
# Environment overrides:
#   INSTALL_DIR   Same as the positional argument.
#   RELOAD_CMD    Full reload command (e.g. "systemctl reload nginx").
#
# Dependencies: bash, curl, jq, nginx
#
set -euo pipefail

INSTALL_DIR="${INSTALL_DIR:-${1:-/etc/nginx/verified-ai-bots}}"

# Official published sources (verified August 2026). All share the Google
# ipranges shape: {"prefixes":[{"ipv4Prefix":...},{"ipv6Prefix":...}]}.
ANTHROPIC_URLS="https://claude.com/crawling/bots.json"
OPENAI_URLS="https://openai.com/gptbot.json https://openai.com/searchbot.json https://openai.com/chatgpt-user.json"

# Refuse to write a list shorter than this many prefixes (guards against a
# truncated / partial response silently shrinking the allowlist).
MIN_PREFIXES=10

RELOAD_CMD="${RELOAD_CMD:-}"

CURL_OPTS="--fail --silent --show-error --location --max-time 30 --retry 3 --retry-delay 5"

log() { printf '%s %s\n' "$(date -u +%FT%TZ)" "$*"; }
err() { log "ERROR: $*" >&2; }

for bin in curl jq nginx; do
    command -v "$bin" >/dev/null 2>&1 || { err "required command '$bin' not found in PATH"; exit 1; }
done

mkdir -p "$INSTALL_DIR"

TMP_DIR="$(mktemp -d)"
# shellcheck disable=SC2064
trap "rm -rf '$TMP_DIR'" EXIT

CHANGED=0

# build_list <name> <output_file> <url> [url...]
# All URLs for a provider must fetch and parse, otherwise the existing list is
# kept untouched - a partial merge would silently drop a whole crawler's ranges.
build_list() {
    name="$1"; out="$2"; shift 2
    staged="${out}.new"   # same directory as $out => atomic rename
    jsons=()

    for url in "$@"; do
        json="${TMP_DIR}/${name}-${#jsons[@]}.json"
        # shellcheck disable=SC2086
        if ! curl $CURL_OPTS -A "verified-ai-bot-ip-updater/1.0 (+nginx)" -o "$json" "$url"; then
            err "fetch failed for ${name} (${url}) - keeping existing list"
            return 1
        fi
        if ! jq -e '.prefixes | length > 0' "$json" >/dev/null 2>&1; then
            err "${name}: ${url} is not valid JSON or has no prefixes - keeping existing list"
            return 1
        fi
        jsons+=("$json")
    done

    {
        echo "# Auto-generated $(date -u +%FT%TZ)"
        for url in "$@"; do echo "# Source: ${url}"; done
        echo "# Managed by update-verified-ai-bot-ips.sh - DO NOT EDIT BY HAND"
        jq -r '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) | select(. != null) | "\(.) 1;"' "${jsons[@]}" | sort -u
    } > "$staged"

    count="$(grep -c ' 1;$' "$staged" || true)"
    if [ "$count" -lt "$MIN_PREFIXES" ]; then
        err "${name}: only ${count} prefixes (< ${MIN_PREFIXES}) - refusing to install, keeping existing list"
        rm -f "$staged"
        return 1
    fi

    # Compare ignoring the first line (the Auto-generated timestamp), otherwise
    # every run would look like a change and trigger a pointless reload.
    if [ -f "$out" ] && cmp -s <(tail -n +2 "$staged") <(tail -n +2 "$out"); then
        rm -f "$staged"
        log "${name}: unchanged (${count} prefixes)"
        return 0
    fi

    mv "$staged" "$out"
    chmod 0644 "$out"
    CHANGED=1
    log "${name}: updated -> ${out} (${count} prefixes)"
    return 0
}

# Portable reload: explicit override wins, otherwise try the common options.
reload_nginx() {
    if [ -n "$RELOAD_CMD" ]; then
        eval "$RELOAD_CMD"
        return $?
    fi
    if command -v systemctl >/dev/null 2>&1; then
        systemctl reload nginx && return 0
    fi
    if command -v service >/dev/null 2>&1; then
        service nginx reload && return 0
    fi
    nginx -s reload
}

# shellcheck disable=SC2086
build_list "anthropic" "${INSTALL_DIR}/anthropic-ips.conf" $ANTHROPIC_URLS || true
# shellcheck disable=SC2086
build_list "openai"    "${INSTALL_DIR}/openai-ips.conf"    $OPENAI_URLS    || true

if [ "$CHANGED" -eq 0 ]; then
    log "No changes; not reloading nginx."
    exit 0
fi

if nginx -t >/dev/null 2>&1; then
    reload_nginx
    log "Lists changed and config valid; nginx reloaded."
else
    err "nginx -t FAILED after updating lists; NOT reloading. Output below:"
    nginx -t || true
    exit 1
fi

The generated anthropic-ips.conf is a plain list of ranges, each with the value 1:

# Auto-generated 2026-08-31T00:00:00Z
# Source: https://claude.com/crawling/bots.json
# Managed by update-verified-ai-bot-ips.sh - DO NOT EDIT BY HAND
136.107.176.208/32 1;
16.58.26.69/32 1;
18.225.238.228/32 1;
216.73.216.0/22 1;
20.102.46.224/28 1;

Installing it

You will need jq if you don’t already have it from the Googlebot project: apt install jq (or the equivalent for your distro).

Create the directory and the files, pasting the contents in from GitHub:

mkdir -p /etc/nginx/verified-ai-bots /opt/scripts

nano /etc/nginx/verified-ai-bots/verified-ai-bots.conf
nano /etc/nginx/verified-ai-bots/verified-ai-bot-allow.conf
nano /opt/scripts/update-verified-ai-bot-ips.sh

Set ownership and permissions:

chown -R root:root /etc/nginx/verified-ai-bots
chown root:root /opt/scripts/update-verified-ai-bot-ips.sh
chmod 755 /etc/nginx/verified-ai-bots
chmod 644 /etc/nginx/verified-ai-bots/*.conf
chmod 750 /opt/scripts/update-verified-ai-bot-ips.sh

Run the script once to generate the IP lists. Do this before adding the includes, or nginx -t will fail on the missing files:

/opt/scripts/update-verified-ai-bot-ips.sh

Load the definitions once, inside the http{} block of your nginx.conf:

include /etc/nginx/verified-ai-bots/verified-ai-bots.conf;

Then add the bypass to each server{} block, on the line above the blocker’s include. Order matters here: the set has to run before the blocker’s if ($validate_client). With ISPConfig, that is wherever you pull blockbots.conf into each vhost - the template or the site’s nginx directives:

include /etc/nginx/verified-ai-bots/verified-ai-bot-allow.conf;
include /etc/nginx/bots.d/blockbots.conf;

Now remove the User-Agent overrides for these crawlers from blacklist-user-agents.conf, and any per-IP 0 entries you had added for them in whitelist-ips.conf (which were also the source of nginx’s duplicate network warnings). Test and reload:

nginx -t && nginx -s reload

And a cron job to keep the lists current. Twice daily is plenty; nginx only reloads when something actually changes:

23 */12 * * * /opt/scripts/update-verified-ai-bot-ips.sh >> /var/log/verified-ai-bot-ips.log 2>&1

The same Cloudflare caveat as before applies: geo matches $remote_addr, so if you are behind a proxy you must be restoring the real client IP or every request will look like it comes from the proxy and nothing will ever verify.

Testing

A spoofed request from your own machine should still be dropped. This is the case that the UA whitelist was getting wrong:

curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])" https://example.com/
# curl: (52) Empty reply from server   <- the blocker's 444, as it should be

A control request with no crawler UA must still return 200:

curl -I https://example.com/
# HTTP/2 200

The genuine crawler you can only check in the logs. The next ClaudeBot hit from 216.73.216.x should be a 200, even though the address is still sitting in the AbuseIPDB list:

216.73.216.108 - - [31/Aug/2026:04:12:07 +0000] "GET /blog/category/servers/ HTTP/2.0" 200 38214 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])"

If you want to prove which layer is firing rather than infer it, a temporary log_format that includes $claimed_ai_bot $is_anthropic_ip $verified_ai_bot $validate_client $bad_bot on the vhost makes it obvious in one line.

What this does and doesn’t do

It exempts crawlers by verified origin, not by UA. That is the whole point, but it also means that if Anthropic or OpenAI publish a range and then one of their addresses misbehaves, it will still get through. Rate limiting is the tool for that, not IP verification.

It only covers operators that publish ranges. Perplexity and others that publish lists would slot in as another geo block and a couple of map lines. Crawlers that don’t publish ranges (Bytespider, most of the smaller AI scrapers) can’t be verified this way, and for those the blocker’s default behaviour - block - is the right one.

The OpenAI lists are merged rather than matched per bot. A GPTBot UA arriving from a ChatGPT-User address will verify. I couldn’t think of a scenario where that matters, and it keeps the composite map to four lines per operator.

And it doesn’t change the AbuseIPDB script at all. That was a deliberate choice - the blocklist stays complete and honest, and the exemption is a separate, visible decision made in nginx config rather than a silent hole in the data.

The full project is on GitHub at robwent/nginx-verified-ai-bots. It pairs with the fake Googlebot blocker - same pattern, same script structure - and with the AbuseIPDB integration that made it necessary in the first place.