IndexNow in production: keys, dedup, and what each engine actually does

IndexNow looks trivially simple, and the protocol is: POST a list of URLs with a key that proves you control the host. Everything that makes it hard lives around that request. This post is the field notes from running IndexNow pipelines at scale — what we learned building indexsubmit's push pipeline.

The key, once

All participating engines — Bing, Yandex, Seznam — read the same key file at your site root (/{key}.txt, root only, no subdirectory tricks). One verified key covers every engine that participates in the protocol. You do not maintain per-engine keys.

Two practical rules:

• Serve the key file with a 200 and the key as the entire body. Redirects break verification.
• If you migrate hosts or CDNs, re-verify before pushing again — an engine that cannot fetch the key silently drops your submissions, and nothing tells you.

indexsubmit verifies the key per host and re-checks it, because the failure mode is silent.

Dedup is your job, not the protocol's

IndexNow has no built-in idempotency. If your CMS fires the same URL three times because a draft was saved twice, that is three submissions. Engines tolerate this, but you burn goodwill and obscure your own signal: when everything is submitted, you cannot tell what mattered.

The discipline that works: deduplicate by content version, not by URL. A URL whose rendered content did not change since the last push does not need a push. A URL whose content did change does — even if it was pushed an hour ago.

In indexsubmit, every push job carries the content version of each URL; repeats within a batch collapse, and the outcome of every submission — accepted, failed, rate-limited, unknown — is recorded per URL and audited.

What each engine actually does

Bing is the most complete implementation: fast ingestion, visible index impact, and the one whose IndexNow dashboard shows the most detail. If your traffic is mostly Google, it is tempting to skip IndexNow. Do not: Bing's index feeds other surfaces, and ingestion today is a precondition for existing there tomorrow.

Yandex participates fully and honors the protocol as documented. Impact on traffic depends on your audience, but the submission pipeline behaves.

Seznam (Czech market) participates. If you do not serve that market, the cost of including it is zero — the same request reaches it.

What no engine does, and this is the part people miss: acknowledgment is not indexing. A 200 or 202 from IndexNow means "received." It says nothing about whether the URL will be indexed, or kept. The only way to close the loop is to check the verdict afterward — which is a URL Inspection question, not an IndexNow question.

A pipeline that survives contact with reality

The minimal production pipeline:

on publish (or content change):
  1. compute content version of the URL
  2. enqueue {url, version} — dedup against the last pushed version
  3. batch (small batches beat one giant request)
  4. POST to api.indexnow.org with your key
  5. record per-URL outcome: accepted | failed | rate_limited | unknown
  6. later: check the URL's index verdict; alert if a pushed URL stays out

Two failure modes to design for:

• Rate limiting. Engines return 429 when you push too hard. Back off per-host; do not retry the whole batch blindly. unknown outcomes happen on network errors — treat them as spent-but-unverified, never silently replay them.
• The loop that never closes. Teams that only push never learn whether the push mattered. Teams that only inspect never accelerate discovery. The value is the pair: push on publish, verify by inspection, alert when a pushed URL fails to show up.

What indexsubmit adds on top

The pipeline above is exactly what indexsubmit runs for you: content-version dedup, small audited batches, per-URL outcome recording (including unknown — which we never auto-replay), and the verification leg that checks whether pushed URLs actually made it into the index, with an alert when they did not.

If you want to wire your own pipeline instead, the API exposes submissions and their outcomes directly. Either way: the POST is the easy ten percent. The other ninety is what makes the difference between "we use IndexNow" and "IndexNow works."