Sitemap engineering for large sites: partitions, lastmod, and what to monitor

Most sitemaps are generated, submitted once, and forgotten. For a site with a few hundred URLs that is fine. Past a few thousand, the sitemap stops being a formality and becomes an inventory-management system — and Google reads the quality of that inventory as a signal.

Partition by what changes together

One sitemap with 80,000 URLs tells you nothing when coverage drops. The whole point of partitioning is that the sitemap file becomes a diagnostic unit:

By section: /blog, /products, /docs each in their own sitemap. When one section's coverage degrades, the boundary is visible.
By change rate: pages that update daily (listings, inventory) apart from pages that never change (docs, legal). Freshness expectations differ, and mixing them makes lastmod meaningless.
By locale/region if you run hreflang: coverage problems are often locale-scoped, and a per-locale sitemap turns "international coverage dipped" into "the de sitemap dipped."

indexsubmit syncs your sitemaps on a schedule and tracks coverage per sitemap section for exactly this reason — the sitemap index is the natural axis for diagnosis.

lastmod: honesty is a strategy

lastmod is the most abused field in the protocol. Every sitemap generator that stamps build time on every URL has taught crawlers to distrust it. The evidence that Google uses lastmod for crawl prioritization when it is accurate is good; the evidence that it ignores it when it smells generated is equally good.

Rules that keep lastmod credible:

lastmod = the last content modification, not the last build, deploy, or cache purge.
If your CMS cannot produce a real content version, omit lastmod entirely. A sitemap without lastmod beats a sitemap that lies.
When you fix a real lastmod, expect the effect to be gradual — trust in the field is per-site and rebuilt over time.

What a sitemap is and is not

A sitemap is a discovery aid — a claim that these URLs exist and matter. It is not a submission protocol, not a guarantee, and not a fix for weak internal linking. Pages that only exist in the sitemap, with no contextual internal links, get the weakest possible treatment: crawled rarely, trusted less.

The practical consequence: use the sitemap to define the inventory, and use internal links to argue for it. If a URL matters, some other real page should link to it.

What to monitor once it is live

The sitemap is the denominator of your entire coverage story. Four numbers, tracked over time, tell you almost everything:

Synced vs. indexed per section — the coverage rate each partition.
Discovered, not indexed per section — Google's quality verdict on that URL class.
404s and redirects — sitemap entries pointing at dead or moved URLs are trust leaks.
New URL time-to-first-index — how long a fresh entry takes to be indexed; a rising number is an early warning about site-wide trust.

If you are tracking these in a spreadsheet by hand, that is the job indexsubmit automates: sitemaps synced on schedule, every URL's verdict recorded daily, each section scored and trended for 30 days, and an alert when a section's coverage degrades.

Start by looking at your sitemap index the way Google's crawler does — as competing inventories — and partition it so the story it tells is one you can act on.

Incomplete syncs and plan limits

A sync that reaches your plan's URL inventory limit remains incomplete. It does not provide evidence that unseen pages were removed. After you increase the limit, indexsubmit fetches the full sitemap again; a cached “not modified” response cannot certify the earlier partial inventory.