A site's sitemap, built ahead of the request with Sitemapper and kept in MishkaCmsCore.Runtime.States.AssetsState under {site.name, "sitemap"} as %{index: index, files: files}. MishkaCmsCoreWeb.SeoController serves it and never builds it.

What goes in it

Its addresses, each once whoever lists it, the first to name a path keeping it, and none ending in a slash — /blog/ and /blog are the one /blog, and the home page is the site's address alone, https://example.com:

1. the site's pages a search engine may be sent to (MishkaCmsCore.Runtime.PageEditions.indexable/1), each at its public addresses (PageEditions.urls/2): a literal page at its one, and a page whose only capture is the language at one per language it answers (PageEditions.served_languages/2);
2. the site's seo.sitemap.paths setting;
3. what a plugin on seo:sitemap:urls adds, below.

Every address is the one a visitor is served at (MishkaCmsCore.Runtime.Languages.url/3): on a site that leaves its primary language unprefixed, /en/pricing, whoever lists it, is listed as /pricing, so no address in the sitemap redirects.

None is listed at the path of a literal page indexable/1 leaves out — a login-only or a noindex one — whichever of them names it: the resolver answers that page there before any pattern. Nor at an address the page that serves it names permissions for, a pattern page included (MishkaCmsCore.Runtime.PageEditions.locked_ids/1, read when the sitemap is built): a plugin's /blog/hello is left out while /blog/:slug is locked, since a crawler is answered the 404 there. Nor at a path a route in front of the pages answers — every :public GET or live route of the configured extensions (MishkaCmsCore.ExtensionRouter), /robots.txt, /sitemap.xml and this sitemap's own files among them — since a page stored there is never served. A path that does not start with / is left out too.

Every address is absolute, on the site's host (MishkaCmsCore.Helpers.site_url/2), and so is every image: one at a path on the site is made absolute there, one with no scheme (//cdn.example.com/a.jpg) takes the site's, an http(s):// one is kept as it is, and anything else, or one with no host, is left out. What a URL may not hold as it is — a space, a control character, anything past ASCII — is percent-encoded in both, and an escape already in them is kept, so no address breaks the file it is in. A lastmod is written to the second.

What is stored

• files — "sitemap-00001.xml.gz", "sitemap-00002.xml.gz"… by name, each a gzipped <urlset> of up to 50,000 addresses and 50 MB, served at /<name> as it is stored. At the root, because a sitemap may list only the addresses under the folder it is served from;
• index — the plain <sitemapindex> of them, served at /sitemap.xml, each file listed at https://<host>/<name> with the time this node first stored those files as its lastmod.

A build that makes the files already stored keeps what is stored, index and all, so a file's lastmod moves only when its bytes do, and a rebuild that changed nothing — a layout's, a plugin chain's — tells a crawler nothing changed. Gzip writes no time into a file, so the same addresses make the same bytes. A node that starts again dates its files from its first build.

A site with nothing to list has no file, and its index is a <sitemapindex> with no <sitemap> in it. The sitemaps.org schemas want at least one entry in an index and in a <urlset> alike, so no document for an empty site validates against them, and Google reports either as an "Empty sitemap" rather than failing to fetch it; the index keeps /sitemap.xml the same kind of document at every size, and a <urlset> listed in it would be a second empty one.

seo:sitemap:urls

A plugin is handed %{site: site, pages: pages, urls: urls}:

• site — the site as MishkaCmsCore.Runtime.States.SiteState holds it;
• pages — the list of the site's indexable pattern pages holding a capture only their owner can fill, /:lang/blog/:slug;
• urls — the addresses so far, an Enumerable of %{path: String.t(), lastmod: DateTime.t() | nil, images: [String.t()] | nil}, images optional.

It returns the map with its own addresses after them, as a stream read a batch at a time, so a site of any size is built without holding its rows — stream_rows/2 reads a query that way:

def call(%{site: site, urls: urls} = data),
  do: {:reply, %{data | urls: Stream.concat(urls, posts(site))}}

defp posts(site) do
  MyExt.Post
  |> Ash.Query.for_read(:public_list, %{}, tenant: site.id, authorize?: false)
  |> Ash.Query.select([:slug, :updated_at, :cover])
  |> SitemapCompiler.stream_rows(tenant: site.id, authorize?: false)
  |> Stream.map(&%{path: "/posts/" <> &1.slug, lastmod: &1.updated_at, images: [&1.cover]})
end

A path listed twice — by core and a plugin, by two plugins, or by one plugin twice, with or without a slash at its end — is listed once, the first time it comes: a build remembers every path it wrote while it reads the stream.

Nothing is read while the chain runs: the build reads the stream once, after it, so a stream that raises then fails the build — the job is retried as Oban retries any, and the sitemap stored before is served meanwhile. A chain that ends in a urls that is not an Enumerable is ignored, and core's own addresses are built alone.

When it is built

At boot, by every node for every open site, two at a time (MishkaCmsCore.Runtime.Handlers.BootHandler.build_sitemaps/0). After that, by the site's "sitemap" job, on a :sitemaps queue of its own, that schedule/1 enqueues when what it lists changes:

• a page or a layout — MishkaCmsCore.Runtime.Workers.RuntimeCompilerWorker, after its own;
• the site switched on, renamed or installed — BootHandler.compile_site/2; its host — MishkaCmsCore.Runtime.Resources.Changes.ScheduleSiteRuntime;
• its languages, how its addresses name them, or its seo.sitemap.paths, and a row an extension lists — MishkaCmsCore.Runtime.Resources.Notifiers.SitemapRefresh;
• a seo:sitemap:urls plugin started or stopped — every open site's (attach/0).

A site has one such job pending at a time — a build that failed and waits to retry counts — and each enqueue moves it later, so a burst of changes is one build. A build is given ten minutes and three tries. A request that finds no sitemap enqueues it too, but leaves a pending or running one where it is, so requests coming faster than it is due cannot keep putting it off or start a second.

An extension whose seo:sitemap:urls plugin lists rows of its own resources adds MishkaCmsCore.Runtime.Resources.Notifiers.SitemapRefresh to those resources' notifiers:, or calls schedule/1 with the row's site where no notifier sees the change.

Each node keeps its own copy. A job rebuilds it only on the node that runs it, so another node serves its previous one until its own next build (ROADMAP_DISTRIBUTED_CMS.md, row 17).

Hooks

• runtime:compile:before_sitemap — %{site: site_name}, before a build;
• seo:sitemap:urls — the addresses a plugin adds, above;
• runtime:compile:after_sitemap — %{site: site_name, urls: count, files: count}, once it is stored or the stored one is kept.

Storing a changed one fires runtime:state:assets:changed (AssetsState.save/1).