# `MishkaCmsCore.Runtime.Compilers.SitemapCompiler`
[🔗](https://github.com/mishka-group/mishka_cms_core/blob/v0.0.1/lib/mishka_cms_core/runtime/compilers/sitemap_compiler.ex#L5)

A site's sitemap, built ahead of the request with `Sitemapper` and kept in
`MishkaCmsCore.Runtime.States.AssetsState` under `{site.name, "sitemap"}` as
`%{index: index, files: files}`. `MishkaCmsCoreWeb.SeoController` serves it and never builds it.

## What goes in it

Its addresses, each once whoever lists it, the first to name a path keeping it, and none ending in
a slash — `/blog/` and `/blog` are the one `/blog`, and the home page is the site's address alone,
`https://example.com`:

  1. the site's pages a search engine may be sent to
     (`MishkaCmsCore.Runtime.PageEditions.indexable/1`), each at its public addresses
     (`PageEditions.urls/2`): a literal page at its one, and a page whose only capture is the
     language at one per language it answers (`PageEditions.served_languages/2`);
  2. the site's `seo.sitemap.paths` setting;
  3. what a plugin on `seo:sitemap:urls` adds, below.

Every address is the one a visitor is served at (`MishkaCmsCore.Runtime.Languages.url/3`): on a
site that leaves its primary language unprefixed, `/en/pricing`, whoever lists it, is listed as
`/pricing`, so no address in the sitemap redirects.

None is listed at the path of a literal page `indexable/1` leaves out — a login-only or a noindex
one — whichever of them names it: the resolver answers that page there before any pattern. Nor at
an address the page that serves it names permissions for, a pattern page included
(`MishkaCmsCore.Runtime.PageEditions.locked_ids/1`, read when the sitemap is built): a plugin's
`/blog/hello` is left out while `/blog/:slug` is locked, since a crawler is answered the 404
there. Nor at
a path a route in front of the pages answers — every `:public` GET or live route of the
configured extensions (`MishkaCmsCore.ExtensionRouter`), `/robots.txt`, `/sitemap.xml` and this
sitemap's own files among them — since a page stored there is never served. A `path` that does
not start with `/` is left out too.

Every address is absolute, on the site's `host` (`MishkaCmsCore.Helpers.site_url/2`), and so is
every image: one at a path on the site is made absolute there, one with no scheme
(`//cdn.example.com/a.jpg`) takes the site's, an `http(s)://` one is kept as it is, and anything
else, or one with no host, is left out. What a URL may not hold as it is — a space, a control
character, anything past ASCII — is percent-encoded in both, and an escape already in them is
kept, so no address breaks the file it is in. A `lastmod` is written to the second.

## What is stored

  * `files` — `"sitemap-00001.xml.gz"`, `"sitemap-00002.xml.gz"`… by name, each a gzipped
    `<urlset>` of up to 50,000 addresses and 50 MB, served at `/<name>` as it is stored. At the
    root, because a sitemap may list only the addresses under the folder it is served from;
  * `index` — the plain `<sitemapindex>` of them, served at `/sitemap.xml`, each file listed at
    `https://<host>/<name>` with the time this node first stored those files as its `lastmod`.

A build that makes the files already stored keeps what is stored, index and all, so a file's
`lastmod` moves only when its bytes do, and a rebuild that changed nothing — a layout's, a plugin
chain's — tells a crawler nothing changed. Gzip writes no time into a file, so the same addresses
make the same bytes. A node that starts again dates its files from its first build.

A site with nothing to list has no file, and its `index` is a `<sitemapindex>` with no `<sitemap>`
in it. The sitemaps.org schemas want at least one entry in an index and in a `<urlset>` alike, so
no document for an empty site validates against them, and Google reports either as an "Empty
sitemap" rather than failing to fetch it; the index keeps `/sitemap.xml` the same kind of document
at every size, and a `<urlset>` listed in it would be a second empty one.

## seo:sitemap:urls

A plugin is handed `%{site: site, pages: pages, urls: urls}`:

  * `site` — the site as `MishkaCmsCore.Runtime.States.SiteState` holds it;
  * `pages` — the list of the site's indexable pattern pages holding a capture only their owner can
    fill, `/:lang/blog/:slug`;
  * `urls` — the addresses so far, an `Enumerable` of
    `%{path: String.t(), lastmod: DateTime.t() | nil, images: [String.t()] | nil}`, `images`
    optional.

It returns the map with its own addresses after them, as a stream read a batch at a time, so a
site of any size is built without holding its rows — `stream_rows/2` reads a query that way:

    def call(%{site: site, urls: urls} = data),
      do: {:reply, %{data | urls: Stream.concat(urls, posts(site))}}

    defp posts(site) do
      MyExt.Post
      |> Ash.Query.for_read(:public_list, %{}, tenant: site.id, authorize?: false)
      |> Ash.Query.select([:slug, :updated_at, :cover])
      |> SitemapCompiler.stream_rows(tenant: site.id, authorize?: false)
      |> Stream.map(&%{path: "/posts/" <> &1.slug, lastmod: &1.updated_at, images: [&1.cover]})
    end

A path listed twice — by core and a plugin, by two plugins, or by one plugin twice, with or without
a slash at its end — is listed once, the first time it comes: a build remembers every path it
wrote while it reads the stream.

Nothing is read while the chain runs: the build reads the stream once, after it, so a stream that
raises then fails the build — the job is retried as Oban retries any, and the sitemap stored
before is served meanwhile. A chain that ends in a `urls` that is not an `Enumerable` is ignored,
and core's own addresses are built alone.

## When it is built

At boot, by every node for every open site, two at a time
(`MishkaCmsCore.Runtime.Handlers.BootHandler.build_sitemaps/0`). After that, by the site's
`"sitemap"` job, on a `:sitemaps` queue of its own, that `schedule/1` enqueues when what it lists
changes:

  * a page or a layout — `MishkaCmsCore.Runtime.Workers.RuntimeCompilerWorker`, after its own;
  * the site switched on, renamed or installed — `BootHandler.compile_site/2`; its host —
    `MishkaCmsCore.Runtime.Resources.Changes.ScheduleSiteRuntime`;
  * its languages, how its addresses name them, or its `seo.sitemap.paths`, and a row an
    extension lists —
    `MishkaCmsCore.Runtime.Resources.Notifiers.SitemapRefresh`;
  * a `seo:sitemap:urls` plugin started or stopped — every open site's (`attach/0`).

A site has one such job pending at a time — a build that failed and waits to retry counts — and
each enqueue moves it later, so a burst of changes is one build. A build is given ten minutes and
three tries. A request that finds no sitemap enqueues it too, but leaves a pending or running one
where it is, so requests coming faster than it is due cannot keep putting it off or start a
second.

An extension whose `seo:sitemap:urls` plugin lists rows of its own resources adds
`MishkaCmsCore.Runtime.Resources.Notifiers.SitemapRefresh` to those resources' `notifiers:`, or
calls `schedule/1` with the row's site where no notifier sees the change.

Each node keeps its own copy. A job rebuilds it only on the node that runs it, so another node
serves its previous one until its own next build (`ROADMAP_DISTRIBUTED_CMS.md`, row 17).

## Hooks

  * `runtime:compile:before_sitemap` — `%{site: site_name}`, before a build;
  * `seo:sitemap:urls` — the addresses a plugin adds, above;
  * `runtime:compile:after_sitemap` — `%{site: site_name, urls: count, files: count}`, once it is
    stored or the stored one is kept.

Storing a changed one fires `runtime:state:assets:changed` (`AssetsState.save/1`).

# `sitemap`

```elixir
@type sitemap() :: %{index: binary(), files: %{required(String.t()) =&gt; binary()}}
```

A site's stored sitemap: its plain index, and its gzipped files by name.

# `attach`

```elixir
@spec attach() :: :ok
```

Attach the rebuild to mishka_installer's compile telemetry. Idempotent.

# `compile_and_store`

```elixir
@spec compile_and_store(String.t()) :: {:ok, sitemap()} | {:error, :no_site}
```

Builds the sitemap of the site named `site_name` and stores it in this node's `AssetsState`.

`{:error, :no_site}` when `SiteState` holds no open site by that name, before the build or once it
is done: a site switched off, removed or renamed while it was built has nothing stored.

# `schedule`

```elixir
@spec schedule(String.t() | nil, keyword()) ::
  :ok | {:ok, Oban.Job.t()} | {:error, term()}
```

Enqueues the `"sitemap"` job of the site `site_id` names, or of every open site for `nil` — a
global row.

A switched-off or unknown site has nothing to build, and is `:ok`. So is every call under the
`:env_test?` flag, as for every other runtime job.

## Options

  * `:replace` — what an enqueue changes on the site's job already pending, as Oban's `:replace`.
    Unset, it moves the job later; `[]` leaves it where it is.
  * `:states` — the states of a job already there that this enqueue leaves as it is instead of
    adding one, as Oban's `unique: [states: ...]`. Unset, a pending one or one waiting to retry;
    `:incomplete` a running one too.

# `stream_rows`

```elixir
@spec stream_rows(Ash.Query.t(), keyword()) :: Enumerable.t()
```

Every row `query` reads, as a stream that reads `100` at a time: each read starts after the
last `id` the one before it returned, one indexed step however many rows come before it, where an
offset would walk them all again on every page.

`query` is an `Ash.Query.for_read/4` of an action that paginates, on a resource whose primary key
is `id`. It is read in `id` order whatever order it asks for, with `id` selected beside what it
selects. `opts` are `Ash.read!/2`'s — the tenant, `authorize?`.

---

*Consult [api-reference.md](api-reference.md) for complete listing*