A site's /robots.txt, /favicon.ico, its home-screen icons and /site.webmanifest, and /sitemap.xml, with the /sitemap-00001.xml.gz… files it lists, each its own, answered on the site's own address.

All are routed through MishkaCmsCoreWeb.Routes' :public list, so the site is the one the request's host resolved to (conn.assigns.current_site), and an offline site answers them with MishkaCms.Plugs.MaintenanceGate's 503 before this is reached. Every URL they name is absolute, on the site's host (MishkaCmsCore.Helpers.site_url/2).

robots.txt

Built on every request, from ETS:

1. the rules every site has — User-agent: * and a Disallow for /admin/, /api/ and /_preview/ — and the sign-in page with a query and the pages under it, at /auth and at its address in each language the site publishes in (MishkaCmsCore.Runtime.Plugs.SiteLookupPlug.auth_address/3): Disallow: /auth$, Disallow: /auth? and Disallow: /auth/, then the same for /fa/auth…, which leave an address that only starts the same way, such as /authors, open;
2. what a plugin on seo:robots:lines makes of them: it is handed %{site: site, lines: lines} and returns the map with its lines;
3. the site's seo.robots.extra setting, a line at a time;
4. a blank line and Sitemap: <the site's /sitemap.xml>.

Sent as text/plain.

favicon.ico

The site's favicon, the picture its pages name in <link rel="icon"> (MishkaCmsCore.Runtime.Seo.favicon/1). A file of the site's own is sent as it is, with the type its first bytes say (MishkaCmsCore.Runtime.SiteIcon.favicon/1), never sniffed into another, and an SVG sandboxed; any cache may keep it a day. A favicon the site does not own — an http://… or https://… address, which is never fetched, or another site's file — is a redirect to it, kept a minute. A browser asks for it on a page that names none, such as an admin page. A site with no favicon, or one set to anything else, is a 404 no cache keeps. Answers whatever Accept a browser sends.

Home-screen icons

/apple-touch-icon.png and /apple-touch-icon-precomposed.png are the 180×180 PNG a phone puts on its home screen, /site-icon-192.png and /site-icon-512.png the ones the manifest names, drawn from the site's App icon, else its Favicon, the first time one is asked for (MishkaCmsCore.Runtime.SiteIcon.file/2). Asked with ?v= and the hash of the picture they are drawn from, as a page's head and the manifest name them, any cache may keep one a year (immutable); asked without it, or with an older one, a day. A site with no picture to draw them from is a 404 no cache keeps.

site.webmanifest

The site's name, start_url, display and the 192 and 512 icons, and its short name, theme color and background color when it set them, as MishkaCmsCore.Runtime.SiteIcon.manifest/1 builds them from ETS and a plugin on runtime:site:manifest leaves them, sent as application/manifest+json. A 404 no cache keeps for a site with no icons, which its pages do not link it for.

sitemap.xml and its files

Served from MishkaCmsCore.Runtime.States.AssetsState under {site.name, "sitemap"}, which MishkaCmsCore.Runtime.Compilers.SitemapCompiler fills — what goes in it, and when it is built, is described there. No request builds it. /sitemap.xml is its <sitemapindex>, sent as application/xml; /sitemap-00001.xml.gz… are the gzipped files it lists, at the root beside it, each sent as it is stored, as application/gzip. Both with x-robots-tag: noindex.

A file answers whatever Accept a crawler sends, as a static file does (MishkaCmsCore.ExtensionRouter.route_accepts/2).

Caching

public, max-age=60 unless said otherwise above, and no session cookie: the session is read, as the gates in front of these need it, and never written back, so a shared cache may keep the answer. An admin let through to an offline site gets private, no-store, as the gate gives every page it lets them through to. A gzipped file adds no-transform, which keeps the server's response compression from gzipping it again on every request.

Refusals

A file name that is not sitemap-<5 digits>.xml.gz, or one the stored sitemap does not hold, or a request for the sitemap with no site, is a 404 that no cache keeps. A site whose sitemap this node has not built yet — one created a moment ago, a node still booting — has its job enqueued and is answered 503 with retry-after: 60 and cache-control: no-store, so a crawler comes back rather than reading the sitemap as gone. A plugin on web:request:sitemap_unavailable hears each first, with %{site: site | nil, reason: :unknown_file | :no_site | :not_built} as its meta, and may answer instead by halting the conn.