Skip to content
Check a site

For website operators

The Afterward crawler.

If you found Afterward in your server logs, someone asked their AI assistant to check a website with Afterward. Afterward only visits a site when a person requests a scan of it. It does not crawl the web on its own, build a search index, or train AI models on what it fetches.

How to recognize it

Afterward/0.1 (+https://afterward.run/bot)

What one scan requests

  • The homepage, over HTTPS and HTTP, to check redirects and HTTPS.
  • Up to 25 HTML pages on the same hostname, at most two links deep from the homepage (10 pages by default).
  • /robots.txt, the sitemap, /favicon.ico, and one deliberately missing page to check how 404s are handled.
  • Up to 50 linked URLs, checked with a HEAD request (or a GET without reading the body if HEAD fails). These can include links to other sites.

How gently

At most two requests at a time to any one hostname, a 10-second timeout per request, a 2 MB cap on each page, and a 60-second limit for the whole scan. Afterward reads server-delivered HTML only: it does not run JavaScript, keep cookies, sign in, or submit forms. It never connects to private or internal network addresses.

robots.txt

Afterward fetches robots.txt to report on it, for example when a staging rule blocks search engines. It does not currently use robots.txt rules to decide which pages to inspect, because a scan is a one-off check that a person requested, not ongoing crawling.

Blocking it

To stop Afterward scanning your site, block requests whose user agent contains Afterward at your server, CDN, or firewall. A blocked site returns an error to the person who asked for the scan. Questions or problems: support@afterward.run.