Robots

Antora can generate a robots.txt file to control which paths site crawlers can visit.

robots key

The robots key is optional. It’s configured under the site key in a playbook. Its accepted values are:

allow: Allow all user agents to access all paths in the published site.
disallow: Disallow all user agents from accessing all paths in the published site.
custom multi-line string: Non-empty, user-specified string that’s used as the contents of the generated robots.txt file.

Example 1. antora-playbook.yml

site:
  title: Site Title
  url: https://example.org
  robots: disallow

Antora will generate the robots.txt file at the root of the site.

The robots key depends on the sibling url key. If the url key isn’t assigned a value, Antora won’t generate a robots.txt file even if robots is set to an allowable value.

Assign allow or disallow to robots

Let’s assume the url and robots keys are defined as follows:

Example 2. antora-playbook.yml

site:
  title: Demo Docs Site
  url: https://demo.antora.org
  robots: allow

This will generate a robots.txt file that allows user agents access to all of the site’s paths.

Generated robots.txt

User-agent: *
Allow: /

Similarly, if the url and robots keys are defined as follows:

Example 3. antora-playbook.yml

site:
  title: Demo Docs Site
  url: https://demo.antora.org
  robots: disallow

This will generate a robots.txt file that disallows access to all paths.

Generated robots.txt

User-agent: *
Disallow: /

Generate robots.txt with custom content

Any other non-empty value assigned to robots will be used as the contents of the robots.txt file. For example, let’s assume the robots key is declared as follows:

Example 4. antora-playbook.yml

site:
  title: Demo Docs Site
  url: https://demo.antora.org
  robots: |
    User-agent: *
    Disallow: /private/

This will result in the following robots.txt file being generated.

Generated robots.txt

User-agent: *
Disallow: /private/