Home/Tools/Robots.txt Generator

Robots.txt Generator

Generate a valid robots.txt file for your website. Control which pages search engines can crawl and index.

Rule 1

User-Agent

Disallow Paths

Allow Paths

Sitemap URL

Crawl Delay (seconds)

Generated robots.txt

# robots.txt generated by Jottler
# https://jottler.io/tools/robots-txt-generator

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

$ jottler --automate-seo

You just did this for one page. Jottler does it for every article — automatically.