Crawler rules

Robots.txt Generator

Build a robots.txt file with common allow, disallow, sitemap, and crawl-delay directives.

SEOrobots.txtOffline

Rules

Configure directives

User-agent

Disallow paths

Allow paths

Sitemap URL

Crawl delay

Output