SEO ToolsFree

robots.txt Generator

Easily configure and generate a robots.txt file with rules for any user-agent. Control which pages crawlers can access, set crawl delays, and specify your sitemap URL - no manual syntax required.

robots.txtseocrawlindexingnext.jstechnical seo

Rule 1

User-agent

Disallow (one per line)

Allow (one per line)

Sitemap URL

Crawl-delay (seconds, optional)

robots.txt

User-agent: *
Disallow: /admin
Disallow: /api
Allow: /

Sitemap: https://example.com/sitemap.xml

Related Tools

Meta Tag Generator

Generate complete HTML meta tags for SEO instantly

Open Graph Preview Tool

Preview how your page looks when shared on social media

Sitemap XML Generator

Generate a valid XML sitemap for search engines