Robots.txt Generator

Create and validate robots.txt files to control search engine crawling

100% FreeTemplatesValidation

Create and validate robots.txt files to control search engine crawling

Templates

Start with a pre-configured template

Configuration

Crawl Rules

Rule 1

User Agent

Action

Path

//admin//private//tmp//cgi-bin//*.pdf/images//api//*?*/wp-admin//*.json/*.xml

Generated robots.txt

Preview and download your robots.txt file

# robots.txt generated by Tools Website
# Generated on 2025-08-14

User-agent: *
Allow: /

Robots.txt Tips

Place robots.txt in your website root directory

Use * as user-agent to apply rules to all bots

Test your robots.txt with Google Search Console

Include sitemap URLs to help search engines

Remember: robots.txt is publicly accessible