🤖

Robots.txt Generator

SEOFreeNew

Generate robots.txt with bot control, presets & live preview

⚡ Quick Presets

🌐 Site Details

Your Domain

Disallow Paths (one per line)

Crawl-delay (sec)

Add Sitemap URL

📄 robots.txt Preview21 lines

User-agent: Yandexbot
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /account/
Disallow: /login/

🔒

100% private. All processing happens in your browser. No data is ever sent to any server.

About Robots.txt Generator

ToolYapa's Robots.txt Generator helps you create the right robots.txt file for your website — whether you want to allow all bots, block specific AI crawlers, or only block your admin and API paths. Choose from 4 quick presets or build custom rules with our bot control panel.

How to use

1
Choose a preset: Allow All, Block All, Block Admin, or SEO Optimal.
2
Or use the custom builder: enter your domain, set disallowed paths, and choose crawl delay.
3
Toggle individual bots: allow Google and Bing while blocking GPTBot, SemrushBot, AhrefsBot.
4
Live preview updates as you build.
5
Copy to clipboard or download as robots.txt ready to place at your domain root.

Who uses this tool

✦
Allow all search engines while blocking AI training bots (GPTBot, ClaudeBot).
✦
Block /admin/, /api/, and /account/ paths from crawling.
✦
Set a crawl delay to prevent aggressive crawlers from overloading your server.
✦
Create a staging-site robots.txt that blocks all bots.
✦
Add your sitemap URL for faster indexing.

Frequently Asked Questions

Where should robots.txt be placed?+

Your robots.txt file must be at the root of your domain: https://toolyapa.com/robots.txt. It cannot be in a subdirectory. In Next.js, place it in the /public folder or use the app/robots.ts route.

Does robots.txt stop Google from crawling my pages?+

Disallowed pages in robots.txt will not be crawled, but they can still appear in search results if other sites link to them. To fully hide a page, use a noindex meta tag or X-Robots-Tag header in addition to robots.txt.

How do I block GPTBot (OpenAI) from crawling my site?+

Add these lines to your robots.txt: User-agent: GPTBot followed by Disallow: /. This prevents OpenAI from using your content for AI training. Similarly use User-agent: ClaudeBot to block Anthropic.

What is Crawl-delay?+

Crawl-delay tells well-behaved bots to wait N seconds between requests. Set it to 1–2 for small sites or shared hosting. Google ignores Crawl-delay — use Google Search Console to manage crawl rate for Googlebot.

Related Tools

🗺️

XML Sitemap Generator

Build XML sitemaps with priorities, frequency…

🏷️

Meta Tag Generator

SEO meta tags, Open Graph & Twitter Card HTML…