Robots.txt Generator
Create a robots.txt file for WordPress, online stores and custom sites, with options to block AI crawlers.
- SEO & Google
- 100% free
- No sign-up
- Private: runs in your browser
robots.txt
Where to put robots.txt
Upload the file to the main folder of your website (in cPanel that's usually public_html), so it opens at yourbusiness.com/robots.txt. Then submit your sitemap in Google Search Console.
Careful: one wrong line, like Disallow: /, hides your whole site from Google. Use the staging option only on a test copy of your site.
How it works
How to use the Robots.txt Generator
- 1
Pick a ready setup, like WordPress or an online store.
- 2
Choose what to hide from search engines and AI crawlers.
- 3
Download robots.txt and upload it to your website's main folder.
Questions about the Robots.txt Generator
What is a robots.txt file?
A small text file at yourwebsite.com/robots.txt that tells search engine crawlers which parts of your site to skip.
Does robots.txt hide a page from Google?
No. It stops crawling, but a blocked page can still appear if other sites link to it. Use a noindex tag to keep a page out of results.
Can I block AI crawlers like GPTBot?
Yes. The generator can add rules for GPTBot, CCBot, Google-Extended and others. Well-behaved crawlers follow them.
Rather not do it yourself?
We can set this up for you, or build your whole website.
Send us a message with what you need. We reply within a few hours.