Robots.txt Generator

Pick a preset or fine-tune the rules, then copy your robots.txt file. Everything runs in your browser.

Your robots.txt


    Copied!
    
  


  

    
Where do I put my robots.txt file?
      
Upload it to the root of your domain so it is reachable at https://yoursite.com/robots.txt — search engines only look there, not in subfolders.

    
What is the difference between robots.txt blocking and a no-index tag?
      
robots.txt stops crawlers from fetching a page, but the URL can still appear in results if linked elsewhere. A no-index meta tag tells search engines to drop the page from results entirely — so don't block pages with robots.txt if you want them de-indexed.

    
Should I include my sitemap in robots.txt?
      
Yes — adding a Sitemap: line pointing to your XML sitemap helps search engines find all your pages faster.

    
Does Disallow hide my site from Google completely?
      
Not instantly and not fully. Blocking crawlers prevents crawling but Google may still list the URL without a snippet. To remove a page from search results, use a no-index tag instead.