Search our Webmaster Database.

Not finding it on Google? Search our Webmaster Database for templates, scripts, graphics and more.

Create your Robots.txt file here.

Create your own customized Robots.txt file - Block bad bots and allow the good ones in.

Who owns the domain you want.

Find out who owns the domain you want by visiting our Whois Lookup page. Any number of TLD's (top level domains) are listed.


Web Hosting Login:
Web Hosting Login
Webmail Login
  • 406-403-5320
  • Web Hosting Login
  • Webmail Login

Robots txt Generator

ReddWebDev - Robots txt Generator

The robots.txt is a very simple text file that is placed on your root directory. An example would be www.yourdomain.com/robots.txt. This file tells search engine and other robots which areas of your site they are allowed to visit and index. Every website on the internet should have a robots.txt file located in the root directory.

You may use the form below to create your own fairly extensive robots.txt file automatically. Simply read and select which bots or options you would want your robots.txt file to contain. When you are finished, simply copy and paste into notepad, name it robots.txt and upload it into the root directory of your domain and you'll be good to go.

A robots.txt file helps various robots (internet parsing utilities) to know which part of your domain to parse and which parts not to parse. There are hundreds of internet parsing agents, or robots on the world wide web, so it's important to give direction to those robots that actually do read the robots.txt file.


Default Robots.txt file setting;

Default - All Robots are:
Crawl-Delay:

Sitemap: (leave blank for none)   
example: http://www.yoursite.com/sitemap.xml


Specific Search Robots;

Search Engine
Choose Setting
Bot-Name
Google
GoogleBot
Bing
BingBot
Yahoo
Yahoo-slurp
Alexa/Wayback
ia_archiver
Ask/Teoma
Teoma *deprecated
Yandex
Yandex
GigaBlast
Gigabot
Scrub The Web
Scrubby
Baidu
Baiduspider
DuckDuckGo
DuckDuckBot
Blekko
ScoutJet


Specific Special Bots;

Search Engine
Choose Setting
Bot-Name
Google Image
googlebot-image
Yahoo MM
yahoo-mmcrawler *deprecated
MSN Pic-Search
psbot
Yahoo Blogs
yahoo-blogs/v3.9 *deprecated


Extra Exclusion features;

Search Engine
Choose Setting
Bot-Name
Google
GoogleBot
Bing
BingBot
Yahoo
Yahoo-Slurp
Baidu
Baiduspider
Yandex
Yandex
DuckDuckGo
DuckDuckBot
Blekko
ScoutJet


Disallow Bad Bots;

Select this option to block the most common "Bad-Bots," also known for spam and email harvesting from accessing your domain.
Disallow Bad-Bots:


Restricted Directories;

The path is relative to root and must contain a trailing "/"
Exclude Directory: "/"
Exclude Directory: "/"
Exclude Directory: "/"
Exclude Directory: "/"
Exclude Directory: "/"

Create The Robots.txt;

 



Copy and paste the text above into notepad and save it as robots.txt.
Upload your robots.txt file into your root directory.

Example: http://www.yoursite.com/robots.txt