Customer Support

• FAQ & Knowledge Base
• System Status
• Contact Support

  Support Categories:

Getting Started FTP Moving Domains E-mail Stat Reports Mailing Lists Apache PHP CGI MySQL Database Other Software Billing & Terms Control Panel E-commerce Pre-Sales

Search for keywords:

Browse by category: -- Select Topic -- Getting Started FTP Telnet/SSH Moving Domains E-mail Traffic Reports Mailing Lists Apache PHP CGI Other Server-Side Scripting MySQL Database Imaging Libraries Other Software Billing & Terms Control Panel E-commerce Pre-Sales

How do I use a robots.txt file?

The robot guidelines are universal and don't depend on any server configuration.

All you have to do is put a robots.txt file in your DocumentRoot (which is normally /htdocs/www/ as you see it when you login with ftp/telnet/ssh). Any of your subdomains has it's own DocumentRoot, so you'll need to make multiple robots.txt files in /htdocs/[subdomain]/ if you want to control robots in there.

When a robot arrives at your site, it requests a robots.txt file from the top level directory (the DocumentRoot) just the same as it requests index.html.

The robot basically goes:

http://www.yourdomain.com/robots.txt

or

http://subdomain.yourdomain.com/robots.txt

This file is the first thing a robot asks for when it arrives, and if the file exists, the robot is supposed to read and obey the directives in that file.

More information about how to direct robots with directives in the robots.txt file is here:

http://www.robotstxt.org/wc/norobots.html

User-Contributed Notes

There are no user-contributed notes for this topic.

Related Questions:

Can I have a crontab?

Why does your service seem slow?

How do I do a traceroute to diagnose network problems?

My POST to a secure form results in a blank page, the same form, or a 500 Internal Server Error.

Internet Explorer shows a blank page on an SSL website like the Control Panel or signup page.

How do I get a command prompt on my own computer?

After I install Bugzilla, I get error 403 Forbidden.

How do I install Gallery?

I am having trouble installing phpBB2

Where can I find links to 'Standards-Compliant' Web browsers?

Do you provide CVS?

Can I use streaming media on my site?

Do you support Flash, Shockwave, Java Applets?

I installed Norton Internet Security, and now I can't see video and other multimedia.

How do I use MnogoSearch Indexer?

Do you run Linux?

Can I use a blog or other content management system?

Do you have Postgres?

Why doesn't mivascript interpret GET variables?

How do I contact my registrar?

What does everything in a Wusage report mean?

Whenever I make changes with Movable Type I get SBOX errors.

Are there any resources for dealing with comment spam in a Movable Type blog?

How Do I Add A Google Sitemap To My Website?

Do you provide SVN?

What can I do to improve Drupal's loading speed?

Browse Categories: Getting Started, FTP, Telnet/SSH, Moving Domains, E-mail, Traffic Reports, Mailing Lists, Apache, PHP, CGI, Other Server-Side Scripting, MySQL Database, Imaging Libraries, Other Software, Billing & Terms, Control Panel, E-commerce, Pre-Sales