Optimize robots.txt for Better SEO

If you aren’t in the know, search engine is not as smart as what you think it is. It can’t tell which pages of your site should be included in the index and which files to ignore. Therefore we need to instruct it on what to do using a robots.txt file.

What is robots.txt file?

A robots.txt file is a set of instructions that tell search engine robots which pages of your site to be crawled and indexed. In most cases, your site is consist of many files or folders i.e. admin folders, cgi-bin, image folder, which are not relevant to the search engines.

Basically the purpose of creating a robots.txt file is to improve site indexation by telling search engine crawler to only index your content pages and to ignore other pages (i.e. monthly archives, categories folders or your admin files) that you do not want them to appear on the search index because it may lead to duplicate content issue.

How to create robots.txt file?

A robots.txt is just a simple text file that can be easily created using any text editor such as Notepad and save it exactly as robots.txt.

Here is the basic syntax of a robots.txt file

User-agent: *
Disallow: [files or folders that are excluded]

User-agent: * The asterisk (*) or wildcard means that all the instructions within the robot.txt file are applicable to all search engine spiders (Google, Yahoo, MSN and others) whereas

Disallow: when leave blank basically tell those spiders to crawl and index the entire site which is of course not the most prudent thing to do with respect to SEO.

Basic example of robots.txt file

If you include an instruction that looks like this:

User-agent: *
Disallow: /wp-admin/

It basically tells all robots that they cannot index the contents of that /wp-admin/ directory. I hope it does make sense so far, doesn’t it?

There seems to be a number of different view points on what should and shouldn’t be included in the robots.txt file. If you’d like to see what the big guys are doing, you can have a look at this collection of robots.txt files from Daily Blog Tips.

If you are interested to know mine, here is the robots.txt file that I use (feel free to copy and tinker with it)

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /wp-*
Disallow: /feed/
Disallow: /trackback/
Disallow: /tag/
Disallow: /cgi-bin/
Disallow: /2008/

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

After you have done creating the robots.txt file, double check to ensure that you have not made any error on it which will otherwise cause the spider not to index any files meant to be indexed.

Optimizing your robots.txt file will help prevent Google penalizing you for duplicate content and also improve your search engine rankings.

Have you optimized yours today?

Update:
It is important to note that your robots.txt file should always reside in the root directory of your site; even though in some cases, your site may be in another sub-directory. Validate it by entering the full url www.yoursite.com/robots.txt in your web browser.

Related posts

• Now It’s Time To Protect Your Blog What have you done lately to protect your blog? You probably never thought of it in the first place, have...
• 15 Most Under-Utilized Plugins for SEO Search Engine Optimization is all about optimizing your blog to position it higher in the Search Engine Results and thus...
• 15 SEO Tips for Beginners Search Engine Optimization (SEO) is all about optimizing your blog to rank high in the Search Engine Results and thus...
• How To Use FTP – The Basic If you are wondering what FTP is, it is an acronym for File Transfer Protocol. It is the simplest and...
• The Way I Transfer My WordPress Blog To Another Host OMG! I did it. I was very nervous in the beginning as this was the first time I attempted to...