Playing with robots.txt – Some tidbits of advice

When you deal with Search Engine Optimization, I believe that it is of utmost importance to understand how to make your website a search engine friendly one in terms of HTML and site architecture.

Therefore, every single Meta description and title tag will definitely factor in.

A Double-edged Sword

In this post, I would like to introduce and make a thorough discussion on what can be considered as a double-edged sword. This may be considered as one of the most promising Meta element that can produce more ills for your website if you use incorrectly. However, if you use it and maximize its potentials, you will see marked improvements in your practice and execution of the Search Engine Optimization.

It’s your friendly neighborhood, googleman! 🙂

Robots.txt file Defined

If you will try to access a certain, you will be brought to a page generating a long list of the website’s directories. This list of directories corresponds to the search engines that a certain website owner wants to flee from or skip. But, in the process of editing your robots.txt file, you should proceed with due caution. If you won’t be careful enough, you might put some information that can hurt your online marketing efforts and can result to a downgrade when it comes to rankings.


Some Do’s and Don’ts to Keep in Mind

The following are the things that you should remember when dealing with robots.txt:

  • You are definitely allowed to stop the search engines from the process of indexing your site directory for contents that are duplicated. This way, the search engine will only index a single version of the content that you are offering.
  • The following are the directories that the search engine should not be indexing. These directories include /wp-admin/, /cart/, /cgi-bin/, and /scripts/. These directories may contain data that should not be indexed.
  • Double-check your robots.txt to see if there is no hindrance when it comes to the process of indexing the major content of your website.
  • Include files that have telephone numbers, email addresses, and other sensitive data in the list of those that you want to disallow the search engine from being subjected to the process of indexing.


If there are things that you should do, the following are the things that you should not do when using robots.txt:

  • Comments should not be included or used in any way in the robots.txt file.
  • Do not add the “/allow” command in robots.txt. No such command exists.
  • Avoid listing all the files in your robots.txt file. This will only open the possibility of making other people find files that you want to keep secret.


When in Doubt, Consult an Expert

There is no harm in getting an expert’s opinion when using robots.txt file. It is only this way that you can be sure and confident that you are doing every single aspect correctly. This way, you will no longer have to think of bigger problems like problems in your ranking in a certain search engine. Do every single step as precisely as you can so that you won’t regret anything in the end.







[REVEALED] 101 SEO Strategies in the Philippines
Download Free eBook
Check out the latest tricks and techniques in SEO with Paul Agabin's free report about the 101 SEO Strategies in the Philippines. Updated for 2013.

Paul Agabin is the Founder and CEO of Wooka Interactive. He is an internet marketer based in the Philippines which deals with varied topics such as local industry news, seo strategies, content marketing, inbound marketing, social media management, website reviews, and the like. You can contact him at 0917-5069839.

Share This Post

Recent Articles

Leave a Reply

You must be Logged in to post comment.

© 2020 Paul Agabin. All rights reserved. Site Admin · Entries RSS · Comments RSS
Powered by WordPress · Designed by Theme Junkie