Disclosure: We webhostingdisk.com owners receive compensation from web hosting companies for REFERRALS.

WordPress KB Robots.txt plugin

WordPress KB robots.txt is another great plugin for your wordpress blogs. If you already don’t know what a robots.txt file is, now you can learn about it here. Robots.txt file is small text file in which you provide the details to search engines as to what they have to index on your site and what they shouldn’t. Normally, all websites and blogs should use a robots.txt file, because it is where the search engines get the instructions regarding which of your sites links, files, pages etc are to be indexed by them. If you don’t use this file on your website or blog, the search engines will index just about anything that they find on your site.

Almost all the major search engines respect the robots.txt file and index the sites or blogs according to the details given in this file. The details provided in a robots.txt file look like this.

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /category
Disallow: /tag
Disallow: /author
Disallow: /trackback
Disallow: /*trackback
Disallow: /*trackback*
Disallow: /*/trackback
Disallow: /*?*
Disallow: /*.html/$
Disallow: /*feed*

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /* –(the instructions of the file varies from site to site depending on what the owner of the site wants indexed and what doesn’t want indexed)–

Here User-agent is addressed for a search engine. You can disallow a particular search engine from indexing your site by placing the name of the search engine bot here. I this example the “*” after User-agent means you allow all search engines to index your website. The term “Disallow” followed by any other terms shows that you don’t want those parts of your website to be indexed.

Now back to the original topic, the KB robots.txt plugin for wordpress blogs. By installing this plugin and activating it you can simply go to the plugin editor area and can simply type your instructions in the provided area and the plugin does all the rest for you. Remember, almost all the search engines look for robots.txt plugin only either after main domain or subdomain. That means they only look for the file at either after example.com/ or subdomain.example.com/. So, if your blog is installed in a sub directory like example.com/blog/ and you placed the file at example.com/blog/robots.txt, then the file is of no use to your site. The search engine won’t follow the instructions of the file and index whatever their bots or crawlers find on your blog.

Disclosure: We webhostingdisk.com owners receive compensation from web hosting companies for REFERRALS.

Best Shared Web Hosting Companies:

1. HostGator 2. BlueHost 3. HostMonster 4. WebHostingHub 5. DreamHost 6. Arvixe 7. HawkHost

Best Reseller Web Hosting Companies:

1. HostGator 2. GreenGeeks 3. HawkHost

Best VPS Server Web Hosting Companies:

1. HostGator 2. LiquidWeb 3. KnownHost 4. WiredTree

Best Dedicated Server Web Hosting Companies:

1. HostGator 2. LiquidWeb 3. SingleHop 4. WiredTree

Best Cloud Server Web Hosting Companies:

1. RackSpace 2. LiquidWeb 3. DigitalOcean 4. StormOnDemand
You need a domain name and a Web Hosting account to start your own website like this. Here are some Web Hosting Reviews of various companies which offer great Web Hosting services.

We recommend Hostgator web hosting be it Shared web hosting or Reseller web hosting or Virtual Private Server (VPS) web hosting or Dedicated Server web hosting
Disclosure: We webhostingdisk.com owners receive compensation from web hosting companies for REFERRALS.

Comments are closed.