robots.txt for your vBulletin forum

[desiboy]

robots.txt for your vBulletin forum

Well many of us might know what this does..but still there are some web masters who don't know about robots.txt and its importance .

Importance of robots.txt in short

When an search engine crawler (spider ) visits your site, it first looks at a file on your site called robots.txt and that file tells the spider, which web pages of your site should be crawled and indexed and which shouldn't be indexed.

so what you have to do is create a new .txt file and rename it as robots.txt
and paste the code which is included below to it and upload it to your server .

If your forum in the main directory

ex :

 http://www.MYFORUM.COM/

User-Agent: *
allow: /
Disallow: /cgi-bin/
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php

Case 2 : if its in a subdirectory / folder
Ex :

http://www.MYSITE.com/forum

User-Agent: *
allow: /
Disallow: /cgi-bin/
Disallow: /forum/ajax.php
Disallow: /forum/calendar.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/global.php
Disallow: /forum/image.php
Disallow: /forum/inlinemod.php
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/member.php
Disallow: /forum/memberlist.php
Disallow: /forum/misc.php
Disallow: /forum/moderator.php
Disallow: /forum/newattachment.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showpost.php
Disallow: /forum/showgroups.php
Disallow: /forum/subscription.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php

Also I have found that some web masters add their admincp directory to the robots.txt file which is not at all recommended , because it will tell the Hackers where your admincp is located.

We can even add sitemap to the robots.txt file

Sitemap: http://www.YOURFORUM.com/sitemap.xml

Tut By DesiBoy !
Hope it helps

Thanks For Reading !!

desiboy Reviewed by desiboy on 17th Sep 2009. robots.txt for your vBulletin forum robots.txt for your vBulletin forum Well many of us might know what this does..but still there are some web masters who don't know about robots.txt and its importance . Importance of robots.txt in short so what you have to do is create a new .txt file and rename it as robots.txt and paste the code which is included below to it and upload it to your server . If your forum in the main directory Rating: 5

[Flash]

Very nice info, thanks for sharing

[desiboy]

Very nice info, thanks for sharing

Welcome

[Dark-BlaZe]

Thank you!

[Luke]

This is not your tutorial also you should not disallow bots from visiting your archive.

[Last Word]

Thank you!

[desiboy]

This is not your tutorial also you should not disallow bots from visiting your archive.

I wrote it..

[weast]

Thanks i will give this a try