Block Spider

[GoPantheoN]

I want to block baidu spider, i have googled and i got results , but it seems baidu spider doesnt obey robots.txt and also i found a htaccess block for that, but still i am not sure whehter it will block it or not...

Do you guys have any suggesstion ?

GoPantheoN Reviewed by GoPantheoN on 10th May 2010. Block Spider I want to block baidu spider, i have googled and i got results , but it seems baidu spider doesnt obey robots.txt and also i found a htaccess block for that, but still i am not sure whehter it will block it or not... Do you guys have any suggesstion ? :) Rating: 5

[Gavo]

Three diferant methods here with pro/con`s

http://www.simplemachines.org/community/index.php?topic=350439.0

is it really using that much bandwidth that you have to block it?

[GoPantheoN]

ya i checked the smf forums.... nah it doesnt use a lot, but its always on my site and the worst thing is it never indexes it...

[Gavo]

check carefully in the thread, it may not be baidu but a fake bot using that useragent.

if so you would need to check the IP in your logs and then ban it.

[GoPantheoN]

ya may be thats the only way

[Mr Happy]

Badu does follow robots.txt laws and will not index your site if you don't want it to but you have to remember it can take several weeks for it to update it's search results so can take a while to remove your site.

Why exactly do you want to block Badu? Putting an exact figure on it's size is hard as it's very secretive and in a country where information is hard to get but most believe it's the second biggest search engine after Google. That makes it bigger than Yahoo and Bing which means your rejecting a huge potential market.

[GoPantheoN]

^^ i would be glad if it get indexed but it doesnt and all the time there are atleast 5 baidu spider...... so just wanted to know they are really baidu spiders or other bots....

[Lock Down]

Easiest way is to get the ip and trace it back to owner. Network-tools.com will do that for you.

[Mr Happy]

oh right. Ya I used to do that for Google. I had a script that scanned forums and gained access pretending it was Google. It's possible that it's not a real Badu spider. As Lock Down said if you just get any of the IP's which will be in your logs it should go back to the Badu website. If not then you know someone is crawling your site pretending it's Badu. This is sorta rare though and I've never heard of anyone pretending to be Badu. It's obviously better to pretend to be Google. Generally Badu doesn't index much though and it's easily possible for them to have 5 of their spiders on your site.