Search Engine Robots and Bandwidth

Being indexed by probably a good news. When I look at the for this , it seems that there are many visitors come to this site from search engines. Okey, my blog pages are indexed by search engine. I notice that search engine robots/spiders consume my hosting bandwidth. I have no problem with this since I have enough bandwidth.
You can see a sample of statistics in the below (the top three). I think it’s pretty big.
Search Engine Robots Stats #1
Search Engine Robots Stats
Yahoo!’s Slurp (search engine crawler owned by company) eats the most bandwidth. MSNBot and Googlebot come after that. It does not mean that Yahoo!’s search engines brings the most visitors to my site. does. I do not have plan of blocking those robots, even the process is so easy using robots.txt. Most search engines obeys the Robot Exclusion Standard. So, robots.txt will be just effective to block unwanted search engine robots.

2 replies on “Search Engine Robots and Bandwidth”

Kus, I was in a period when I have a very limited bandwidth. But, I think my current webhosting bandwidth is more than enough.
Well, my site traffic is not high… less than 3 Gb/mo.
So, that’s how I define the word: “no problem with bandwidth”. :)