Search Engine Robots and Bandwidth

Being indexed by search engines probably a good news. When I look at the statistics for this site, it seems that there are many visitors come to this site from search engines. Okey, my blog pages are indexed by search engine. I notice that search engine robots/spiders consume my hosting bandwidth. I have no problem with this since I have enough bandwidth.
You can see a sample of website statistics in the image below (the top three). I think it’s pretty big.
Search Engine Robots Stats #1
Search Engine Robots Stats
Yahoo!’s Slurp (search engine crawler owned by Yahoo! company) eats the most bandwidth. MSNBot and Googlebot come after that. It does not mean that Yahoo!’s search engines brings the most visitors to my site. Google does. I do not have plan of blocking those robots, even the process is so easy using robots.txt. Most search engines obeys the Robot Exclusion Standard. So, robots.txt will be just effective to block unwanted search engine robots.

2 replies on “Search Engine Robots and Bandwidth”

  1. Kus, I was in a period when I have a very limited bandwidth. But, I think my current webhosting bandwidth is more than enough.
    Well, my site traffic is not high… less than 3 Gb/mo.
    So, that’s how I define the word: “no problem with bandwidth”. :)

Comments are closed.