Being indexed by search engines probably a good news. When I look at the statistics for this site, it seems that there are many visitors come to this site from search engines. Okey, my blog pages are indexed by search engine. I notice that search engine robots/spiders consume my hosting bandwidth. I have no problem with this since I have enough bandwidth.
You can see a sample of website statistics in the image below (the top three). I think it’s pretty big.


Yahoo!’s Slurp (search engine crawler owned by Yahoo! company) eats the most bandwidth. MSNBot and Googlebot come after that. It does not mean that Yahoo!’s search engines brings the most visitors to my site. Google does. I do not have plan of blocking those robots, even the process is so easy using robots.txt. Most search engines obeys the Robot Exclusion Standard. So, robots.txt will be just effective to block unwanted search engine robots.