Wow that Googlebot can get hungry. I admin a site and forum for my 4WD club and the google crawler has been sucking up the bandwidth big time. Over 60% of the bandwidth for the month is from Google. If you do a search for "excessive bandwidth usage by googlebot" it turns out there are a few people who are having the problem.
The forum is phpBB so there is a lot of cruft around the messages which has to get sent to Google with every page that they request, even though the content is only small. Some clever person should write some code that determines if the browser agent is a bot and only return a very simple version of the page with the key content and some simple links for it to follow.
Well hopefully it will finish indexing soon ...
No comments:
Post a Comment