I know I should be grateful that they’re indexing my sites frequently. That would mean that we’re getting more exposure and more people visit the site.
But should their indexing take up so much bandwidth?! I mean for one subdomain alone, we got 14gb of used bandwidth within 7 days. Isn’t that insane? Shouldn’t our bandwidth be reserved for people who actually have an interest on our site, rather than robotic machines that only report back to google about our pages?
Bandwidth doesn’t come for free; in fact, it’s rather expensive if any exceeded bandwidth is not part of your hosting plan. How expensive? Just imagine that a few months ago, this blog racked up 100gb of bandwith, and yes, just for this little blog alone.
Now our hosting package only allows for 20gb per month and that’s more than enough for us. Unfortunately, we didn’t expect the googlebot to ransack it.
How about using the robots.txt file? People say it works.
It doesn’t. I tried it for several months and it did stabilize my bandwidth towards the middle but in the end my bandwidth is still sky-rocketing.
Now if only I knew how to stop googlebot from sucking up our bandwidth and yet not lose our page ranking in google. T_T