I Hate Google
I know I should be grateful that they’re indexing my sites frequently. That would mean that we’re getting more exposure and more people visit the site.
But should their indexing take up so much bandwidth?! I mean for one subdomain alone, we got 14gb of used bandwidth within 7 days. Isn’t that insane? Shouldn’t our bandwidth be reserved for people who actually have an interest on our site, rather than robotic machines that only report back to google about our pages?
Bandwidth doesn’t come for free; in fact, it’s rather expensive if any exceeded bandwidth is not part of your hosting plan. How expensive? Just imagine that a few months ago, this blog racked up 100gb of bandwith, and yes, just for this little blog alone.
Now our hosting package only allows for 20gb per month and that’s more than enough for us. Unfortunately, we didn’t expect the googlebot to ransack it.
How about using the robots.txt file? People say it works.
It doesn’t. I tried it for several months and it did stabilize my bandwidth towards the middle but in the end my bandwidth is still sky-rocketing.
Now if only I knew how to stop googlebot from sucking up our bandwidth and yet not lose our page ranking in google. T_T
You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
I just did some random reading and found out that spiders/crawlers these days don’t even honor the robots.txt exclusion codes anymore. However, now that you’re on a Linux server, you should be able to use .htaccess freely w/o much restrictions anymore. This might be useful for Kachou-sama.. ^^
[dive into mark] How to block spambots, ban spybots, and tell unwanted robots to go to hell
Ganbatte. ^^