Halls of Tolkien


Well folks, it's been a great run, but I've had to bulldoze down the Halls of Tolkien. Why? Well, mostly because of about 5 or 6 sites which have really stupid web crawlers that download the entire archive 3 sometimes 4 times a month. At the extreme that is 200megs * 6 * 4 = 3.6 gigs of traffic just from those sites. And with the movies coming out I get at least that much traffic again. It's enough that we're seeing a hit during peak hours on our work line so I am voluntarily remove the archive for now. And yes, I have tried many different methods of removing the crawlers, but they just change IPs and come back the next month. It's very disheartening that a few people with really crappy programs (how hard is it to compare timestamps with a HEAD request... sheesh) can ruin something so many of you come to see. I wish there was a way I could keep the system up and running, but it's just not feasible.

My sincerest apologies,

Scott Powers