Comments on: Wanted: Hard drive boys for our new ginormous data center

By: Andries Louw Wolthuizen

Andries Louw Wolthuizen — Mon, 05 Jan 2009 00:07:49 +0000

@Mike:
And still, that is how Google is doing it. Harddrive failure? Just let it. Server failure? We don’t care. A whole rack down? So, what?

They replace the whole thing if there are enough racks down. Did you know, that Google even is using today, some old Pentium 2/3′s?

Expensive? Do you know how expensive it is, to let some guys in your datacenter 24/7 replacing spare parts?

Check this video from Google: http://www.uwtv.org/programs/displayevent.aspx?rID=2879 (Very interesting video!!)

By: Mike

Mike — Fri, 26 Dec 2008 01:09:22 +0000

Fuzzball: You’d be surprised, on average all brands perform/last just as well. I think a lot of love for Seagate came from their 5 year warranty, but this is being reduced to 4 year IIRC to cut costs.

aka: That sounds like the most expensive solution to an easy problem. A complete rack would cost a lot of $ having even 10% of a rack sitting broken/unused is such a waste of space/resources.

By: Fuzzball

Fuzzball — Tue, 16 Dec 2008 22:57:08 +0000

I suggest staying away from IBM/Hitachi DeathStar, I mean, DeskStar HDDs. Seagate is probably your best bet.

By: bronius

bronius — Mon, 15 Dec 2008 21:52:50 +0000

Maybe datacenters can implement an intelligent BURN-E solution like on WALL-E (slightly more interesting than a tape jockey)

By: aka

aka — Mon, 15 Dec 2008 10:06:00 +0000

Google (and probably anyone running something of that size) don’t replace hard drives. They ignore all failures and leave the dead hardware in place until a sufficent % of machines are dead in a particular sector then just rip them all out and replace them with new machines.

Apparently Microsoft are using an even broader approach for their new data centers – everything will be done at rack level – new racks full of machines will arrive from the hardware vendors ready to plug in and run, then left untouched until enough has died to make it worth replacing the whole rack.

By: Steph

Steph — Fri, 12 Dec 2008 19:39:04 +0000

I think you’re right. At 1 failure every 45 minutes, you’d have just enough time to find the physical machine, pull out the HD, replace it, and restore a disk image onto it as another drive would fail.

Especially for a large system with 400,000 drives. That’s a lot of physical space. You could almost have races to see who could get to (or even find) the faulty machine first

By: Michael Biven

Michael Biven — Fri, 12 Dec 2008 17:57:33 +0000

Sounds a lot like the “tape jockeys” we would call to mount the next set of tapes.

Michael

By: Donald

Donald — Fri, 12 Dec 2008 14:55:22 +0000

We’ll probably do what we’ve always done with repetitive tedious tasks. Batch em. If there’s a 16 drive failure rate/day. Place so many hot spares or hot nodes around to sustain the center for a week and batch replace them once a day/week/month.