Shared storage for film & TV - the next podcast

Hugh and Phil are joined by Rupert Watson from root6 to talk about SANs, NASs and shared storage for film and TV - see the wiki

Reliability of modern hard drives

A couple of things have got me thinking about the reliability of hard drives.

1. The eldest boy asked me about RAID levels and what extra reliability they bring.
2. I sold an old 500gig SATA drive from my media center to a guy on eBay and he's quibbling because he's found three bad sectors

Although RAID has bought tremendous increases in the speed and reliability of storage systems there are some basic engineering considerations around combining many unreliable parts into a whole. The Mean Time Between Failure for modern hard-drives (MTBF) is in the order of 30,000 hours. MTBF is a complicated field but if you look at the figures provided by manufacturers then they assume a Gaussian distribution with the 30,000 hours figure at the peak; a few drives will fail after a day, a few will last 100,000 hours, but the bulk will fail around 30,000 hours (around three and a half years). It's why I say to people "...it's not if this drive will fail, rather it's when!"
So - with this in mind I decided to find out the MTBF of a rack of ten drives, each with a MTBF of 30k hours.The formula is;

So, if you stick 30k hours in for D1 through D10 you find the MTBF for the entire system is only 3k hours - less than twenty weeks! In fact it's worse than this as the PSU in the enclosure and the RAID management card will have MTBFs to take into consideration as well. In fact if you ask any broadcast engineer "how often are you replacing drives in RAID arrays" and they'll tell you it's nearly a weekly occurrence for any decent sized facility, and this is why! Although it's been nearly a decade since I ran engineering in a good-sized facility I was often uneasy about how often RAID enclosures failed (loosing all of the media, which is what happens with a RAID-0 striped set). I had that 30k hours figure in my head but never calculated the system MTBF.

RAID-1 (mirrored drive sets), RAID-5 (distributed parity) and RAID-6 (double distributed parity) along with some of the advances that better file systems bring (Isilon's OneFS and Linux's ZFS) mean that failure of a single drive is no longer the disaster it once was, but if someone doesn't notice a drive has died OR (heaven forbid) a second drive dies whilst replacing the first you're stuck. Don't forget a lot of the chassis we're installing now have a couple of dozen drives (Isilon's NL36 nodes - done of a few of them recently) and so MTBF is even worse than the 3k hours above (however, server-grade SAS or Fibre Channel drives are considerably more reliable than domestic-grade SATA drives).

We also know that modern multi-terrabyte drives pack data so densely (similar sized platters to the first 10Mbyte drives of yesteryear, but hundreds of thousands more bits/mm-sq. of disk surface) that the disk's error correction/error recovery system is working flat-out all the time. The newer 2Tbyte drives have a Viterbi decoder to try and statistically extract correct data from the very noisy signal coming off the drive's heads. Additionally the drive's SMART system has to know about the number of bad sectors due to manufacturing imperfections (contained in an EPROM-based p-list table) as well as the number of grown bad sectors (which get swapped out as per the g-list). Spinrite is the best utility I've found for drive maintenance/recovery as it forces the SMART system to pay attention to bad sectors and swap them out. In a Hitachi Ultrastar 7k RPM, 500gig SATA drive there are 10,000 hidden spare sectors on the drive (each sector is only 4k bytes in size) to allow the drive to swap-out failed sectors. According to the data sheet Hitachi would replace a new drive if it had more than twenty bad sectors from the factory - any less and they regard it as being well inside manufacturing tolerances. If you Google "how many bad sectors is acceptable for a new drive" you'll find hundred of IT experts claiming that no bad sectors are acceptable. I don't know what planet they live on, presumably one where quantum mechanics operates in a different manner and electrons don't bump into each other leading to electrical noise!
Oh - the eBay guy; he ran a utility on the drive I sold him that reported three bad sectors. He asked me for a refund. Apparently a second-hand disk drive should carry a better guarantee than that provided by the factory when new!

Whole drive encryption and disk performance

I've been interested in volume encryption for a while. TrueCrypt ticks all the boxes. Being a piece of security software it should be open source (you don't want any back-doors after all). One thing peaked my interest on a recent edition of Security Now! - Steve Gibson discovered that booting Windows off a system partition that has the TrueCrypt driver installed gives a system that has a significant improvement in disk performance;

...so I wrote a little batch file using that EndTimer tool and the Windows defrag and Vopt and Windows defrag. I ran those three in sequence. With no encryption, Windows defrag took 8 minutes and 35.765 seconds. Vopt took 4 minutes and 31.046 seconds. And then a final Windows defrag took 1 minute, 54.765 seconds. Okay, so just look at the first number, 8 minutes and 35 seconds. I did it; I did it again. That is, I restored the image, ran the script again, and it was 9 minutes and 1 second. So, you know, about 8 minutes and 45 seconds on average. And the difference are just we're doing a lot of head-seeking. And so where the disk's rotation happens to be is going to affect timing a little bit.
They say on their web page that they've got 100 percent pipelining of some sort. Apparently once upon a time it was too slow, and boy did they fix it.

I intend to start using TrueCrypt - so I'll blog about it when I've got it figured.

Hard Drive Reliability

I love the Security Now! podcast - episode eighty-one this week was a corker. Leo and Steve discuss the distressing results and implications of two recent very large population studies (more than 100,000 drives each) of hard drive field failures. Google and Carnegie Mellon University (CMU) both conducted and submitted studies for the recent 5th USENIX conference on File and Storage Technologies.
Since I have a modest amount of experience with hard drive reliability I thought I'd drop Steve the following;

For the last twenty years I've been working in broadcast engineering and the track of my career has mirrored the uptake of commodity computers over bespoke television equipment.
I had a couple of points - one interesting and one informative that I thought you might enjoy;

• In the mid-nineties the whole industry was switching over from editing video-tape to cutting shows on workstations. Consequently large and fast hard-drives were needed and Micropolis (now out of business) was one of the manufacturers of choice for most TV facilities. They'd launched a nine gig model (big full-height 5 1/4¨ device!) that the company I was working for was buying in number - even at $2,500 a pop they were thought to be good value! After about six months a number of these drives started to fail. The manufacturer had given us a SCSI utility to see which sectors had failed and it appeared to be the same ones on each drive. In a couple of cases we did a low-level format of the drives (which mapped out the bad sectors) and continued to use them. Those drives then showed problems and when analysed their sectors were clearly failing in very similar patterns. In the end the representative from Micropolis told us that in the case of that series of drives the lubricant they used would leak from the spindle-bearings and spread out across the platters, getting progressively more spread from the centre of the disk.


• The company I currently work for specialises in editing systems, particularly for film and high-definition television. In the case of HD the data rate off of the videotape is either 1.48 or 3 gigabits per second (unlike the domestic HDV format that manages to compress the video to a paltry 18 megabits per second!). In the case of film (either from a digital film camera or telecine film scanner) the data rate can be much higher. The upshot of all this is that the stand-alone storage systems and SANs (storage area networks) have to stripe many drives together to achieve the required through-put. We are very used to having ten or more drives with data striped across them and the dirty little secret we shy away from is that the mean-time between failure of a ten-way drive set is only one tenth of a typical drive. Consequently our tech-support department is always trying to resurrect dead fibre-channel drives. We tell (or even try and bully) customers into keeping backups but with many terabytes of data it is a hard thing to enforce. This is why mixed striped/mirrored drive sets are becoming popular. Anyhow - one of the things I have found to be useful in temporarily reviving a dead drive (and I've done it maybe a dozen times) is to freeze a disk. It might sound crazy but it you consider that the most common reason for mechanical failure in a drive is the bearings becoming loose and the drive spinning eccentrically you can see the reason. The cold temperature tightens everything up as it shrinks and (temporarily) allows the thing to work at specification. The only option is to clone the drive and then throw the suspect one away.

More disk drives in freezers! - Yes, it worked again - got MTV out of trouble by chilling a dead 73gig SCSI drive - see my origional post here.

I'd heard of people putting a nearly-dead hard drive in the freezer to temporarily revive it but always believed that it was geek folk-law. Imagine my surprise last night when a drive that would stay alive for about four minutes before starting to click and whirr (and become unavailable to Windoze) stayed alive for half an hour after a couple of hours nestling amongst the peas and ice-pops - I was able to clone it off to another drive and all is well!