Reliability of modern hard drives

A couple of things have got me thinking about the reliability of hard drives.

1. The eldest boy asked me about RAID levels and what extra reliability they bring.
2. I sold an old 500gig SATA drive from my media center to a guy on eBay and he's quibbling because he's found three bad sectors

Although RAID has bought tremendous increases in the speed and reliability of storage systems there are some basic engineering considerations around combining many unreliable parts into a whole. The Mean Time Between Failure for modern hard-drives (MTBF) is in the order of 30,000 hours. MTBF is a complicated field but if you look at the figures provided by manufacturers then they assume a Gaussian distribution with the 30,000 hours figure at the peak; a few drives will fail after a day, a few will last 100,000 hours, but the bulk will fail around 30,000 hours (around three and a half years). It's why I say to people "...it's not if this drive will fail, rather it's when!"
So - with this in mind I decided to find out the MTBF of a rack of ten drives, each with a MTBF of 30k hours.The formula is;

So, if you stick 30k hours in for D1 through D10 you find the MTBF for the entire system is only 3k hours - less than twenty weeks! In fact it's worse than this as the PSU in the enclosure and the RAID management card will have MTBFs to take into consideration as well. In fact if you ask any broadcast engineer "how often are you replacing drives in RAID arrays" and they'll tell you it's nearly a weekly occurrence for any decent sized facility, and this is why! Although it's been nearly a decade since I ran engineering in a good-sized facility I was often uneasy about how often RAID enclosures failed (loosing all of the media, which is what happens with a RAID-0 striped set). I had that 30k hours figure in my head but never calculated the system MTBF.

RAID-1 (mirrored drive sets), RAID-5 (distributed parity) and RAID-6 (double distributed parity) along with some of the advances that better file systems bring (Isilon's OneFS and Linux's ZFS) mean that failure of a single drive is no longer the disaster it once was, but if someone doesn't notice a drive has died OR (heaven forbid) a second drive dies whilst replacing the first you're stuck. Don't forget a lot of the chassis we're installing now have a couple of dozen drives (Isilon's NL36 nodes - done of a few of them recently) and so MTBF is even worse than the 3k hours above (however, server-grade SAS or Fibre Channel drives are considerably more reliable than domestic-grade SATA drives).

We also know that modern multi-terrabyte drives pack data so densely (similar sized platters to the first 10Mbyte drives of yesteryear, but hundreds of thousands more bits/mm-sq. of disk surface) that the disk's error correction/error recovery system is working flat-out all the time. The newer 2Tbyte drives have a Viterbi decoder to try and statistically extract correct data from the very noisy signal coming off the drive's heads. Additionally the drive's SMART system has to know about the number of bad sectors due to manufacturing imperfections (contained in an EPROM-based p-list table) as well as the number of grown bad sectors (which get swapped out as per the g-list). Spinrite is the best utility I've found for drive maintenance/recovery as it forces the SMART system to pay attention to bad sectors and swap them out. In a Hitachi Ultrastar 7k RPM, 500gig SATA drive there are 10,000 hidden spare sectors on the drive (each sector is only 4k bytes in size) to allow the drive to swap-out failed sectors. According to the data sheet Hitachi would replace a new drive if it had more than twenty bad sectors from the factory - any less and they regard it as being well inside manufacturing tolerances. If you Google "how many bad sectors is acceptable for a new drive" you'll find hundred of IT experts claiming that no bad sectors are acceptable. I don't know what planet they live on, presumably one where quantum mechanics operates in a different manner and electrons don't bump into each other leading to electrical noise!
Oh - the eBay guy; he ran a utility on the drive I sold him that reported three bad sectors. He asked me for a refund. Apparently a second-hand disk drive should carry a better guarantee than that provided by the factory when new!

Colourimetry 2 - Calibrating monitors for TV

After the intro to colourimetry Hugh and I talk about calibrating monitors for film and TV use. Find it on iTunes, vanilla RSS, YouTube or the show notes website.

TV Colour - the next podcast

Hugh and I lay the groundwork for the next podcast about monitor calibration. This episode concerns perception of colour in TV. Find it on iTunes, vanilla RSS, YouTube or the show notes website.

Speeding up HTTP

The big overhead with protocols that run on top of TCP/IP is the number of connections they open. A modern web page has many different kind of assets (HTML, scripts, Java, GIFs, JPEGs, etc etc) from many places (the primary domain, adservers, Google+ buttons etc etc) such that when you load the front page of facebook.com you may well have opened and closed a thousand TCP connections to a dozen servers. It's amazing that it works at all, but when you consider that each connection had to not only do the three-way TCP handshake, but it also had to run TCP flow-control starting slowly and ramping up packet rate until it started to drop packets and then dropping back. When Mr Burners-Lee wrote his original http server in the early nineties there is no way he could have anticipated how the web would grow.

It seems there are several approaches to optimising this - for the past ten years there have been various attempts to re-define TCP; MUX, SMUX, and SST protocols were valiant efforts that died on the vine because they essentially break how the infrastructure works. Whatever comes after http has to work over all the same IP v.4 routers, switches and proxies. In the last year I've become aware of two projects that work and don't require funky routers etc.

1. Amazon Silk - this is embodied in the browser that comes on the colour Kindle. Essentially it is a mega-proxy that not only collects all the assets for the page but re-renders pictures etc for the smaller screen and sends the whole lot in a stream. So one connection allows the whole pre-rendered page to arrive with the assets from Double-Click and Google (and any other third-party elements in the page) pre-collected for you by Amazon. It runs of their EC2 platform and does depend on them providing the service.
2. SPDY is a Google-sponsored project that doesn't replace http but optimises it. By employing pipelining (i.e. keeping a TCP connection open for all the assets from one domain), compressing headers and marking those headers that don't change so they don't need to be re-sent it speeds up the browser by around three-fold. Further speed increases come if the web-server is able to collect the third-party assets and deliver them over the same pipeline.

As ever Steve Gibson has covered these very comprehensively in Security Now! - SN320 for Silk and SN343 for SPDY.

What's in your rucksack?

Phil and High go over the basic set of tools and test kit that broadcast engineers on the hoof need! Find it on iTunes, vanilla RSS, YouTube or the show notes website.

Synchronous ain't always best!

Engineers have been trained since time immemorial that with multiple sources of video the best thing is that they are always locked together and timed to a common reference - ideally station black & burst (or TriSyncs nowadays). The reasons for this are numerous, but a couple;

1. Studios cameras into a vision mixer have to be locked to achieve clean cuts between the cameras - it would look rubbish if you had a frame roll every time you cut between sources. The same is true of sources into a continuity suite etc.
2. In the case of Avid for the longest time you had to make sure the VTR was locked to the same reference as the Media Composer (all the way from v.7 ABVB systems in the mid-90s to the last revision of Adrenaline in 2007!) otherwise the audio and video capture portions of the machine would free-run WRT each other and within minutes you'd be loosing frames of sync.

So - I had a coffee and chat with a pal this morning who works for a big facility that has won an archive digitising project and they are using the BlackMagic DeckLink 4K card to allow them to ingest 4 x DigiBeta tapes at once. The capture software is ToolsOnAir and they found that after the first clip was captured the second clip would be a frame out of sync, and progressively worse after that - unless you re-booted between captures! It turns out that if the four VTRs are allowed to free-run then you don't get the problem. Perhaps processing the vertical syncs places a burden on the card/software and if it happens simultaneously on all four inputs trouble ensues?!

It reminded me of a situation with a big broadcaster who was distributing their regional variations over Astra on a single multiplex. The stat-mux was very unhappy with material that was (for the most part) identical; only the ad-breaks differed. Most video cuts occurred at the same time across all six SDi feeds. The solution was to apply a two frame delay between all the sources (so o/p 6 was now 12 frames late WRT o/p 1).

Mains 101 - part 2 / Postscript

Hugh and Phil wrap up the previous mains safety podcast with some corrections and photos from Nigeria! Find it on iTunes, vanilla RSS, YouTube or the show notes website.