New grading room - a few notes.

I've been building a Baselight room recently - the monitor is a Sony X300 UHD/HDR display (expensive!) with an AKA custom grading desk. Here are some photos;

All courtesy of my colleague & pal Graham McGuinness

1. The X300 is a heavy beast at 16Kg and if you want to hang it off a monitor arm - the only one I found suitable was from Novus; we've bought a few things from them recently and they are a very high quality manufacturer. The TSS-range is here but we tend to purchase from our friends at MW/ThinkingSpace as they hold stock and are super-helpful.
2. The X300 out-of-the-box was a tad hot in 2k mode (they distinguish between rasters as 2k/4k - not HD/UHD; just to keep the DCI film snobs happy!). I was expecting the EBU recommended 100Cd/m² but was 125 Cd/m² and a bit blue-in-the-whites, but not enough to bother adjusting unless you were pointing a probe at it! So - this one is correct for rec.709 but the customer (hasn't yet) decided what their wide-colour-gamut and high-dynamic-range workflows will be.
3. I had my friend and carpenter Tony Andrews (Andrews Construction - for all your carpentry and building requirements!) built the lightbox. In the past I've talked about Crown Plain Grey 5574 Matt Emulsion as being suitable for the backwall, but that's now a discontinued paint and so an excellent match is Delux's "CN8 Grey Steel 3".
4. Display Port monitors and Baselight-1 - so running Baselight v.6 on an HP Z840 means you have a four display-port output nVidia card. Historically they would prefer you to use DVI monitors, but this is 2016 and a pair of HP Dreamcolor 27" monitors are being extended via Amulet (my favourite KVM extender). In this case you need to have Filmlight provide a new version of xorg.conf for X to see higher order monitors as primary and secondary GUI displays. 
5. The Blackboard2 control panel is fed over a DVI adaptor hanging off port 2 of the nVidia. Be warned! It runs at a very funny raster (3460 pixels wide!?) at only 15 FPS. It doesn't respond to EDID requests and so consequently you have to use a very simple extender to run it to the suite - no KVM-over-IP, just a fibre balun (I used one of these).
6. Similarity the USB for the Blackboard declares itself as an HID device but then ignores HID-probes, so again, only use a balun to extend the USB, no over-IP.

Memories of 1999 at Resolution Soho

I'm working the weekend in a building that has been a TV facility for the last couple of decades. I had the privilege of being Chief Engineer here during 1998-2003 when it was the Soho branch of the Resolution Post Group. 

Having opened a few cable-risers it looks like this place is largely the same (aside from the decoration) and in many of the rooms there is still equipment I either purchased or had the pleasure of fixing or calibrating.

Custom talkback box; this is a little thing I home-brewed around 2000 - it's just a talkback box with a cue-light button on it. Still in use though!

This wooden-surround for the Wacom tablet for a Softimage DS workstation had an little audio switcher built into it (selects between DS, VTR return and CD). The grahics tablet was originally hidden inside the polycarbonate sheet so you couldn't see it.

This monitor shelf was a bit of a mission; galvanised scaffolding poles are held in a triangle shape by chandler's wire & connectors. The tear-shape glass shelf sits on top of that (the ends of the poles have rubber caps) and the monitor's weight stresses the structure to keep it solid. I thought it would collapse in a week but here it is a decade-and-a-half later!

Also notice the PMC TB1 loudspeakers; most of the rooms here still have them. They were all purchased between 1999 and 2001 and are all still sounding good.

Back in 1999 there were no WiFi walkabout 'phones or office-grade DECT handsets. So - I bought the best domestic ones I could find and then spent a few days seeing where the best place was for the base-stations for maximise coverage for the runners, engineers and tape-ops (we still had those!). It turned out to be an external wall at the rear of the building. They're still there! (Although no longer used).

STOP PRESS!

My good pal Malcolm Baldwin reminded me that he'd made the talkback box - he commented that he'd never really liked it but I thought it was splendid. Just shows that some engineers have higher standards than other...!

OLED TVs in grading rooms?

I was over at a big facility this morning in one of their grading rooms setting up a Sony BVM-E250 OLED display for grading (the usual - rec.709 80Cd/m² - BBC-style) and as even the monitor was only a bit out; If I didn't have the probe out and ready I might not have bothered!

However, they also wanted me to see if I could match an LG-55EC9300 55" OLED tele. It was one of those new jobs that has a subtly curved screen. I'm convinced that's just for strengthening as the panel is so thin there is no way it could be mechanically stable if it wasn't build on a curved sub-frame!

Anyhow - that model only has the domestic-type tweaks; you can turn ClearPixel_TM and SmoothMotion_TM (or whatever they call those horrors!) on/off but no colour-balance in the blacks and so I spent a while changing the "warm" and "dynamic" setting to get a consistent 709'ish balance between the blacks and whites and thought that on test signals and real pictures it looked pretty close; a tad magenta in the dark-areas and I couldn't quite get the saturation to match exactly on all material, but if the two monitors weren't next to each other you wouldn't be able to remember the differences.

Here are the settings (if they're useful?)

Picture Mode: Expert1
Picture Adjust
OLED Light: 60
contrast: 63
Brightness: 52
H/V Sharpness/Tint: 0
Colour: 45
Expert Control
Dynamic Contrast/Edge/Colour filter/Expert Pattern: Off
Gamma: 2.4
White Balance:
Colour Temperature: Warm2
Method: 20 points
Pattern: Outer
IRE: 100
Luminance: 100
Red: 0
Green: -16
Blue: -2
Colour Management System (Saturation, Tint, Luminance)
Red:0, 0, 3
Green: 0, 0, -5
Blue: 0, 0, 5
Cyan: 5, 9, 1
Magenta: 0, 0, 3
Yellow: 10, 1, 2
Picture Options
Noise Reduction/MPEG/Real Cinema/ Motion Eye care/TruMotion: Off
Black Level: Low

Earth Leakage - in water!

I recently bought an Agilent U1191A clamp-meter. This is a piece of test equipment that can measure current flowing in a conductor without having to break the circuit (how you would if all you had was a digital multimeter). The jaws physically couple around the conductor in question and by induction you can measure the electrical current flowing in the conductor. 

Clamp meters have moved on somewhat since I last had to buy one (a Fluke; sometime in the late nineties). This one is a pretty competent DVM as well as being able to sample and hold min, max and average values across all setting. For most days it could definitely do double duty against my Amprobe 37XR multimeter - EXCEPT the Agilent doesn't have non-contact voltage detection (the Amprobe does!). Anyhow - how do you get four and a half digits of resolution across multiple ranges on a brand-name test set for less than a hundred quid? Engineers today, don't know they're born...!

Today I was called to a customer's site where they have three canal-barges, each with two or three edit rooms on board. In the bilges of each boat there is room for little half-height equipment cabinet where they have the shared-storage chassis, network switches etc. They've been suffering an unusual number of equipment failures (motherboards dieing etc) and since they also seem to have RCDs tripping out as a regular feature my first thought was earth leakage.

Here is a picture of the electrical termination point for each boat - two 32A feeders go into the hull, one for the pumps and one for the mains distribution board. The cables are permanently suspended in the water (and have been for many years!) and the ones I inspected had clearly been submerged for so long there has been lots of water ingress into the rubber jacket of the cables. One felt almost ready to crumble in my hands.

Insulation has both electrical resistance and capacitance – and it conducts current through both paths. Given the high resistance of insulation, very little current should actually leak. But -- if the insulation is old or damaged, the resistance is lower and substantial current may flow. Additionally, longer conductors have a higher capacitance, causing more leakage current. Attaching the clamp meter to the incoming earth bond (pre-the consumer unit) measured a massive 100mA of leakage current. This not only risks the equipment being fed off this supply - there is an imbalance between the live and neutral cores and Class-1 equipment is often upset by this, and power supplies can pass this residual current to the earth-plane on PCBs.

More worryingly you've also compromised the safety action of any RCD (Residual Current Devices) in the feed. 

So - my advice was; replace those 32A feeders with marine-grade power cable as soon as possible.

The death of videotape; long time coming.

Say it ain't so! I read the news today, oh boy.....

Sony is to stop selling its range of ½-inch tape machines and camcorders in just over a year’s time. The manufacturer has targeted March 2016 as the date by which it will cease sales and distribution of its professional VTRs and camcorders, owing to what it described as “the global trend of migration towards file-based operation”.

VTs have been a constant feature of my 26 years in broadcast engineering - I spent three years in VT maintenance when I was in BBC TV News and all through my time in facilities in the nineties/early noughties and my last dozen years working for a reseller the most dense way of storing data (which is what video has been for twenty years) is on magnetic tape using a rotating head-drum. 

VTRs are mechanical and hence unreliable; you can't pull rust-on-sellotape (a crude description of videotape) over a rotating metal drum without things wearing out and when I started I estimate that at least a half of all broadcast engineering hours were spent fixing decks. I certainly enjoyed that mix of electronics and mechanics and when I left BBC TV news my supervisor in VT maintenance had this made for me - at the time he reckoned I had done more than a hundred head-drums.

So, here are a few memories about VT formats I have had to deal with. It was all analogue when I started with the D1 format just starting to make inroads. By the mid-90s DigitalBetacam had become the predominant format for most production and post-production and Sony continued their domination of tape formats with HDCam and HDCamSR in the late nineties/early noughties. Since then it's been disk-based (XDCam) and flash-based (SxS, P2 etc etc.) - I haven't done anything more than cleaning a tape path or head-drum in the last decade but I used to be a pretty good VT-fixer!

• 2" Quad; The original broadcast tape format which was on the wein when I joined the Beeb. At Lime Grove studios we did have a couple of Ampex VR 2000 machines. These beasts needed a compressed air feed to hold the tape on the transversely rotating drum. They weren't used for editing, just for archiving P as B recordings. I remember watching an episode of "Star Trek" (original!) being transmitted and marveling at home good composite pictures could look.
• 1" C-format; specifically the Ampex VPR-2B (which was the BBC's 1" of choice) was a bit more of a workhorse machine. Again, BetaSP and Umatic HiBand where more prevalent at BBC News when I was there but when I went out into the independent industry in 1993 1" was a lot more widely used, particularly the Sony BVH-series machines (the choice when I was at CTV in St John's Wood) and then the Ampex VPR3 when I got to Soho in 1994 mastering to an analogue format was already diminishing.
• BetacamSP; The BVW75P was the first piece of broadcast equipment I got to know to the component level. The summer after I joined the BBC they bought three hundred of them and I jumped on the overtime to do acceptance testing. Consequently I got to know the signal path and then in 1990 I got transferred to VTR maintenance and pretty much serviced the same machines I'd been taking delivery of two years before. They were the broadcast workhorse until the late nineties and I still see them. When I went to Nigeria the whole place was still running on them. Here are some photos of the insides.
• D1; I didn't get into Soho until the second generation of D1s had arrived - the DVR2000 series (the 1000 series had half a rack of processing for trick-speed playback). D1 was the first 8-bit uncompressed SD VTR format. All the high-end facilities in Soho made good money out of them - when you could hire a 3 machine D1 room for £650 an hour! Many a time I heard engineers dismiss DigiBeta for it being compressed but I've NEVER seen Digi compression artifacts but I have seen shallow ramp banding on D1 (8-bit vs 10-bit). D1 decks were very expensive (£100k) and cost an arm and a leg to maintain. However - being able to do more than half a dozen generations on and off tape gave rise to all those effects heavy pop videos in the early nineties.
• D2; Sony quickly realized that they'd need a composite digital machine for run of the mill TV production work (i.e. people who needed a drop-in replacement for 1") and so they bought the format off Ampex in some complex licensing deal that allowed Ampex to sell badged BVW75s. Ampex's VPR300 was a terrible machine; we had them at Oasis TV and you could often not get recordings made in the morning to playback on the same machine in the afternoon. After that the Sony DVR28 was an eye-opener. It could stripe tapes at high speed as well!
• D3; Like D2 a digital composite machine but like D5 a 1/2" tape path which meant a practical all-in-one camcorder was possible. The BBC embraced the AJD350 the year I left and according to a pal at Panasonic of the first 98 machines they never got more than sixty working simultaneously. They format got really good after v.2 software when stability, RF performance etc improved. The operational side of the machine was entirely unlike Sony with a very complicated screen surrounded by buttons. Panasonic had to replace heads - almost no usable serviceable parts inside...!
• D5; Channel Four were the only UK broadcaster to commit to D5 which was Panasonic's answer to D1 (but, a 1/2" format with 10-bit video, uncompressed - eventually there was an HD variant as well). You could tell the machine ran so close to the edge in terms of heat performance. We had two of them at Oasis and every morning I would pull out the long boards (below the tape transport) and re-seat all the chips - a day's worth of heat made all the socketed devices rise up. The machine shared mechanics with the AJD-350 D3 machine and with an option decoder board would replay D3 tapes. The tape-stock was the same in both cases and when I was at CTV we would send "D5" masters to Channel Four which were really D3 recordings with a D5 card in the tape sleeve! They never spotted it and it saved us hiring D5 machines (we owned D3). The rumor at the time was that C4 had been given the initial set of machines free to establish the format which was (even then) viewed as entirely unsuitable for a broadcaster. I never id much maintenance on them save cleaning etc. You had to send them back to Panasonic for heads etc.
• DigiBeta; Whereas the first gen digital VTs (uncompressed, either 230mBits/sec for D1 & D5, 155mBits/sec for D2 & D3) required manual tracking for record like 1" the DVW-series 1/2" Digi had a pilot tracking tone system that allowed the machine to track itself for record. It could even do an insert edit if the control track was damaged by driving the phase of the scanner-lock based on the difference in signal strength between the pilot tones and the head and tail of each video track. Consequently I rarely saw machines that made incompatible recordings (that was a constant feature of all the earlier digital VTR formats).  The DVW was also the first machine to feature a Viterbi decoder in the bitstream path off tape and so you tended to get a green channel light (no error correction or concealment) until 1800hours of tape wear and then over a couple of days it would go to orange (error correction) to red channel condition (error concealment). Compared to all those earlier formats (I used to clean the heads on D1 & D2s every day of use!) they had a very low TCO.
• BetacamSX;
• DVCPro / DVCam / miniDV;
• IMX;
• HDCam;
• HDCamSR;
• Umatic;
• VHS;

Barnfind's high-speed data router and optical CWDM for TV infrastructure

I've been a very slack blogger over the last five weeks due to work (installations, running training, getting trained!) and holiday (splendid). I spent a few days last week in Norway as the guest of Barnfind in Sandefjord.

Norway seems to be a lovely country if not a tad expensive (£24 for a round of three pints).  Barnfind are a small company whose engineers used to be with Nevion - you've probably come across their VikinX range of HD/SDi and other facilities routers. 

I had a long Skype chat with Barnfind a month ago and kind of 'got' their range. It's not a one-for-one replacement for any other specific products rather a platform that nicely ties together all digital signals within a facility; synchronous (SDi, MADI, AES, etc) and asynchronous (ethernet - copper & fibre, fibre-channel). They also make CWDM very do'able in a broadcast environment. As we move towards an entire IP infrastructure these are the kind of platforms that allow an easy transition. 

The basic product (the BarnOne BTF1-01) is a 32x32 generic data router and 16 bi-directional SFP ports. The SFPs can be any MSA-compliant units but Barnfind manufacture their own at very reasonable costs (much less than Cisco!). You could insert Ethernet, SDi i/o, fibre or any of around 150 variations they offer. This allows you to route SDi in and out over fibre, insert AES into an SDi stream, convert ethernet to/from fibre etc etc. 

3G HD/SDi input/output SFP

Clearly some signal types don't sensibly convert; routing an SDi stream to a fibre channel-equipped port won't replace an HP workstation running Avid! But where is does make sense everything is taken care of for you. In the case of all video signals (composite, SDi and HDMI are all supported) the signal is converted in the SFP to 3G SDi before it is passed to the 32x32 router.

The BarnOne range extends to several variations - the lower board which carries the router and the first sixteen ports can be joined by two upper boards carrying BNCs, more SFP holes or (more interestingly) CWDM fibre modules. Essentially having BNCs on an upper board allows you to avoid SDi SFPs (it's marginally cheaper to do it on an 8-way board than an extra 8-holes with video-SFPs).

 

The other end of the link could be easily served by their BarnMini units - essentially replacing Blackmagic or AJA converters but integrating very nicely with the BarnOne. For less than £400 you can get either two BNCs with an SFP hole or two SFPs. 

The whole thing makes sense when you realise that all the signal intelligence is in the SFPs - the dual-port BarnMini can do anything that makes sense; maybe you need to route some ethernet coming in on single-mode fibre and send it out over existing multi-mode cable. Again, AES, SDi, MADI as well as all fibre and copper networking are supported. 

Before I start banging on about Course Wave Division Multiplexing it is worth including a photo of the insides of a BarnOne so you can see the control card they use.

That's right! It's a RaspberryPi! When re-invent the wheel; they claim they tested a few Linux SOC boards and found the humble Pi to be the most reliable and they make use of the watchdog timer to ensure it's always listening for config updates. Their BarnStudio software not only allows you to configure the system (including all the monitoring via SNMP) but you can also control the router. They also support several manufacturers generic control panels if that's what's needed.

CWDM

Single mode cable (more often than not) is used to carry a network feed, a 3G video feed or some other data. We quote wavelengths for fibre (typically 1310nM) rather than frequency and so often forget that the sidebands of the signal we put down a fibre are tiny relative to the centre frequency. 1400nM of wavelength is around 200Thz (yes, 200 x 10^12 Hz!) which makes the 4.5Ghz bandwidth of the best-quality HD video coax look very modest. So, we could divide up the single-mode range into many wavelengths and use each for a different purpose; a bit like the radio stations on the VHF band.

These are the standard wavelengths for CWDM working; sixteen channels and by convention the two colours run in different directions (SDi input/output or ethernet Tx and Rx for example). This allows you to have SFPs that put the signal they send/receive onto specific wavelengths. Then, with a simple passive optical splitter/combiner (which is all a CWDM multiplexer is). With all this in place you can use your Banrfind kit to multiplex sixteen functions in and out of a single 9/125u fibre. If that's all you have between premises then this is a lifesaver. The multiplexers are in the £400 range. 

We opened one up last week and it was a thing of beauty; all passive optical engineering with tiny dichroic filters. So, by careful planning you could send multiple SDi signals, ethernet, fibre channel and other things to and from a remote site over a single strand of mono-mode fibre. It could be up to 80kms away. 

There is a further development of the multiplexing technology called Dense Wave Division Multiplexing - DWDM which allows up to 192 channels and very far distances (by the use of Eridium-doped optical amplifier - but in the case of Barnfind (and other manufacturers) the cost of DWDM vs CWDM is fivefold!

The BTF1-07 is the box I've ordered as my demo unit; it has sixteen SFP holes, eight bi-directional BNCs as well as a CWDM multiplexer.

Bryant's network control/measuring PDUs

http://www.bryant-unlimited.co.uk/eyepower.html

My good pal Simon Quill at Bryant has been developing this line of intelligent bay mains distribution for the last few years and I bought a couple last year and have (due to the weight of work) only just got around to playing with them. Quite a lot of manufacturers offer remotely controllable power strips with either a bit of client software or a web interface to control the various circuits. Some even offer current monitoring (typically via a shunt-resistor so you get apparent current; the heating effect, in effect!) but Bryant claims to actually measure the current via a clamp-inductor and they calibrate each circuit's 16-bit ADC prior to it leaving the factory. Simon tells me each circuit can measure to 50A to an accuracy of 1mA and they sample at 1Khz so you can see any harmonic content that is being put on by UPSes etc. 

Some of the other features;

1. Macro start-up, close sequence. You can (for example) get o/p 2 to hold off powering up until o/p 1 has settled.
2. Control of each circuit via the web interface - re-power that server?
3. Inspect the current draw (and power factory) on a per circuit basis; worried that an array of disks is starting to show odd consumption; maybe a drive is about to fail?
4. Look at the quality of the incoming mains - it's rarely a sine wave nowadays!

Simon & I intend to do an episode of The Engineer's Bench on IP controlled PDUs (not just these Bryant ones) but until then here are some screen-grabs.

 This one shows the graph for the third circuit which has an AlicePak audio balancing interface attached and is (we hope!) a linear supply and hence should be an entirely resistive load.

 This is my trusty Tektronix WFM7120 which is clearly using a switch-mode supply; notice how the current draw is when the driver transistor switches as the voltage passes a set value.

 This one shows the earth leakage current against incoming mains; a bit more sine-like but remember the effect is made worse by the presence of the Tek & the AlicePak.

This is a circuit which has no load and so consequently we're seeing the auto-ranged current draw which is just the quantised noise of the ADC.

nice!

System Design with Excel - The Engineer's Bench Podcast

Phil and Hugh go over a few tips and tricks for using MS Excel in the design of film & TV facilities. Find it on iTunes, vanilla RSS, YouTube or the show notes website.

A lie can travel half way around the world while the truth is putting on its shoes.

Not that Mark Twain knew much about fibre optic cable. If he did he'd realise that he was only talking about seventy milliseconds...!

I often have to provide proposals and quotations in response to customer's tender documents and in the last six months I have seen three such tenders specifying tight-buffered fibre for their internal networks.  When I push their staff engineers for a justification they "um and err" and are easily persuaded to do the right thing (i.e. use spliced loose-tube fibre). If you need to read up then I've written a few things in the past.

Last week I came across the following from the Argosy website;

Tight buffered cables are intended for indoor applications. They are more hardwareing than loose-tube cable, as such they are well suited for long indoor LAN connections, burial or complete even submersion in water. Tight buffered cables have a special two-layer coating. The first layer is plastic, the other a waterproof acrylate.

I wonder if this mis-information is their doing?

Chassis vs Signal earth on RS422 remotes

Grounding is essential to reliable operation of any RS422 connections. It is also the most overlooked and least understood. The easiest way to ground your RS422 equipment is to simply use "Earth" ground as your return path. Although easy this may not be the best method for grounding your application, because current leaking from equipment, electro-static discharge (ESD), and lightning all drive current through this path which results in high noise content. The reason for this increased noise level is due to the fact that "Earth" ground presents a relatively high resistance. RS422 is designed to operate normally with a ground potential difference of +/- 12 Volts. During normal operations this is typically not a problem, however during fault conditions or lightning strikes even within ½ mile the ground potential difference can reach hundreds and in some cases thousands of volts. This will most likely result in damage or failure of one or more devices on the RS422 router.

In TV facilities I most often come across three methods of earthing;

1. No earth - assume that the mains return is good (all equipment is class-1 and bolted into it's bay and that signal and chassis earths will be close)
2. Use pins 1 and/or 9 on the 9-pin D-type to couple the chassis earths together; please, don't get me started about intentionally connecting mains earths between different areas! Do you like dealing with induced hum between different areas or buildings?!
3. The best way; using pins 4 & 6 - the signal screens (it's what they're there for).

The job I'm finishing at the moment had a problem with Digital Rapids workstation which wouldn't run through the RS422 router although patching around the router worked - the PC could control a VTR. My first port of call was to test the cabling/router patch by sticking an old Sony RM450 edit controller at the back of the Rapids and pretend the RM450 was the workstation - all good; VTR control and timecode return worked fine (so Tx and Rx doing their things). 

So my first thought was to measure the impedance between the signal ground on the router and the mains earth - high Z so no return patch for the RS422 via the router if it was relying on the mains earth (scenarios 1 & 2 above) and since the router is optically isolated on it's data inputs I wasn't surprised. That is the way it should be done.

Now then; most PCs that are running video apps and have to control a piece of broadcast kit use an RS232 port with an external RS232-422 adaptor. These essentially just balance the Tx and Rx pins and there are several models. None of them (in my experience) actually use a pair of rep coils to properly balance rather they use a pair of op amps in a differential input configuration. This is fine but doesn't have the noise immunity that you get with coils (common mode rejection). What it does mean is that all of the noise immunity of the circuit comes from the electrostatic shielding of the earth and so you better get it right!

I cracked open the cheap'n'cheerful '232-422 adaptors supplied with the rapids and they had the screen connected to the shield of the 9-pin on the RS422 side (so relying on scenario 1 above). Moving that to pins 4 & 6 (scenario 3) fixed the problem. 

As an aside the Adenda "Rosetta Stone" adaptors that Avid supply do the right thing!

Recent presentation 'Amulet KVM over IP; a technology whose time has come'

Failure of the BBC's "Digital Media Initiative" and other large IT projects

I still describe myself as a Broadcast Engineer rather than a project manager - and in fairness I do spend more time looking at cable-schedules and schematics than Gantt charts. However; I am often responsible for other people's money in achieving what they want in TV and data facilities. Compared to the BBC DMI project the largest project I've had overall financial responsibility for came to a bit more than 1% of the size of that gig so I am in no means an expert. However - I have worked for dozens of large and small broadcasters and I think I've seen some of the best and worst aspects of other people's project management styles.

I feel sorry for John Linwood - the BBC's CTO who has been suspended over the whole debacle. It's telling that the Beeb now have a CTO rather than a Chief Engineer as that latter term implies that you've had a career in broadcast engineering - you've calibrated monitors, fixed studio cameras (and then racked them in live productions), installed Media Composer (and supported the editors), replaced the head-drums in VTRs as well as the myriad other bits of experience that the top technical job at the world's most prestigious broadcaster would imply. John is a similar age to me and so I'd expect him to remember mk.2 telecines, tubed cameras and 1" VTRs but he's a software guy; ex-Yahoo, ex-Microsoft and it's there you'll find both the justification for him being the Beeb's top technical guy and the reasons for the pickle he's now in. The DMI was a software project BUT software projects have a huge propensity to fail. Around 30% of software projects in commercial industry fail - but that's OK; you have to take risks and great things don't come if there wasn't a danger of failure (that's why it's a risk!). However - in government IT projects the risk of failure is an awful lot higher - typically 70% for publicly-funded IT projects. You'd expect project manager who are being paid with tax-dollars to be more risk-averse but the opposite seems true. Clearly this has been the case at the BBC with the DMI. 

Ross Anderson has written extensively about this kind of thing; he did an interview with Stephen Fry on Radio 4 a couple of years ago which makes a lot of these points; as an aside his brilliant book "Security Engineering" is now in the public domain.

Here are a few thoughts on big-organisation technical projects;

• The danger of "not invented here" - when I was at the BBC most custom projects (i.e. equipment and solutions not bought in from external manufacturers) were often specified and implemented by Research Department. In fact too many projects were as there was an attitude of "nobody understands what we do except us" and so consequently too many things were done by people who might be doing it for the first or second time (chief engineers of facilities will have designed/built maybe two or three machines rooms in their twenty year career - I've done it dozens of times in the last decade!). 
• "Gold plating" everything - In the late eighties/early nineties there was a guy in BBC Research Department who liked the DEC MicroVax 3100-series running VMS (at the time it was a £35k industrial computer) and so whenever a project needed an external computer we'd see a MicroVax appear in the machine room. Automated upload of new weather symbols to the Quantel Paintbox - throw a MicroVax at it. Download realtime financial data from Reuters to make the Aston strap for the breakfast news financial segment; control the caption generator with a MicroVax! However - when it fell on the maintenance department to make something work they typically used a BBC Micro (we all had them at home and new how to program/homebrew them) - the ASTED project to control external logo generators and make them work with the Aston caption machine for news programmes was all done via a £350 BBC Micro, not a £35k MicroVax.
• In a similair vein I had a friend who was working on the NHS unified records system for EDS - he spent eighteen months working on a new secure-VPN protocol; that's not a problem that needs solving! That one is already done with a choice of closed and open-software solutions.
• Don't despise project management methodologies. Lots of engineers have a distrust of Prince2 and it's ilk and for sub-£1m projects the overheads are too onerous, but there are so many valuable lessons that proper project managers bring. In a recent discussion with a colleague about how one project had turned quite painful we realised that the PM101 principle of fully involving the users had been almost totally missed by the customer and it was proving hard to get the poor editors and assistants to buy-into the new system once they saw the implementation.
• "Good, Fast, Cheap - choose only two" - this is the warning PMs often give and it's regarded as customers as a prohibition rather than a good principle to run a project by. The tension of having to hold those three ideas and adjust the sliders as necessary means you don't push them all to the max and expect the best outcome. If they had regarded this principle there is no way they would have let the thing drag on for five years. Timely projects are the best kind.
• Specify, specify, specify - the more you leave to fortune or to the contractors' discretion means you have too many undefined problems.
• Don't disregard experience - the engineer who taught me all I really know about broadcast SI - Chris Clegg - had one thing he used to say about designing facilities; "...given a crew of qualified operators they should be able to run this studio/OB truck/machine room with only fifteen minutes instruction from the usual operator". You can't do that without intimately understanding how TV workflows interact with facilities and how operators and assistants operate. The point is that the best broadcast project managers have years of experience in those areas. Professional project managers (who aren't experienced engineers) don't have those insights.

One thing I can't understand is why Siemens were initially given the project; are they renowned for any of the things that were trying to be achieved? - big database, large video storage, transcoding, version management and the underlying filesystem to make it all hang together? Given that it also has to work with your editing, transmission, and VOD platforms why on Earth not get Avid, Isilon, Google etc involved; OneFS for the backend and GFS for the database sound like they were designed for this (and they also run most of the big Internet media sites).

It'll be interesting to see if the BBC takes on an experienced engineer as their next CTO.

Beware the fibre contamination!

These are two pictures taken down my hand-held fibre microscope - you can see the core and buffer of two examples of fibres that were spliced maybe three months ago in a new building. I suspect builder/decorator dust contamination and even though they have had their cover-pieces on they aren't air-tight and so we're now looking at an extra 0.2dBs of loss on those circuits.

Not a whole hill of beans, but worth noting.  It is jolly hard to position and focus an iPhone over the eyepiece, but here is an image of what a new, uncontaminated fibre pigtail looks like (these are all multi-mod OM3 BTW). Aside from the JDSU microscope I also carry a Cletop cleaner when I'm on fibre mission. It's a small cleaner with a replaceable cartridge that allows you to swipe the end of the patch cord or pigtail with a fresh piece of dry-clean, lint-free material and it removes even stubborn stains.

Temporal Dithering; good for colour depth, bad for extending!

I've become quite a fan of products based on Teradici - an on-the-fly compression system that allows dual-DVI w/peripherals to be extended over ethernet. It works very well and you can't tell you're not looking at something that's been packetised and extended over a network. Howver, a fly in this ointment is the current gen ATI graphics drivers on OS-X. They use a technique called Temporal Dithering;

Temporal dithering is a technique employed by some graphics cards to simulate colors that they cannot natively display by rapidly changing the colors of pixels, tricking the eye into seeing “in-between” colors. During PCoIP remote sessions, temporal dithering can cause extremely high bandwidth utilization because the rapidly changing pixels force the PCoIP protocol to constantly deliver large screen updates to the remote desktop.

The upshot of this is that when you have nothing going on screen - no mouse movement etc (where you'd expect to see no data traveling on the network) you get;

And the effect on screen is compression artifacts and a sluggish mouse pointer. The guys at Amulet Hotkey are looking into it for us, but it might be a deal-breaker.

Flailing around the web revealed someone at Disney who has produced tweaks to X's configuration file for use under Linux; unfortunately this doesn't work in OS-X as the dithering is done by the driver. However, just for reference (or the lulz!)

xorg.cong

Section "Screen"
    Identifier "Screen0"
    Device "Videocard0"

    Monitor        "Monitor0"
    DefaultDepth    24
    Option        "CIOverlay" "On"

    Option        "CursorShadow" "Off"
    Option        "NvAgp" "3"
    Option        "TwinView" "1"
    Option        "SecondMonitorHorizSync" "21.0-140.0"
    Option        "SecondMonitorVertRefresh" "47.0-72.0"
    Option        "TwinViewOrientation" "LeftOf"
    Option        "MetaModes" "1280x1024_72,1280x1024_72; 1280x1024_60,1280x1024_60; 1024x768_72,1024x768_72"
    Option        "TwinViewXineramaInfoOrder" "DFP"
#RandR    Option        "RandRRotation" "true"   # Requires xorg 6.8.1+
#NoEDID    Option        "UseEDIDFreqs" "FALSE"
#NoEDID    Option        "UseEDIDDpi" "FALSE"
#NoEDID    Option        "ModeValidation" "NoEdidMaxPClkCheck"
#ExactMODE    Option          "ExactModeTimingsDVI" "On"
    Option        "DPI" "85x85"
    Option        "AllowGLXWithComposite" "true"
    Option        "DisableGLXRootClipping" "true"
    Option        "AddARGBGLXVisuals" "true"
    Option        "AllowSHMPixmaps" "true"
#FALSE    Option        "Dac8Bit" "true" # forced spatial dithering instead of temporal for teradici performance
#FALSE        Option        "RegistryDwords" "DitherAlgo8=3;DitherAlgo6=3"


# Other possible options. Enable at your own risk
#    Option        "RenderAccel" "On"
#    Option        "Overlay" "On"
#    Option        "SWCursor" "On"
#       Option          "TwinViewXineramaInfoOrder" "DFP-1,CRT-0"
#       Option        "ConnectedMonitor" "DFP-0,CRT-1"
# This can be used to clarify orientation
#    Option        "TwinViewOrientation" "DFP-1 LeftOf CRT-0"
#    Option        "IgnoreDisplayDevices" "CRT"

    Subsection "Display"
        Depth        24
        Modes        "1280x1024_72"
    EndSubsection
EndSection

Audio podcast 2 - the engineering!

Hugh and I continue our discussion of audio and make particular mention of cabling for TV facilities.
Find it on iTunes, vanilla RSS, YouTube or the show notes website.

Synchronous ain't always best!

Engineers have been trained since time immemorial that with multiple sources of video the best thing is that they are always locked together and timed to a common reference - ideally station black & burst (or TriSyncs nowadays). The reasons for this are numerous, but a couple;

1. Studios cameras into a vision mixer have to be locked to achieve clean cuts between the cameras - it would look rubbish if you had a frame roll every time you cut between sources. The same is true of sources into a continuity suite etc.
2. In the case of Avid for the longest time you had to make sure the VTR was locked to the same reference as the Media Composer (all the way from v.7 ABVB systems in the mid-90s to the last revision of Adrenaline in 2007!) otherwise the audio and video capture portions of the machine would free-run WRT each other and within minutes you'd be loosing frames of sync.

So - I had a coffee and chat with a pal this morning who works for a big facility that has won an archive digitising project and they are using the BlackMagic DeckLink 4K card to allow them to ingest 4 x DigiBeta tapes at once. The capture software is ToolsOnAir and they found that after the first clip was captured the second clip would be a frame out of sync, and progressively worse after that - unless you re-booted between captures! It turns out that if the four VTRs are allowed to free-run then you don't get the problem. Perhaps processing the vertical syncs places a burden on the card/software and if it happens simultaneously on all four inputs trouble ensues?!

It reminded me of a situation with a big broadcaster who was distributing their regional variations over Astra on a single multiplex. The stat-mux was very unhappy with material that was (for the most part) identical; only the ad-breaks differed. Most video cuts occurred at the same time across all six SDi feeds. The solution was to apply a two frame delay between all the sources (so o/p 6 was now 12 frames late WRT o/p 1).

Mains 101 - part 2 / Postscript

Hugh and Phil wrap up the previous mains safety podcast with some corrections and photos from Nigeria! Find it on iTunes, vanilla RSS, YouTube or the show notes website.

Phil & Hugh's next podcast - electrical safety.

Find it on iTunes, vanilla RSS, YouTube or the show notes website.

Flame ain't all that

The facility that I'm building at the moment has a couple of Flame Premium suites - the very best version of Autodesk's TV and film compositing and finishing tool. Now I know very little about Autodesk products - I've never really worked in facilities where they were used. Flame, Smoke and the rest of them have been around for nearly twenty years and so you'd think they are what software people would describe as "mature products".
Anyway - a currently model Flame runs on an HP Z800 workstation with a Kona K3 card for video i/o (more about that in a minute!), an nVidia 4500 for desktop and their own proprietary fibre attached storage called "stone". They can typically handle two streams of 2k in realtime.
All very good, but everything is specified in a system - even down to the Eizo monitor you can use for the desktop display. This is because the SDi output for preview (which is not the output of the K3) is made by looping the 2nd DVI output of the graphics card into a daughter card that converts it to SDi. Consequentially you often see bits of the GUI on the HD video monitor. Now the reason for them specifying a certain model of GUI monitor becomes evident - they have to run the output at a rate near video for the DVI output to be convertible! This gets even more silly when you find out you have to use their own provided long DisplayPort cable to run it. We'd run in LC loose-tube fibre with DVI extenders which initially only worked whilst Linux was booting in text mode (i.e. before the X11 display subsystem could run). I had to throw in a Lindy EDID manager to fool Linux into thinking it really had the Eizo directly connected before we could run the desktop (and hence the Flame application).
This, along with a ton of little bits that the kit ships with; an 8-port ethernet hub, Lucid AES->analogue converters etc gives the impression they expect you to install the system on your dining room table rather than in a pro video facility. It really has the feel of a v.7 Avid from 1996! Cobbled together from third-party bits and only just running (everything on the hairy edge).
It reminds me of the Abekas DVEous (again, 1996) - one of the design engineers confided in me that if every bit of silicon on the video-processing board only performed to published spec then the system couldn't work. They relied on everything outperforming itself.

Estimating the time a job takes

For the last eight years I have spent a lot of my time quoting for broadcast installation builds. From small single edit suites to £1.5m broadcast editing centers I think I have a good feel for how long most jobs take. I recently did a search of my arrogated purchase orders folder and discovered I've bought more than 500 equipment cabinets, nearly 1000 video jackfields, and just over 100 kilometers of bulk fibre optic cable - all in the last eight years!
Anyhow - I often get quotes thrown back at me by chief engineers who assume I'm trying to fiddle them and they will typically say;

You've quoted ten days for a wiremen to wire those audio patch panels back to krone blocks; I'm sure my guy could do it in seven.

Even as a chief engineer he's probably only built one or two big machine rooms in his time. He might think he has experience of wiring technical facilities, but I've hired and paid wiremen to do many hundreds of audio panels in recent years. Possibly more than all the panels every chief engineer in Soho will oversee this decade! I know how long things take.

Anyway - for the last five years I've done an audit at the end of each year to compare the number of wiremen and engineer days each job used against what I thought when I quoted. It turns out that I naturally underestimate the time required by around 25% - If I quote 100 wireman days it'll be nearer to 130 when all is said and done. This isn't down to me not knowing how long each part of the job takes - it's a function of wasted time;

• Client won't provide passes to everyone and so guys have to wait idle in rooms or go looking for the pass.
• Deliveries are late, guys are idle
• Customers change their mind, but not enough (or it isn't politic) to warrant a change order
• Faulty parts - we always have to replace those for free!
• Various other things you didn't expect.

So - I've made it my habit to still estimate the number of days as I think it should be and then add on the 25% extra. Consequently my quotes have been getting a lot more accurate over the last few of years. I also discovered the same effect with parts but in the other direction; I always over-estimate the amount of cable/connectors required by around 10%.

So far I haven't done anything about that!