USB current draw by HID devices and Amulet Zero Clients

One of the complaints I often get from Amulet users is that they get strange results when using power-hungry USB devices. The prime culprits are Wacom tablets (particularly the Intuos and Bambo series) but today I went to a facility where they were getting intermitent behaviour from the left hand control and shift keys which in Media Composer is a real pain. 

The most common third-party Avid coloured-keycap keyboards are from Logic. 

They come with a USB hub built in and so you'd expect them to draw a bit more than an HID device, but it's worth looking up what the spec is for USB power draw. This is stolen from Wikipedia;

There are limits on the power a device may draw, stated in terms of a unit load, which is 100 mA, or 150 mA for SuperSpeed devices. There are low-power and high-power devices. Low-power devices may draw at most 1 unit load, and all devices must act as low-power devices when, starting out as, unconfigured. High-power devices draw at least 1 unit load and at most 5 unit loads (500 mA), or 6 unit loads (900 mA) for SuperSpeed devices. A high-powered device must be configured, and may only draw as much power as specified in its configuration.

Or, in summary;

However, flipping the keyboard over gives a different story;

Wow, 1.1A - more than twice what the spec says. However - these keyboards are used the world over and it seems that motherboard USB ports are more than able to power them. I thought I'd get my own reading to see what the current draw was so I stuck one of those USB volts/current monitors in line. The keyboard continued to work but the USB-meter gave me a 0mA reading; it was working the day before, but today, no dice. When I got it back to the workshop I opened it up and discovered the current-shunt resistors was blown away!

 see bottom left on the +Vcc rail - where's the resistor?!

So I conclude that the keyboard is actually pulling a lot more than 1.1A (at least at in-rush).

So, Amulet can supply 2.0A across the USB hub on it's Zero Client with a maximum of an amp on any one port. All much better than the spec but somewhat less than the Logic keyboard is pulling. In the case of power-hungry, non-compliant keyboards and tablets I stick in a powered hub and that normally fixes the problem.

The Engineer's Bench podcast - "KVM-over-IP and Data Encryption"

Hugh and Phil talk about KVM-over-IP systems with particular reference to Teradici and Phil's favourite manufacturer Amulet Hotkey. They also go over the basics of encryption with symmetric and public-key crypto.


Find it on iTunes, vanilla RSS, YouTube or the show notes website.

Upgrading Linux SOC firmware on Amulet DXip rack cards

I've banged on about Amulet a lot in the past - it really is the best KVM-over-IP implementation of Teradici there is. One of the reasons for this is their external DXiP cards and rack system that allows you all the benefits of Teradici without having to stick in a PCI-e card (either for reasons of turnkey compatibility or no spare slot!). We've installed hundreds of seats at broadcasters, VFX, post and educational facilities. 

The external card although compatible with all other Terdici-based gadgets is a much more complicated gadget than an internal one. The reason for this is that when installing on a PCI-e bus you have the bus and OS to host the audio and USB chips; no drivers are required (it's just another Realtek audio device and USB hub) but in the case of an external card with analogue audio i/o and USB in you have to have an additional Linux System-On-Chip machine to handle those functions.

Upgrading the Teradici firmware is a cinch, either from the web interface of each card or using the Teradici Management Console but upgrading the SOC firmware is a different matter!

The guys from Amulet took me through the upgrade on a couple of our demo cards but that was a month ago and I hadn't really made any notes and so I spend yesterday afternoon familiarising myself with the process. 

Here are my notes with particular attention to the gotchas that had me scratching my head! The principle is that you use the serial header on an external rear-card (they call it the "personality" card), and power it from a bench supply (22v with at least 0.75A - don't set the current limit any lower!).

For networking connect to the AUX port - although when the card is in it's usual mode the AUX and MAIN network ports behave exactly the same, for uploading new firmware you need the AUX. 

Position yourself such that you can hold the ident button on the front of the card and push the power button on the front of your bench-supply. You should then be able to swivel to your computer whilst keeping the ident button held in so you can reach the keyboard.

You probably don't have a serial port on your computer (definitely not if you're rocking a Mac!) and you will need wired ethernet. 

So - fire up your favorite serial-terminal program (doesn't the whole world use PuTTY?!) - CoolTerm if your on OS-X and make sure you have your USB-serial adaptor (I use the Keyspan) with the following serial parameters set;

However, this is where my first gotcha reached up to bit me in the backside! Using PuTTY (I run Windows on my Mac in Parallels) I could see the serial terminal output from the card, but could send keypresses back to it until I realised PuTTY sets it's own serial parameters and ignores the port settings in Windows!

So - make sure it hasn't re-set flow control to XON/XOFF - it has to be None. If you want a little RS232 primer then Hugh and I did an Engineer's Bench podcast on it a while ago.

So, with this all set, power up the bench supply with the ident button held in and keep it held down until you've read to the end of this section! You should see the following in the terminal window;

When it gets to the hit any key to stop autoboot do that and only then can you release the ident button on the front of the card. If you release the button any sooner it disables the networking on the card which is a bummer 'cause you need that for the card to reach out and acquire its new firmware!

Now, notice a couple of things in the terminal; the IP address of the network port on the card AND the expected IP address of the TFTP server that is going to provide the firmware images. You could reset those by using;

setenv ipaddr_GT xxx.xxx.xxx.xxx (the DXiP-2 IP address)
setenv serverip_GT xxx.xxx.xxx.xxx (the TFTP PC IP address)
setenv gatewayip_GT xxx.xxx.xxx.xxx (is the network gateway IP address)
ipc GT flash (This write the IP settings to flash)

But, for my money life is too short and I'd rather just set my computer's IP to match the requirement for the TFTP server.

So, TFTP server - there are lots to choose from including TFTP32/64 for Windows but since OS-X has one built in (although it's a faff to configure - use Fabrizio Larosa's front end)

I've got the UNZIP'ed archive of the package from Amulet in the TFTP root.  Now the best thing to do it make sure the card can see the server - although it is a very minimal kernel it does have PING;

I did this screen grab with a different card that was looking for the TFTP server at 192.168.200.161 rather than the 192.168.200.127 shown in the rest of these notes.

The card itself will not respond to PINGs whilst in this mode (which rules out using my favorite multicast PING 224.0.0.1 which everything should respond to!)

So - you're logged into the card over RS232, you've got your TFTP server running (either on the same machine you're terminal is on or some other machine on the same LAN - remember the requirement for IP addresses - in this case everything is 192.168.200.x) now you can tell the card to go and collect the new boot-loader using the command RU

This happens quite quickly - the card retrieves the new boot-loader from the TFTP server and then does a reset; as soon as you see it doing that press the ident button and keep it held in exactly like the first part of the process.
Next it's time to re-flash the Linux SOC image, to do this use the command RL

This take a few minutes. Once it's done you can verify all is well with the command V

The Image Name line should match the CN-version you got from Amulet.

CN-887 is the current version

The next thing to do is take the card, put it in a DXip chassis and test it works as a proper Amulet sender. This firmware improves USB stability for bad-behaved devices, improves audio noise and stops the requirement for the occasional card-reboot when re-starting a machine.

Avid, KVM systems and the cry of "...not fit for purpose"!

There are several things that happen when you plug a monitor into a graphics card. I'm assuming DVI (this is the 21st century!) and all other displays standards; HDMI, DisplayPort and Thunderbolt all follow a similar principle.

1. Pin 16 on the DVI connector on the graphics card is refereed to as the "hot plug detect" pin and is held logic-high (at +5v) through a very high value resistor. When you attach a monitor the pin is momentarily taken low to alert the graphics card to the fact that a monitor has been connected. 
2. The graphics card generates an interrupt on the PCI-e bus
3. Windows sees this and uses it to generate an EDID exchange request
4. The monitor responds with it's EDID profile
5. If the EDID profile is the same as last time the graphics card driver does nothing; it's the same monitor after all
6. If the EDID is different the driver re-sets the system resolution - this is particularly important if it's either a lower resolution or frame rate than it was running at before; if it didn't do this you'd get black screens. Nobody wants that.

Now then, Avid Media Composer does something different! It listens for the interrupt and when it sees it it halts operation and displays an error message saying it needs to re-start. Even if all that has happened is that your monitor cable has fallen out of the back of the computer and you've reconnected it - go figure.

The upshot of this is that when you're using a KVM routing solution of any kind and you assign a new pair of monitors to the Avid it insists on re-starting. I've been aware of this for a while and I let people know about it when demo'ing or presenting at trade shows; but it's just the way of things. There is nothing you can do whilst Avid does the wrong thing. Amulet - my favorite KVM-over-IP system does exactly the right thing; it holds off asserting pin-16 (and triggering the chain of events above) only when absolutely necessary. A media operator can be switching around four Media Composers and each workstation is unaware that the operator is being promiscuous. Amulet only asserts pin-16 when there really is no other option; when a new desk-end "zero client" takes control of a machine it hasn't yet seen. The only place I've seen this to be a problem so far is when an editor starts a layback to videotape, presses disconnect on his desk-end zero-client and calls his operator saying "...I've started the layback, I'm off home; can you watch it through to the end?". Then, the operator tries to acquire that Avid and by necessity the Amulet has to alert the computer to a new pair of monitors and Media Composer halts (ruining the layback).

This was the issue at ITV Salford and I home-brewed a little gadget to stop the Avid being able to tell when new monitors where attached; I essentially neutered it's ability to detect pin-16. 

So, I think Amulet does exactly the right thing, if it didn't you'd loose the ability for workstations to detect what monitors were attached and pretty soon you'd have rooms where you had black screens; nobody wants that. The bogeymen here are;

• Avid - why on earth doesn't it do what Windows does?
• Editors who don't watch their own laybacks!

We are now in an argument with a customer who I explained this all to when I was demo'ing Amulet, but we didn't win the SI quote, but we still supplied the KVM. The SI who is installing is shouting blue-murder about "...not fit for purpose" and we're having to brew up 150 adaptors to keep everyone sweet. My feeling is this will cause more trouble further down the line just for the ability to not close an Avid project for those occasions when you need to hand a machine off to someone else. What kind of workflow needs that?!

Recent presentation 'Amulet KVM over IP; a technology whose time has come'

Amulet with the Linux Wacom driver under sub v3.8 kernel

Amulet are a great company - I believe their KVM-over-IP extender to be MUCH better than Avocent, AdderLink and ThinkLogical's offerings, but like everyone else, once you packetise up DVI and USB and send it over a network you are going to have to deal with funny effects of devices and drivers that play fast & lose with the specifications. This is from their engineering team;

Wacom tablet and Red Hat Linux has been traced to a bug in the Red Hat Linux Wacom driver. The driver has a very obvious bug in it whereby outward data is declared as inward. The Wacom tablet simply ignores this when it is connected directly, although this works it is actually because the tablet USB stack is sloppy. The rack card however cannot ignore anything else we wouldn’t be able to operate at all so it believes what the Red Hat driver is saying and waits for an event which then never happens.

As detailed below in our analysis of the Red Hat and main Linux code drivers this issue has been resolved in the latest Linux build, with the fix implementing exactly what we would expect to see. This however has not yet found its way in to Red Hat, although may do so later this year, again as per the details below.

To enable use of the Wacom tablet on the current version of Red Hat (5.3) you are using we have added a work around to the rack card firmware. It appears to work but will require testing in the lab  over the next couple of next week before we can release. The workaround has to circumvent many checks in the code that are designed to catch driver bugs just like this. As a result the workaround is very specific to Wacom devices and this exact bug so we can’t guarantee it will work if anything changes.

Red Hat 6.3 (Amulet Hotkey test version)

• Red Hat version - Red Hat 6.3 (Santiago)
• Kernel version - 2.6.32-279.el6.x86_64
• RPM package - 2.6.32-279.el6.x86_64
• Wacom driver source - wacom_sys.c
• Driver bug - Incorrect bitmask causing incorrect data direction expectation in USB driver.

Code copied below shows the addition of the correct bit mask command (in red) in the latest Linux driver

• Code in current driver - USB_REQ_GET_REPORT, USB_TYPE_CLASS | USB_RECIP_INTERFACE
• Code in latest Linux v3.3 driver - USB_REQ_GET_REPORT, USB_DIR_IN, USB_TYPE_CLASS | USB_RECIP_INTERFACE

Red Hat 5.3 (Client version)

Although versions numbers differ to Amulet Hotkey test version the Wacom driver is the same and has the same bug present

• Latest Red Hat release (6.4)
• RPM package - kernel-2.6.32-358.el6
• Wacom driver source - wacom_sys.c
• Driver bug - Still present as per RPM package 2.6.32-279.el6.x86_64

Main Linux code base

• Version - v3.2.45
• Driver bug - Present
• Version - v3.3
• Driver bug - Resolved

Future Red Hat 

• Wikipedia gives the time frame for the next major RHL release (7.x) as Q3/4 2013
• We can't be sure whether this would incorporate the fix to the Wacom driver however Wikipedia states "Red Hat Enterprise Linux 7 (?), 2013-6+ Will be based on Fedora 18,[17] which as of March 2013 uses Linux kernel 3.8.[18]". This would indicate that the upcoming release of Red Hat 7 will fix the issue, as it will use a newer kernel, v3.8

Amulet's temporal dithering KEXT & fear of OS-X's console!

I've written previously about Amulet's fix for both nVidia and Radeon graphics cards doing temporal dithering under OS-X and how that spoils any KVM-over-IP system's ability to do compression. I had a good meeting with Amulet on Friday relating to a proposed customer's requirements but also got to chat to James Seward (@jamesoff on Twitter). He showed me a couple of interesting things:

• My assumption that the current version of his Kext wasn't working under Snow Leopard was actually due to the method he was using to signal the Kext had installed correctly - the error basically says that NScolor doesn't understand the call - never fear the Console! It showed that all was well;

• His worry that any other KVM-over-IP manufacturer could just take his Kext and use it to make OS-X displays work using their system has been circumvented by checking for the presence of Amulet hardware (that's an expensive dongle!);

Now the week after next is BVE, this time it's at the Excel Center in Docklands (goodbye Earl's Court!) and I'm doing some training tasters but all of the workstations on our stand will be extended over Amulet. I think this is one of the most significant technologies we've taken on in recent years and so come and grab me for a demo.

EDID isn't the only thing that graphics cards look at.

Avid Media Composer insists on re-starting when it sees a change of monitor. I always assumed that it was done via Windows detecting a new EDID profile but it turns out that if you unplug and re-plug the same monitor (i.e. EXACTLY the same EDID data - even s/n) then the same happens and so something deeper is at work.

Look at pin-16; Hot Plug Detect. Basically it is held low by the graphics card but pulled high when a monitor connects - this generates an interrupt in Windows which forces the card to do an EDID refresh - request a new profile and possible re-do the HDCP handshaking if needed. The interrupt is also seen by Avid and used to force a re-start of Media Composer.

Now then, Amulet (our favorite KVM-over-IP technology) spoofs all of this; it caches the EDID profile until a new client connects and when that happens it asserts the Hot Plug Detect pin as if a monitor had been connected - essentially spoofing what happens in the real world. There are various registry tweaks to for the graphics driver to ignore pin-16 but they work variably - change driver and suddenly pin-16 is being listened to again.

When I figure out a solution I'll update this post. Meantime the customer's dream of starting off a layback or capture and then handing it off to another client is still on hold....

Amulet - they fixed OS-X's Temporal Dithering issue

As mentioned in a previous post temporal dithering on OS-X has proved troublesome with Amulet (and in fact it bedevils all KVM-over-IP systems). The guys at Amulet have written a Kernel Extension that stops the card turning on temporal dithering. James, their engineer, explained to me that it's a different technique between nVidia and Radeon, but they've got it licked;

 

 

kexstat is a utility to show which Kernel Extensions are loaded. Here I've grep'ed the output to exclude all the Apple ones. You can see the Amulet one at 0x2000

The proof of the pudding is that now all the MacPro clients on this particular Amulet system look splendid; even full-screen replay of 1080P material.