Enabling Simulcast for Video Conferencing Tools
Simulcast development - the story of a heroic team that restored peace among rebellious employees tired of having their calls dropped in the midst of important meetings with clients.
We’ve all been there. Remember experiencing delays that caused all participants to interrupt each other’s...sentences and hearing an echo that acts as a painful reminder of how awkward your own voice may sound? Maintaining video call uptime and stream quality is a difficult task that raises the issues of poor connectivity and server overload, especially considering that each client is different. This is where simulcast can help by overriding some of these problems and by providing improved call quality.
Which video conferencing tool did we work on?
Our client is Highfive, a provider of integrated hardware and software video conferencing solutions. The product has two main hardware components: a wirelessly controlled wide-angle HD camera with high-fidelity microphone array and a Dolby Conference Phone with integrated microphones and 360 degree audio. Highfive allows you to share your screen on a TV or other such devices and to connect via a browser or a native app.
Challenge number 1: Migrating to another platform
Highfive had to abandon the original platform it was using, because Google stopped supporting the NPAPI Chrome plugin. As a temporary solution, it offered the Highfive product as a native app for desktop computers using the older version of Chrome that supported NPAPI. Yet as the idea behind Highfive is to provide users with simple and seamless access, the company had to switch to another platform to achieve this.
The alternative was to opt for WebRTC (Web Real-Time Communication) and JVB (Jitsi Videobridge), which are both open-source projects that simplify A/V communication. WebRTC allows direct peer-to-peer communication, enabling real-time communication inside web pages without the need for downloading plugins or native apps. JVB is a SFU (Selective Forwarding Unit) that is compatible with WebRTC and allows running thousands of video streams from a single server. Symphony helped with the platform migration and the adjustment of all of the features to the new technologies.
Three for one and one for all: Enabling three stream simulcast
Every video conferencing tool faces the same difficulties: preventing dropped calls, maintaining video and sound quality, and handling a large number of users at once. In Highfive's case, the Symphony team helped port the simulcast patches over to newer versions of Chrome. Since every company that uses Highfive has a different network infrastructure and number of employees, we had to devise a way to adjust the broadcast quality so that users would remain online even when the link/uplink was weak or when the server was loaded with many users at once.
We enabled simulcast in the Chromium browser by working directly on the Chromium project, adding new features and changing it to adapt to the needs of Highfive. The result was a simulcast that has three independent stream channels which come in 180x180px, 360x360px, and 720x720px. At the cost of a larger uplink load, the end result was an adapted stream where users with a slower connection would have lower video quality but the connection would remain uninterrupted. If this were not the case, the connection would simply break up once the uplink fell below a certain threshold. Using a link estimation algorithm, JVB makes the decision on which of the three streams to provide. In order to improve the end result, Symphony further optimized this algorithm.
When your components don't play well together
The first generation of Highfive hardware devices and most mobile phone hardware support H.264 hardware encode/decode. This is why Highfive went with H.264 codec, which is probably best known as one of the video encoding standards for Blu-ray Discs. Unfortunately, there were some bugs and incomplete features in Chrome’s support of WebRTC H.264 and this made the implementation of the simulcast difficult.
Our solution: Symphony fixed the bugs and added new features, freeing simulcast William Wallace style (albeit with slightly less mooning). The Highfive native app has H.264 support which allows for high video quality that lets you see every pimple on your interlocutors' face (probably the only downside of high quality streaming).
Bonus tech pr0n
In case you were wondering about the technologies used in these projects, here are the juicy details:
Front-end: AngularJS, WebRTC
Back-end: Jetty for the RPC server with Java 8 Guice framework for dependency injection, Java 8 Jitsi Videobridge for media transfer
Native app: Electron with customized Chromium, which comes with H.264 codec support and full simulcast with three streams
Infrastructure: coreOS with AWS EC2 instances
If you're curious about any detail that we didn't cover in this article or you can imagine yourself working on challenging but highly rewarding projects such as these, drop us a line at firstname.lastname@example.org.