We realised this was not a new problem. Clouds vendors have been selling this potential flexibility for years so we decided to experiment in using the public cloud to do the heavy lifting in terms of live video production.
We care about four types of primary media:
For camera and cook with mainly use raspberry pies with camera modules. These raspberry pies have been configured as RTMP servers that can be connected to from OBS.
For audio capture we generate another video feet with a static background. This allows us to create the same RTMP server that we use for the camera but instead use it primarily to open the audio from a USB sound device.
For presenter screens, we use a Lenkeng HDMI extender, again these are connected to a Raspberry Pi over ethernet.
In terms of bitrates, we tend to send about 2MBps for each video feed. Audio feeds can be around 192kbps given that the video element in these feeds is a static image.
We do however store raw media on a storage device connected locally to the Raspberry Pi. This allows us to use higher quality assets for the final edit.
Interstitials, lower thirds and backgroud media are stored on the machine running OBS.
OBS is Flexible, what it can do and what it allows us to do …
Remote Desktop in Linux isn’t quite as good as you would assume. It took some testing, but eventually we found a good pairing of Xrdp and Remina.
Xrdp is an RDP layer on top of VNC. Allowing us to utilise better compression and display state management over a network.
Remina is a fairly modern Remote Desktop client with support for a few Remote Desktop protocols.
With all capture devices configured to connect to a central OpenVPN Server. We can configure the AWS VPC routing tables to route all traffic to the private OpenVPN subnet, this allows us to connect our Cloud OBS instance to the camera devices transparantly.
Where RDP is reasonably for overall control of OBS, it can often provide a frustrating experience with a low bitrate. Instead, we configured some local hardware to provide this level of control.
A lightweight Midi to Websocket bridge allows us to control OBS using a midi controller. We can make use of push-buttons for scene control, while also using dials to manage volume, and pan/tilt/zoom.
For video feed monitoring, we can use relatively inexpensive 10” 1080p monitors connected to an HDMI switch. This provides real-time feedback as to whether our cameras configured appropriately.
For stream monitoring, we are able to take advantage of the netowrk connectivity on AWS. Using the Telegraf/ InfluxDB / Grafana (TIG) Stack, we are able to build a lightwieght dashboard for system health.
As I’ve alluded to in previous talks, we can also use InfluxDB to storage business events. Every time an operator presses a button to change scene, each time a new raw file is created, these events are all stored in a custom database in InfluxDB. Exporting these to CSV super helpful the post-processing phase.
]]>This is a great opportunity to write this small article, outlining our Voctomix configuration, some of the learnings that we have made during Linux App Summit and detail some of the plans that we have going forward.
Tuxedo computers have been very supportive of our efforts towards free software live streaming and as such, have loaned us 2 of their laptops to use during events and drive our solution forward.
Traditionally, one of the primary constraints that we’ve worked around is CPU power, leading us to look into solutions that allow us to delegate CPU and GPU intensive workloads to post-processing. Having access to these laptops allowed us to live-stream the event.
Linux App Summit was a great opportunity to test and develop our current configuration. We’ve found that free software events are great milestones to work towards.
https://github.com/voc/voctomix
Voctomix was a free software tool developed by the team behind the Chaos Communication Congress event. Voctomix is a collection of components, written in Python that that co-ordinate GStreamer pipelines across multiple devices. From a Live Streaming point of view, Voctomix provides a video mixer capable of mixing 2 live cameras and 1 screen grabbing input.
Some of the key capabilities of Voctomix that matter to us:
For Linux App Summit, we decided to configure 1 camera to track the presenter with another to show a wide view of the conference hall. Alongside the two camera inputs, we also configured an HDMI Grabber to capture the output of the presenter’s laptop.
As for output feeds, we capture the Voctomix output and split it into 2 feeds
A common feature of most of our video systems is that we normally capture video or audio on different nodes on the same LAN. Where it’s possible to send raw video directly to the Voctomix server in raw format, the bandwidth required to send 1080p video at 30 frames per second is fairly prohibitive on most commodity hardware.
As we normally don’t work with broadcast-quality cameras, it’s possible to take a small hit on perceived video quality to solve this problem. We can encode video to H264 on a low-cpu preset and send video to a shared RTMP server on the same LAN.
In this configuration, we run a pre-configured Docker container to manage live transport of video from the source hosts.
https://hub.docker.com/r/tiangolo/nginx-rtmp/
This container is run as a service on the same machine running Voctomix to limit the impact to latency.
1 | docker run -d -p 1935:1935 --name nginx-rtmp tiangolo/nginx-rtmp |
Firewire as a technology is still impressive by today’s standards. We can rely on Firewire as a transport mechanism for 1080i video, it is, however, unlikely that we will find modern hardware with a firewire port.
For this event, we were using the firewire output of a Canon HDV30 camera. This was connected to a Lenovo X200 Laptop with a Firewire X200 ExpressCard. We take the video from the firewire input, then using FFmpeg, convert this to H264 and send it to the RTMP Media Bus that we are running on the video mixer.
On the mixer laptop, we can consume the RTMP feed, convert it back to raw video and pass this directly on to the Voctomix Server.
1 | ffmpeg \ |
1 | confdir="`dirname "$0"`/../" |
As a second camera input, we used a C920 webcam. The C920 is commonly used by game streamers on twitch and has been designed to make several assumptions regarding lighting and focus and maximise the quality in the majority of cases.
1 | confdir="`dirname "$0"`/../" |
Similar to the way that we can ingest video feeds into Voctomix, we can consume the output stream in the same way. We can use either FFMPEG or GStreamer to consume these feeds, process, then either stream to another location or save to a local file. In this case, we worked with 2 consumers.
1 | while [ 0 ] |
1 | while [ 0 ] |
Following the event, we’re able to focus on 3 primary challenges:
We had some challenges around audio input, we’re going to be trying a few strategies to improve audio capture across multiple devices. We would traditionally deploy several boundary microphones as a backup, I’m interested in finding a more practical way to utilise these on a live stream.
Voctomix has been built following a microservice style approach, this gives us some flexibility around the deployment of components. I would like to apply some DevOps principals to how we develop our video pipeline, using tools such as Ansible to deploy changes across each device. It would also be sensible to build system packages for some of our components.
Post-processing is currently a very manual process. This year, we looked into storing all mixer events in a time-series database while retaining all raw footage for later processing. I’m planning to investigate how we can automatically generate a Kdenlive file from this data, allowing us to “Re-Master” our live events.
]]>Back in November, I was fortunate to be involved in the Linux App Summit conference in Barcelona. Linux App Summit was a joint venture between multiple communities, including KDE and Gnome.
LAS was a great opportunity to use the event infrastructure components that we’ve been building for KDE over the past few years. Now that we’ve been able to test both Frab and our Event Registration tooling with more communities, we’ve been able to mature our requirements some more for further development in the coming months.
LAS was run as a single-track conference event with breakout sessions during each day. On doing this, I was really impressed with the content structure and themes during each day.
During the event, I spent some time working on the video recording of the talks. Although we had some issues with the audio feed, we had a great opportunity to test out a Voctomix based streaming configuration. We currently have a couple of Laptops on loan from Tuxido computers, I’ll be publishing an entry on this with more details next week.
Thanks as always for KDE e.V. for giving me the opportunity to work on really fun projects like LAS, always giving me more scope to improve our event infrastructure and work with amazing people.
]]>