Redistributing live video with nginx-rtmp

On the Company I'm currently working, we have over two dozen offices spread around the country. As a public company, we have limited budget when it comes to things like links and bandwidth, so every one of those offices gets an MPLS link to the Company headquarters in Brasilia, but a limited one. As an example, the largest link we have is around 16 Mbps, for an office with around 150 employees, with the same amount of VoIP headsets and workstations. How to deliver live, high definition video to all these people under this circumstances?

Distributing video locally

With enough bandwidth, we could just stream to youtube and call it a day, but with the limited amount we have, 3 or 4 users streaming simultaneously would be capable of totally consuming all of the resources, stopping all the activities. Even thought we have low bandwidth, we have a local server on each office, running Xen/Debian. This opens a lot of possibilities for solving the streaming problem.

RTMP, HLS and other acronyms

The structure built to stream starts with OBS, an incredible open source software designed to capture and stream video on multiple platforms. It receives the video stream from cameras running RTMP, or even locally attached via USB or FireWire.

In our setup, we use a common security camera, bought because of it's pan, tilt and zoom features, that were more than enough for our headquarters auditorium. This camera provides a RTMP feed, which is captured and mixed on OBS with the screen of a desktop dedicated to the slides and multimedia content presented.

So far, not that different from what would be done with youtube or facebook live. The big cost saving feature of our setup is an internal nginx server running the nginx-rtmp-module. This module enables nginx to receive, process and re-stream video content. It can be configured to relay video on RTMP, HLS and MPEG-DASH formats, enabling a web browser to directly connect to it and also to forward video streams to other servers. With a server in each office, we just need an instance of a nginx server in each office. The whole architecture looks like this:

The RTMP module introduces an configuration block prefaced by (guess what?) RTMP, with all of the server configurations. Here, we are going to focus only on the video redistribution settings, as the documentation on github is very good. Below, a sample of an application called 720p, which pushes video to servers stream01, stream02 and stream03:

rtmp_auto_push on;
rtmp_socket_dir /tmp;
rtmp {
    access_log access.log combined;
    server {
        listen 1935;
        chunk_size 4000;

        application 720p {
            live on;
            hls on;
            push rtmp://stream01.evandrofisi.co/720p;
            push rtmp://stream02.evandrofisi.co/720p;
            push rtmp://stream03.evandrofisi.co/720p;
            hls_path /tmp/720p;
            hls_fragment 3;
            hls_playlist_length 60;
        }
    }
}

Besides pushing video to other RTMP servers, this block also serves the video content over http, using HLS, so a client accessing through a web browser with the URL http://streammaster.evandrofisi.co/720p/video.m3u8 will receive a HLS playlist, indicating the path for each video fragment generated live.

Front end AND finding content

So, with this setup we have video being distributed and accessible via a web browser, but, how about the clients? We've already established that we are using a web compatible protocol to deliver, but to certify cross compatibility, we use a simple player based around videojs, using a http-streaming extension. As we currently only have two streaming channels that never broadcast simultaneously, an static client pointing to a single URL solves the issue.

But still, how to access content on the closest server and avoid the network overload of hundreds of persons using a single 16Mbps link? We have two options, with varying levels of complexity.

Content redirect and a bunch of 302s

A simple strategy consists of having a redirector script. We developed a simple php script that detects the client ip source. As each office has a /22 network, we simply catalogue each one of them, and redirect the client to the closest server.

One big downside of this strategy is that we still have a single point of failure, on the redirector, and even with kinda large fragments, the redirector receives a lot of connections, and this method does not scale very easily. One advantage is that as we have a single distribution point, it is relatively simple to count live audience.

DNS Views, or how the grownups do video distribution

A more complex strategy consists of using DNS for content location. This is the preferred method on most CDNs. It consists of answering the client's DNS query according to it's IP address, with the closest mirror/server. This technique has the main advantage of being transparent to the application layer, but necessitates access to the DNS server, and also, as a downside, if live audience numbers are necessary, code running on the backend must be capable of centralizing such values.