ott-streaming-packager

Optional Text

OTT live streaming encoder and packager (or independent packaging) supporting ABR streaming for HLS and DASH

This application is intended to serve as a reliable and scalable OTT streaming repackager (with optional transcoding) to deliver content as part of an overall media streaming platform. There are two key variations of OTT streaming technologies that this software accommodates:

HLS (HTTP Live Streaming) - Transport Stream HLS and Fragmented MP4 HLS (CMAF style)
DASH (Dynamic Adaptive Streaming over HTTP) - Fragmented MP4

With this application, you can ingest live MPEG2 transport streams carried over UDP (Multicast or Unicast) for transcoding and/or repackaging into HTTP Live Streaming (HLS) (both TS and MP4) and DASH output container formats. The application can optionally transcode or just simply repackage. If you are repackaging then the source streams need to be formatted as MPEG2 transport containing H264/HEVC and AAC audio, however if you are transcoding then you can ingest a MPEG2 transport stream containing other formats as well. In the transcode mode, we will also process the SCTE35 messages (CUE-OUT/CUE-IN) into the HLS manifests.

There are two ways to use this application. The first and simplest method is use to the command version of the application. You can quickly clone the repository, compile and easily start streaming. The Quickstart for the web application is further down in the README and is a bit more involved to get setup and running, but provides a scriptable API as well as a nice clean interface with thumbnails and other status information in the transcoding mode. The web application is still in the early stages and I will continually be adding features for managing these types of streaming services.

I would also appreciate any funding support, even if it is a one time donation. I only work on this project in my spare time. If there are specific features you would like to see, a funding donation goes a long way to make it happen. I can also offer support services for deployment to address any devops type of issues, troubleshoot hardware (or software issues), or just offer general advice.

Quickstart (NodeJS Web Application - Ubuntu 20.04 Server/Desktop)

If something doesn't work here for you, then please post a bug in GitHub. I know this process can be a pain in the arse to get going especially given everyone has a different environment, so please be patient in following the instructions. If you do have an issue, I am more than willing to help out. I will soon be deprecating the CPU based transcoding since it requires a lot of my time to maintain both code pathways.

Please follow the directions below *very* closely:

cannonbeach@insanitywave:$ sudo apt install git
cannonbeach@insanitywave:$ sudo apt install build-essential
cannonbeach@insanitywave:$ sudo apt install libz-dev
cannonbeach@insanitywave:$ git clone https://github.com/cannonbeach/ott-packager.git
cannonbeach@insanitywave:$ cd ott-packager

*IMPORTANT* *IMPORTANT* *IMPORTANT* *IMPORTANT* *IMPORTANT*
(VERY IMPORTANT: Please advise - if you are planning to run on a NVIDIA GPU system, you need to make sure that prior to running setuptranscode.sh that the cudainclude and cudalib directories are set correctly in the script, otherwise it will fail to setup properly).  Please also make sure that MakefileTranscode also has the correct paths.

The Dockerfile is currently setup to use nvidia image for CUDA 12.1.1.  You can get the combo CUDA+Driver from here:
https://developer.nvidia.com/cuda-12-1-1-download-archive
You will download th file: cuda_12.1.1_530.30.02_linux.run
And it will install the CUDA 12.1.1 and the NVIDIA Driver

You can get updated NVIDIA patch here to enable more encoding sessions on consumer hardware.  This works extremely well and you should install it for maximum performance!!
https://github.com/keylase/nvidia-patch

If you need a different version, you need to update the Dockerfile_Transcode to the correct base image and update all of the other corresponding paths.  Email if you need help.  I have limited access to a lot of GPU hardware and I am looking to get my hands on something that supports AV1.  You can donate or provide me with cloud credits or even some other arrangement.

*IMPORTANT* *IMPORTANT* *IMPORTANT* *IMPORTANT* *IMPORTANT*

cannonbeach@insanitywave:$ chmod +x setuptranscode.sh
cannonbeach@insanitywave:$ ./setuptranscode.sh

*IMPORTANT* *IMPORTANT* *IMPORTANT* *IMPORTANT* *IMPORTANT*
(VERY IMPORTANT: If you are not compiling on an NVIDIA GPU system, when you get to the x265 setup (which is towards the end of the script execution, please set ENABLE_SHARED to OFF and set ENABLE_ASSEMLBY to ON, then hit the letter 'c' for configuration and then hit 'g' for generate and exit)
*IMPORTANT* *IMPORTANT* *IMPORTANT* *IMPORTANT* *IMPORTANT*

cannonbeach@insanitywave:$ chmod +x setupsystem.sh
cannonbeach@insanitywave:$ ./setupsystem.sh
cannonbeach@insanitywave:$ ./mkpkg.sh
cannonbeach@insanitywave:$ sudo dpkg -i fillet-1.1.deb

Then point web browser to port 8080, for example: http://10.0.0.200:8080 and the web application will come up.  If for some reason, it does not come up, you need to review the steps above to make sure you followed everything correctly.

You will notice that the Apache web server was also installed.  It allows you to easily serve content directly off the same system.  The content will be available from the directories that you specified in your configurations.

Optional Text

More advanced quickstart (Command Line Packager or Encoder Mode)

The software install guide here is for Ubuntu 20.04 server only, however, you can run this on older/newer versions of Ubuntu as well as in Docker containers for AWS/Google cloud based deployments. I do not maintain a CentOS installation guide.

There are now two versions of the application that get built. The transcode/package (fillet_transcode) and the independent packager (fillet_repackage).

The fillet application must be run as a user with *root* privileges, otherwise it will *not* work.

usage: fillet_repackage [options]

INPUT PACKAGING OPTIONS (AUDIO AND VIDEO CAN BE SEPARATE STREAMS)
       --vsources      [NUMBER OF VIDEO SOURCES - TO PACKAGE ABR SOURCES: MUST BE >= 1 && <= 10]
       --asources      [NUMBER OF AUDIO SOURCES - TO PACKAGE ABR SOURCES: MUST BE >= 1 && <= 10]

INPUT OPTIONS (when --type stream)
       --vip           [IP:PORT,IP:PORT,etc.] (THIS MUST MATCH NUMBER OF VIDEO SOURCES)
       --aip           [IP:PORT,IP:PORT,etc.] (THIS MUST MATCH NUMBER OF AUDIO SOURCES)
       --interface     [SOURCE INTERFACE - lo,eth0,eth1,eth2,eth3]
                       If multicast, make sure route is in place (route add -net 224.0.0.0 netmask 240.0.0.0 interface)

OUTPUT PACKAGING OPTIONS
       --window        [WINDOW IN SEGMENTS FOR MANIFEST]
       --segment       [SEGMENT LENGTH IN SECONDS]
       --manifest      [MANIFEST DIRECTORY "/var/www/hls/"]
       --identity      [RUNTIME IDENTITY - any number, but must be unique across multiple instances of fillet]
       --hls           [ENABLE TRADITIONAL HLS TRANSPORT STREAM OUTPUT - NO ARGUMENT REQUIRED]
       --dash          [ENABLE FRAGMENTED MP4 STREAM OUTPUT (INCLUDES DASH+HLS FMP4) - NO ARGUMENT REQUIRED]
       --manifest-dash [NAME OF THE DASH MANIFEST FILE - default: masterdash.mpd]
       --manifest-hls  [NAME OF THE HLS MANIFEST FILE - default: master.m3u8]
       --manifest-fmp4 [NAME OF THE fMP4/CMAF MANIFEST FILE - default: masterfmp4.m3u8]
       --webvtt        [ENABLE WEBVTT CAPTION OUTPUT]
       --cdnusername   [USERNAME FOR WEBDAV ACCOUNT]
       --cdnpassword   [PASSWORD FOR WEBDAV ACCOUNT]
       --cdnserver     [HTTP(S) URL FOR WEBDAV SERVER]

PACKAGING AND TRANSCODING OPTIONS CAN BE COMBINED

And for the transcode/package, usage is follows:

usage: fillet_transcode [options]


INPUT TRANSCODE OPTIONS (AUDIO AND VIDEO MUST BE ON SAME TRANSPORT STREAM)
       --sources      [NUMBER OF SOURCES - TO PACKAGE ABR SOURCES: MUST BE >= 1 && <= 10]

INPUT OPTIONS (when --type stream)
       --ip            [IP:PORT,IP:PORT,etc.] (THIS MUST MATCH NUMBER OF SOURCES)
       --interface     [SOURCE INTERFACE - lo,eth0,eth1,eth2,eth3]
                       If multicast, make sure route is in place (route add -net 224.0.0.0 netmask 240.0.0.0 interface)


OUTPUT PACKAGING OPTIONS
       --window        [WINDOW IN SEGMENTS FOR MANIFEST]
       --segment       [SEGMENT LENGTH IN SECONDS]
       --manifest      [MANIFEST DIRECTORY "/var/www/hls/"]
       --identity      [RUNTIME IDENTITY - any number, but must be unique across multiple instances of fillet]
       --hls           [ENABLE TRADITIONAL HLS TRANSPORT STREAM OUTPUT - NO ARGUMENT REQUIRED]
       --dash          [ENABLE FRAGMENTED MP4 STREAM OUTPUT (INCLUDES DASH+HLS FMP4) - NO ARGUMENT REQUIRED]
       --manifest-dash [NAME OF THE DASH MANIFEST FILE - default: masterdash.mpd]
       --manifest-hls  [NAME OF THE HLS MANIFEST FILE - default: master.m3u8]
       --manifest-fmp4 [NAME OF THE fMP4/CMAF MANIFEST FILE - default: masterfmp4.m3u8]
       --webvtt        [ENABLE WEBVTT CAPTION OUTPUT]
       --cdnusername   [USERNAME FOR WEBDAV ACCOUNT]
       --cdnpassword   [PASSWORD FOR WEBDAV ACCOUNT]
       --cdnserver     [HTTP(S) URL FOR WEBDAV SERVER]

OUTPUT TRANSCODE OPTIONS
       --transcode   [ENABLE TRANSCODER AND NOT JUST PACKAGING]
       --gpu         [GPU NUMBER TO USE FOR TRANSCODING - defaults to 0 if GPU encoding is enabled]
       --select      [PICK A STREAM FROM AN MPTS- INDEX IS BASED ON PMT INDEX - defaults to 0]
       --outputs     [NUMBER OF OUTPUT LADDER BITRATE PROFILES TO BE TRANSCODED]
       --vcodec      [VIDEO CODEC - needs to be hevc or h264]
       --resolutions [OUTPUT RESOLUTIONS - formatted as: 320x240,640x360,960x540,1280x720]
       --vrate       [VIDEO BITRATES IN KBPS - formatted as: 800,1250,2500,500]
       --acodec      [AUDIO CODEC - needs to be aac, ac3 or pass]
       --arate       [AUDIO BITRATES IN KBPS - formatted as: 128,96]
       --aspect      [FORCE THE ASPECT RATIO - needs to be 16:9, 4:3, or other]
       --scte35      [PASSTHROUGH SCTE35 TO MANIFEST (for HLS packaging)]
       --stereo      [FORCE ALL AUDIO OUTPUTS TO STEREO- will downmix if source is 5.1 or upmix if source is 1.0]
       --quality     [VIDEO ENCODING QUALITY LEVEL 0-3 (0-BASIC,1-STREAMING,2-BROADCAST,3-PROFESSIONAL)
                      LOADING WILL AFFECT CHANNEL DENSITY-SOME PLATFORMS MAY NOT RUN HIGHER QUALITY REAL-TIME

H.264 SPECIFIC OPTIONS (valid when --vcodec is h264)
       --profile     [H264 ENCODING PROFILE - needs to be base,main or high]

Simple Repackaging Command Line Example Usage:

cannonbeach@insanitywave:$ sudo ./fillet_repackage --vsources 2 --vip 0.0.0.0:20000,0.0.0.0:20001 --asources 2 --aip 0.0.0.0:20002,0.0.0.0:20003 --interface eno1 --window 10 --segment 2 --hls --manifest /var/www/html/hls

This command line tells the application that there are two unicast sources that contain audio and video on the loopback interface. The manifests and output files will be placed into the /var/www/html/hls directory. If you are using multicast, please make sure you have multicast routes in place on the interface you are using, otherwise you will *not* receive the traffic.

This will write the manifests into the /var/www/html/hls directory (this is a common Apache directory).

cannonbeach@insanitywave:$ sudo route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0

In addition to having the multicast route in place, you may also need to turn off the reverse-path filter for multicast traffic: In /etc/sysctl.conf, there are multiple entries that control reverse-path filtering. In some instances depending on how your network is setup and where the source is coming from, you may have to disable reverse-path filtering. Older variations of Linux had this enabled by default, but it can cause issues with multicast coming from a different subnet.

net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.eth0.rp_filter = 0
and so on...

After you've made those changes, please run the following for the changes to take effect

sudo sysctl -p

You should also be aware that the fillet application creates a runtime cache file in the /var/tmp directory for each instance that is run. The cache file is uniquely identified by the "--identity" flag provided as a parameter. It follows the format of: /var/tmp/hlsmux_state_NNNN where NNNN is the identifier you provided. If you want to start your session from a clean state, then you should remove this state file. All of the sequence numbering will restart and all statistics will be cleared as well. It is also good practice when updating to a new version of the software to remove that file. I do add fields to this file and I have not made it backwards compatible. I am working on making this more configurable such that it can be migrated across different Docker containers.

Transcoder and Packager Operation

The packager has several different optional modes:

Standard Packaging (H.264/AAC and/or HEVC/AAC)
Transcoding+Packaging Mode 1 (H.264/AAC) - HLS(TS,fMP4) and DASH(fMP4)
Transcoding+Packaging Mode 2 (HEVC/AAC) - HLS(fMP4) and DASH(fMP4) - NO TS MODE FOR HEVC In the transcoding modes, the packaging is bundled together with the transcoding, but I am exploring options to separate the two for a post v1.0 release.

This is a budget friendly packaging/transcoding solution with the expectation of it being simple to setup, use and deploy. The solution is very flexible and even allows you to run several instances of the application with different parameters and output stream combinations (i.e., have a mobile stream set and a set top box stream set). If you do run multiple instances using the same source content, you will want to receive the streams from a multicast source instead of unicast. The simplicity of the deployment model also provides a means for fault tolerant setups.

A key value add to this packager is that source discontinuities are handled quite well (in standard packaging mode as well as the transcoding modes). The manifests are setup to be continuous even in periods of discontinuity such that the player experiences as minimal of an interruption as possible. The manifest does not start out in a clean state unless you remove the local cache files during the fast restart (located in /var/tmp/hlsmux...). This applies to both HLS (handled by discontinuity tags) and DASH outputs (handled by clever timeline stitching the manifest). Many of the other packagers available on the market did not handle discontinuties well and so I wanted to raise the bar with regards to handling signal interruptions (we don't like them, but yes they happen and the better you handle them the happier your customers will be). If the source signal goes away in transcoding mode, the software will backfill by repeating video frames and filling the audio with silence. This is great if you have a short signal interruption and need a way to maintain your output.

Another differentiator (which is a bit more common practice now) is that the segments are written out to separate audio and video files instead of a single multiplexed output file containing both audio and video. This provides additional degrees of freedom when selecting different audio and video streams for playback (it does make testing a bit more difficult though).

H.264 Transcoding Example

cannonbeach@insanitywave:$ ./fillet_transcode --sources 1 --ip 0.0.0.0:5000 --interface eth0 --window 20 --segment 2 --identity 1000 --hls --dash --transcode --outputs 2 --vcodec h264 --resolutions 320x240,960x540 --manifest /var/www/html/hls --vrate 500,2500 --acodec aac --arate 128