Skip to main content

Featured Post

OpenCV 4.5 simple optical flow GPU tutorial cuda::FarnebackOpticalFlow

This OpenCV tutorial is a very simple code example of GPU Cuda optical flow in OpenCV written in c++. The configuration of the project, code, and explanation are included for farneback Optical Flow method. Farneback algorithm is a dense method that is used to process all the pixels in the given image. The dense methods are slower but more accurate as all the pixels of the image are processed. In the following example, I am displaying just a few pixes based on a grid. I am not displaying all the pixes. In the opposite to dense method the sparse method like Lucas Kanade using just a selected subset of pixels. They are faster. Both methods have specific applications. Lucas-Kanade is widely used in tracking. The farneback can be used for the analysis of more complex movement in image scene and furder segmentation based on these changes. As dense methods are slightly slower, the GPU and Cuda implementation can lead to great performance improvements to calculate optical flow for all pixels o

Opencv HSL video stream to web

This tutorial will show you all components, configuration, and code needed to steam video output results from Opencv C++ to your Web player. The C++ program will take any video input and process this video. The processed video will be stream outside of OpenCV using the GStreamer pipeline (Windows part). The HLS video produces one Playlist file and several MPEG-TS segments of videos. This several HLS outputs are stored in the Windows file system. I am using WSL 2, windows subsystem for Linux to run Ubuntu distribution. Here the NGINX is installed with the RTMP module. NGINX is distributing a video stream from the windows file system to the web.  Let's have a look at this in more detail. 

What is covered?

  • Opencv C++ part + GStreamer pipeline
  • NGINX configuration
  • Architecture
  • Web Player for your HLS stream
What is not covered?

opencv web stream

The architecture of the streaming system

This reads video from file or video stream by VideoCapture cap("/samp.MOV");
Video is processed in the c++ app. In my case, there is a very simple Yolo Darknet detector implemented. The results of the detector are written into the video frame. The processed video frame is pushed to GSTREAMER pipeline used in OpenCV VideoWriter writer. The HLS pipeline is storing HLS fragments and playlist on the windows file system. The NGINX server in UBUNTU WLS 2 distributes this content from the windows file system to the web. This is possible as the file system from windows c:/ is reachable under /mnt/c/.

opencv hls streaming

Installation and prerequisites

1) Compile OpenCV with GStreamerGStreamer OpenCV on Windows or Here.

2) Enable WSL 2 virtualization in windows. Install Ubuntu 20.04.1. I tested WSL 2. I am not sure about the WSL 1 file system sharing with Windows. 

3) NGINX with RTMP module in UBUNTU WSL 2

There are some standard prerequisites like Visual Studio 2019. Knowledge of Linux and be able to find documentation to enable WSL 2, Install NGINX with RTMP module. It is a lot. 

HLS protocol brief introduction 

HLS stands for HTTP Live Streaming. The protocol was introduced by Apple and become quite popular over the years. The major advantage is the ability to adapt the video streams based on network throughput. The HLS in version lower than 4 is putting video into the MPEG-TS containers. This file with .TS is holding the actual media content. The TS are segments of videos with defined length. The video segments are organized in the playlist. The playlist holds the reference to the MPEG-TS segments. 

Advantage of HTTP streaming

One of the significant advantages is the HTTP protocol itself. The HTTP can traverse easily the firewall and proxy servers. This is the main advantage over the UDP based RTP and RTSP protocol. The standard support as well as a secured transfer based on HTTPS.

HLS MPEG-TS

The video in GStreamer is encoded by H.264 together with other audio coded and placed into a file container called MPEG-2 Transport Stream. The stream chunks are of the same length defined in GStreamer.  

HLS playlist M3U8

This simply refers to concrete file.TS to play the actual video content.

GStreamer opencv pipeline

GStreamer needs to be properly set up, all environmental path variables need to be correct and OpenCV needs to be compiled with opencv. ! To use x264enc and his use full installation of GStreamer and not the recommended one. Standard OpenCV VideoWriter pipeline can look like this. Appssrc means that your C++ OpenCV program is the source for GStreamer. Then the following values are video conversion and scale. The x264enc is the encoding of the video output and packed in mpegtsmux containers (ts segments). HLSSINK then produces an HLS playlist and TS segments from your video. You need to point to the location shared with your UBUNTU in location and playlist. This will produce several ts segments and one playlist. The Playlist root is the value that needs to be updated based on IP and PORT. 172.22.222.39:8080 is IP and PORT depends on the NGINX and UBUNTU networking setup. This value is stored in the playlist and points to concrete TS segments. Both Playlist and Segments then need to be correctly served by the NGINX server.

Videowriter GStreamer pipeline example

VideoWriter writer(
        "appsrc ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 
! x264enc ! mpegtsmux !
hlssink playlist-root=http://172.22.222.39:8080/live/
 location=C:/streamServer/hls/segment%05d.ts
 playlist-location=C:/streamServer/hls/playlist.m3u8",
        0,
        20,
        Size(800600),
        true);

Just remember to update locations and IP and PORTS in this example as described in previous text.

How to deliver a Playlist and content?

The content stored in windows location C:/streamServer/hls/ needs to be enabled and rechable by simple web player. This requires server to provide a few thinks. 1) Web page with your simple video player. 2) The hls playlist and ts segments are reachable under some IP address. This task is fully under control of NGINX server running under Ubuntu in WSL 2. The NGINX needs to be compiled with RTMP module. 

I am not expert on NGINX. HLS is on in my rtmp module and hls_path poinst to location of content generated from opencv. 

hls_path /mnt/c/streamServer/hls

In part related to http is server enabled to port 8080 of local host, where index.html is available under / location and hls playlist is available under /live

NGINX config for HLS streaming


#user  nobody;
worker_processes  auto;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;
#pid        logs/nginx.pid;


events {
    worker_connections  4;
}
#
rtmp { 
    server { 
        listen 1935
        application live { 
            live on
            allow publish 127.0.0.1;
            allow publish all;
            allow play all;
            interleave on;
            hls on
            #Path where fragments want to go
            hls_path /mnt/c/streamServer/hls
            hls_fragment 15s
            dash on;
            dash_path /mnt/c/streamServer/dash;
            dash_fragment 15s;
        } 
    } 

http {
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout 65;
    server {
        listen       8080;
        server_name localhost;
        location / {
            root   /mnt/c/streamServer;
            index  index.html index.htm;
        }
         location /live {
             types {
                application/vnd.apple.mpegurl m3u8;
                video/mp2t ts;
                text/html html;
            }
            alias   /mnt/c/streamServer/hls;
            add_header Cache-Control no-cache;
          }
          location /tvds {
            root   /mnt/c/streamServer/dash;
            add_header Cache-Control no-cache;
          }
    }
}

Web player

I just found some example based on video.js and videojs-contrib-hls/5.14.1/videojs-contrib-hls.js specifically for HLS. All you need to change is IP:PORT address in video html element src="http://172.22.222.39:8080/live/playlist.m3u8. This is exactli the same location set by NGINX to serve playlist.

         location /live {
             types {
                application/vnd.apple.mpegurl m3u8;
                video/mp2t ts;
                text/html html;

<!-- CSS  -->
<link href="https://vjs.zencdn.net/7.2.3/video-js.css" rel="stylesheet">

<!-- HTML -->
<video id='hls-example'  class="video-js vjs-default-skin" width="800" height="600" controls>
<source type="application/x-mpegURL" src="http://172.22.222.39:8080/live/playlist.m3u8">
</video>

<!-- JS code -->
<!-- If you'd like to support IE8 (for Video.js versions prior to v7) -->
<script src="https://vjs.zencdn.net/ie8/ie8-version/videojs-ie8.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs
/videojs-contrib-hls/5.14.1/videojs-contrib-hls.js"></script>
<script src="https://vjs.zencdn.net/7.2.3/video.js"></script>

<script>
var player = videojs('hls-example');
player.play();
</script>

C++ opencv gstreamer windows part

The input video from video capture is process by yolo darknet neural network. you can find more about this Yolo darknet detection. VideoWriter writer is using discussed gstreamer pipeline and once the video is processed the output us pusth to pipeline just like this.

resize(img, img, Size(800600));
        writer.write(img);

Just be sure that your size of input image and sizes in your pipeline match. Remember to change IP and PORT as disccused in chapter anout pipeline and NGINX.


#include <iostream>
#include <opencv2/opencv.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/video.hpp>
#include <opencv2/dnn.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/core.hpp>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/photo.hpp>

using namespace cv;
using namespace std;
using namespace dnn;

int main()
{
    VideoWriter writer(
        "appsrc ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 
! x264enc ! mpegtsmux ! hlssink playlist-root=http://172.22.222.39:8080/live/
 location=C:/streamServer/hls/segment%05d.ts
 playlist-location=C:/streamServer/hls/playlist.m3u8 ",
        0,
        20,
        Size(800600),
        true);

    VideoCapture cap("/samp.MOV");
    std::string model = "/yolov3-tiny.weights";  
    std::string config = "/yolov3-tiny.cfg"; 
    Net network = readNet(model, config, "Darknet");
    network.setPreferableBackend(DNN_BACKEND_DEFAULT);
    network.setPreferableTarget(DNN_TARGET_OPENCL);

    for (;;)
    {
        if (!cap.isOpened()) {
            cout << "Video Capture Fail" << endl;
            break;
        }
        Mat img;
        cap >> img;
        static Mat blobFromImg;
        //blobFromImage(img, blobFromImg, 1.0, Size(Width, Height), Scalar(), swapRB, false, CV_8U);
        //network.setInput(blobFromImg, "", scale, mean);
        bool swapRB = true;
        blobFromImage(img, blobFromImg, 1Size(416416), Scalar(), swapRB, false);
        cout << blobFromImg.size() << endl;
        float scale = 1.0 / 255.0;
        Scalar mean = 0;
        network.setInput(blobFromImg, "", scale, mean);
        Mat outMat;
        network.forward(outMat);
        // rows represent number of detected object (proposed region)
        int rowsNoOfDetection = outMat.rows;

        // The columns looks like this, The first is region center x, center y, width
        // height, The class 1 - N is the column entries, which gives you a number, 
        // where the biggest one corresponding to most probable class. 
        // [x ; y ; w; h; class 1 ; class 2 ; class 3 ;  ; ;....]
        // [x ; y ; w; h; class 1 ; class 2 ; class 3 ;  ; ;....]
        int colsCoordinatesPlusClassScore = outMat.cols;


        // Loop over the number of the detected objects. 
        for (int j = 0; j < rowsNoOfDetection; ++j)
        {
            // for each row, the score is from element 5 up to number of classes
  //index (5 - N columns)
            Mat scores = outMat.row(j).colRange(5, colsCoordinatesPlusClassScore);
            Point PositionOfMax;
            double confidence;

            // This function find indexes of min and max confidence and related
// index of element. 
            // The actual index is matched the concrete class of the object.
            // First parameter is Mat which is row [5fth - END] scores,
            // Second parameter will give you the min value of the scores. NOT needed 
            // confidence gives you a max value of the scores. This is needed, 
            // Third parameter is index of minimal element in scores
            // the last is the position of the maximum value. This is the class!!
            minMaxLoc(scores, 0, &confidence, 0, &PositionOfMax);

            if (confidence > 0.0001)
            {
                int centerX = (int)(outMat.at<float>(j, 0) * img.cols); 
// thease four lines are
                int centerY = (int)(outMat.at<float>(j, 1) * img.rows); 
// first column of the tow
                int width = (int)(outMat.at<float>(j, 2) * img.cols + 20); 
// identify the position 
                int height = (int)(outMat.at<float>(j, 3) * img.rows + 100); 
// of proposed region
                int left = centerX - width / 2;
                int top = centerY - height / 2;


                stringstream ss;
                ss << PositionOfMax.x;
                string clas = ss.str();
                int color = PositionOfMax.x * 10;
                putText(img, clas, Point(left, top), 12Scalar(color, 255255), 2false);
                stringstream ss2;
                ss << confidence;
                string conf = ss.str();
                
                rectangle(img, Rect(left, top, width, height), Scalar(color, 00), 280);
            }
        }
        resize(img, img, Size(800600));
        namedWindow("Display window", WINDOW_AUTOSIZE);// Create a window for display.
        imshow("Display window", img);
        waitKey(25);
        resize(img, img, Size(800600));
        writer.write(img);
    }

}

Youtube tutorial about problems with Opencv HLS streaming

This tutorial will show you the main problems during the development of such a streaming service.


Comments

Popular

Opencv GStreamer (windows) video streaming tutorial + full source code for RTSP HLS streaming

Opencv C++ simple tutorial to use GStreamer to send video to Server that converts RTSP to HLS video stream. The source code and all needed configurations are included.  O pencv is a powerful computer vision library. You can use it in production and use it for image and video processing and modern machine learning. In some applications, You may want to stream your processed video results from your C++ OpenCV app outside and not just use a simple OpenCV graphical interface. The video streaming of your results is what you are looking for. Do you want to stream processed video from your IoT device? Yes, This is mainly for Linux. Do you want to stream processed video to the Web player, broadcast the video or just use VLC to play video processed by OpenCV? You may be interested in reading the next lines.  Opencv video stream to VLC or WEB There are basically two main options with OpenCV. The first one is to write a streaming application using FFMPEG. This is a little bit more advanced appro

Opencv Web camera and Video streams in Windows subsystem for Linux WSL, by FFmpeg and GStreamer

Opencv in Windows Subsystem for Linux (WSL) is a compatibility layer designed to running Linux binary executables (in ELF format) natively on Windows 10. I love it. There are some limitations to mention. The first biggest is the lack of support of CUDA, which could be a limitation for deep learning application and learning in WSL. The second trouble for Opencv development is the lack of Web camera support. This suspends WSL almost on a useless level for me until now.  VideoCapture cap;   is not working in WSL for now cap.open(0);  FFMPEG to WSL opencv program and back to WEB browser in windows This Video capture is right now not possible at in Ubuntu running under Windows (WSL). I will hit this limitation in this article. I will show you how to reach a video camera and learn something more about video streaming. Yes, the opencv processed frames will be stream to the web player on simple web site. Check the goal of this opencv tutorial on this video What you will learn

Compile Opencv with GStreamer for Visual Studio 2019 on windows 10 with and contribution modules

The goal of this tutorial is a simple step by step compilation of Opencv 4.2 with contribution extra modules with GStreamer as a bonus. The environment is Windows 10, Visual Studio 2019 C++ application. This took me almost one day of correcting of CMake setting. The goal of this tutorial is: compiled a set of OpenCV libraries with GStreamer and FFmpeg on Windows. I focus mainly on GStreamer. It is a little bit more tricky. You will reach the following information about your Opencv environment by compile and run this simple code. The Opencv GStreamer is turned as YES. GStreamer gives you a great opportunity to stream OpenCV output video outside of your program, for example, web application. I recently compiled with opencv 4.4. The update at the end of the post.  It is working!! wow, The working app and configuration in future tutorials. #include   <opencv2/opencv.hpp> #include   <iostream> using namespace cv; int   main () {      std ::cout <<  &q

OpenCV 4.5 simple optical flow GPU tutorial cuda::FarnebackOpticalFlow

This OpenCV tutorial is a very simple code example of GPU Cuda optical flow in OpenCV written in c++. The configuration of the project, code, and explanation are included for farneback Optical Flow method. Farneback algorithm is a dense method that is used to process all the pixels in the given image. The dense methods are slower but more accurate as all the pixels of the image are processed. In the following example, I am displaying just a few pixes based on a grid. I am not displaying all the pixes. In the opposite to dense method the sparse method like Lucas Kanade using just a selected subset of pixels. They are faster. Both methods have specific applications. Lucas-Kanade is widely used in tracking. The farneback can be used for the analysis of more complex movement in image scene and furder segmentation based on these changes. As dense methods are slightly slower, the GPU and Cuda implementation can lead to great performance improvements to calculate optical flow for all pixels o

Opencv C++ Tutorial Mat resize

Opencv Mat Resize   Resize the Mat or Image in the Opencv C++ tutorial. It is obviously simple task and important to learn. This tutorial is visualized step by step and well-described each of them. The main trick is in that simple code. Mat Input; Mat Resized; int ColumnOfNewImage = 60; int RowsOfNewImage = 60; resize(Input, Resized, Size( ColumnOfNewImage , RowsOfNewImage )); This code just takes an Input image and resized save to output Mat. How big is the resized image is based on the Size? Size just contains two parameters. Simple numbers of how the result should be big. The simple number of columns (width) and rows (height). That is basically it. Enjoy                                                 Boring same face again and again.  Load Image, resize and save Opencv C++ #include <Windows.h> #include "opencv2\highgui.hpp" #include "opencv2\imgproc.hpp" #include "opencv2\video\background_segm.hpp" #include &qu

Opencv C++ tutorial : Smoothing, blur, noise reduction / canceling

Smooth or blur, gaussian blur, and noise-canceling, This tutorial will learn OpenCV blur, GaussianBlur, median blur functions in C++. Additionally, the advanced technique for noise reduction  fastNlMeansDenoising family  will be introduced with a code example for each method.   You can use blurring of the image to hide identity or reduce the noise of the image.  Blur can be a very useful operation and it is a very common operation as well. For example, the anonymization of pedestrians, face or is one possible target for blue operation. The blur is the most common task to perform over the image to reduce noise. The noise reduction is more task for Gaussian blur than for simple blur operation. The various blur operations are very common for image processing on mobile devices.  The more important is the robustness issues of the data in pre-processing for machine learning. Sometimes, by blurring the images of the dataset can have a positive effect on the robustness of the achieved de

Opencv tutorial RTMP video streaming to NGINX restream as HLS

Video streaming Tutorial of sending processed Opencv video to NGINX and distributing video from NGINX (broadcast) by HLS stream for a wider audience, like multiple web players, VLC, or any other video stream receiver. Opencv application HLS streaming by GStreamer and NGINX  We will use GStreamer to send video from the Opencv application by rtmp2sink to an RTMP module in NGINX. In our example, the server is a widely used NGINX server with an Nginx-RTMP-module. The NGINX will receive RTMP video from Opencv and restream as an HLS video stream considered for multiple end consumers. This is a follow-up to the previous article about Video streaming from Opencv to RtspSimpleServer by rtsp protocol.   The goal is the same. Send video from Opencv to the server and restream the video for a wider audience. The difference is that RtspSimpleServer running on windows, NGINX is running in docker (WSL2). The one-to-one communication between Opencv and RtspSimpleServer was established by RTSP protocol

Opencv 4 C++ Tutorial simple Background Subtraction

This method is used to learn what belongs to the background of the image and what belongs to the foreground. The static cameras that monitor the area can very easily recognize, what is part of the image that is always here or there is something that is new and moving over the background.  Background subtraction Visual studio 2019 project setup If you have Opencv 4+ compiled or installed only steps you need to do is set the include directory with OpenCV header files. Set the Additional library Directories that point to \lib folder. Just note that Visual Studio 2019 should use VC16 \lib. Finally, As additional dependencies, specify the libs used to resolve the function implementation in the code. The list for Opencv 420 is here. The different version of opencv is using different numbering for example opencv 440 will use opencv_core440.lib.  opencv_bgsegm420.lib opencv_core420.lib opencv_videoio420.lib opencv_imgproc420.lib opencv_highgui420.lib opencv_video420.lib  Background sustract

Opencv VideoCapture File, Web Camera, RTSP stream

Opencv VideoCapture File, Camera and stream Opencv tutorial simple code in C++ to capture video from File, Ip camera stream and also the web camera plug into the computer. The key is to have installed the FFMPEG especially in case of reading the stream of IP cameras. In windows just use Opencv Installation by Nugets packages  Here . Simple easy under 2 minutes of installation. In Linux, you need to follow the instruction below. If you are on Debian Like package system. Under Fedora Red hat dist just use a different approach. Code is simple and installation is the key..  Windows use nugets packages Linux you have to install and build Opencv With FFMPEG. Also simple.  It is easy to capture video in OpenCV Video capture  in OpenCV is a really easy task, but for a little bit experienced user.  What is the problem? The problem is the installation of Opencv without recommended dependencies. Just install all basic libs that are recommended on the website. # Basic packa