Wednesday, December 23, 2015

Opencv tutorial, Combine two overlapping videos frames into one, MatchTemplate example

Combine two overlapping frame by simple correlation

Share this for more tutorials and computer vision post from me.. Thanks best Vladimir
Merge Two mat video frames

Today morning, I read this post combine two overlapping video frame on stackoverflow. 
Problem is to merge 2 images together or two video stream. I am thinking about the most simple not so much dummy solution. I find really simple solution that can be extend to more complex one. 

First of all, I took 2 images from phone like on picture (color images). Program selects Rectangles region from both source images and resize end extract this roi rectangles. Idea is find the "best" overlapping Rect regions by normalized correlation and combine this images on place with maximal correspondence.

Merge Two mat video frames

Merge Two mat video frames

I know well solution by SIFT and SURF, but it is little bit laborious than this.

It is not the best solution but one of the most simple. If your cameras are fixed in stable position between themselves. This is good solution i thing. I hold my phone in hand :)

You can also use this simple approach on video. The speed depends only on number of rectangle candidate you want to compare.

You can improve this by smart region to compare selection.

overlapping regions by optical flow

Also, I am thinking about idea to use optical flow by putting images from camera at same time to sequence behind each other. From the possible overlapping regions in one images extract good features to track and find them in region of second images.

matchTemplate Example

#include <Windows.h>
#include <fstream>
#include <iostream>
#include "opencv2\highgui.hpp"
#include "opencv2\imgproc.hpp"
#include "opencv2/imgcodecs/imgcodecs.hpp"
#include "opencv2/videoio/videoio.hpp"

using namespace cv;

using namespace std;

int main(int argc, const char** argv)


Mat OneCamInput;

Mat SecondCamInput;

// load and resize source images of video

OneCamInput = imread("1.JPG");

SecondCamInput = imread("2.JPG");

resize(OneCamInput, OneCamInput, Size(800, 600));

resize(SecondCamInput,SecondCamInput, Size(800, 600));

// Show Both imput images

imshow("input1", OneCamInput);


imshow("input2", SecondCamInput);


//Convert both to gray images

cvtColor(OneCamInput, OneCamInput, COLOR_BGR2GRAY);

cvtColor(SecondCamInput, SecondCamInput, COLOR_BGR2GRAY);

// Prepare Mat for MatchTemplates

Mat res(1, 1, CV_32F);

// Prepare values for max correspondance

float resMax = 0;

Rect RectOver1;

Rect RectOver2;

int iRes;

// For loop over over compared rectangles with different size

for (int i = 20; i <= OneCamInput.cols / 4; i = i + 1) {

      // Extract rectangles from both source images

       Mat M1 = OneCamInput(Rect(OneCamInput.cols - i, 0, i, OneCamInput.rows));

       Mat M2 = SecondCamInput(Rect(0 , 0, i, SecondCamInput.rows));

       imshow("Overlap Rect1", M1);


       imshow("Overlap Rect2", M2);


      // Match how similar selected rectangles are

       matchTemplate(M1, M2, res, TM_CCOEFF_NORMED);

         // Convert Mat res(1, 1, CV_32F); to flow

         float resFloat = ((float *)([0];

//Save max correspondence

       if (resFloat >= resMax) {

             resMax = resFloat;

             cout << res << endl;

             iRes = i;

             RectOver1 = Rect(OneCamInput.cols - i, 0, i, OneCamInput.rows);

             RectOver2 = Rect(0, 0, i, SecondCamInput.rows);



Mat HM;

// Crop original images defined by border of max correspondence

Mat On1Res = OneCamInput(Rect(0, 0, OneCamInput.cols -iRes -RectOver1.width/6 , OneCamInput.rows));

Mat On2Res = SecondCamInput(Rect(0, 0, SecondCamInput.cols - iRes , OneCamInput.rows));

//Horizontal merge of this two images together

hconcat(On1Res, On2Res, HM);

imshow("Result", HM);


imwrite("Result.jpg", HM);


No comments:

Post a Comment