Skip to main content

Opencv 4 Linux application in visual studio 2017- Ubuntu Linux subsystem on windows 10

This tutorial is step by step guide for the development of Opencv 4 Linux application in Visual Studio 2017 under windows. The motivation is simple. I would like to work in Visual Studio 2017, which is simply great. I personally love it. It is not the only motivation. I am a windows user, but the market is full of Linux based cloud machines and docker. So, I need to target Linux. The OpenCV running on Linux is faster from my 5-year experience(consider any possible optimization options). Another motivating point is a windows file system. Write an application for Linux that accesses the windows file system.
linux development in visual studio










Again, I am windows person with lots of Linux experience, lots of datasets, images, and video stored on windows machines. I would like to use this data without any transfer or copy of my datasets. Just directly open c:/myPositiveImages image directory in Linux app as /mnt/c/myPositiveImages. Yes, it is possible.  I am doing this to prepare my datasets stored in the windows file system and prepare the proper inputs for Yolo darknet. Yolo is not so great on windows and hard to compile. Data, images prepared in windows are not compatible with most of the Linux apps and mainly Yolo input. You can say some argument like. Prepare your dataset on windows and transfer to the Linux and read by Yolo, learn your detector and enjoy deep neural networks. No this doesn't work. Yolo has a problem to read dataset prepared on windows. This is why Visual Studio 2017 and Linux subsystem, where data from windows are processed by a Linux app. This is only one motivation from the many possible. I simply like Visual Studio, but my target is linux cloud, linux container or IoT devices like rasbery pi.

What will I learn in this opencv tutorial? (win-Linux tutorial)

  1. Prepare environment, where Visual Studio communicates with remote or local Linux
  2. Install opencv from the GIT
  3. Build Linux Opencv application from windows, linker and compiler settings
  4. Use windows file system in Linux Opencv app

Skills needed for visual studio connected to Linux

Hopefully, you can follow this tutorial without too many problems. Anyway, this is advance staff and some knowledge is beneficial for the smooth process of configuration the development environment.

-Windows
-Do not afraid of Linux and command line
-Linux basics
-Do not afraid to read step by step this text
-Try to figure out what went wrong if something fails
-Certificates, ssh
-Visu Studio
-Opencv

Environment on windows

1)Windows 10, just 10. This is not working in windows 7. Linux subsystem can be turned on and Linux easily installed from windows store, but just on W10. Open PowerShell as Administrator and run following command

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

The Windows subsystem Linux can be turned on as well in control panels (It is harder and harder to find control panels in W10). Just google it if you do not have PowerShell.

2)The second part of preparation installs Linux from the windows store. Just find and install Ubuntu 18.04.
Linux under windows

3)Install Visual studio 2017 from scratch or modify. Insclude all C++ packages and make sure to install Linux development with C++. Mark the option on the following menu in red circle.
Cross platform development in Visual Studio


Summary, 1) The windows subsystem linux is enabled. 2) Ubuntu is installed from windows store. 3) Visual Studio is installed with Linux development C++. 

Prepare Linux Environment to communicate with Visual Studio

Now, it is the time to start your Linux. Find the Ubuntu icon and start. This will open the Ubuntu command line for you.  Let's start with the upgrade the package database and install updates by following two commands.

sudo apt-get update                  update your package database
sudo apt-get upgrade                upgrade packages in your linux

The system is prepared for installation of necessary additional packages. The process is visible on the following image and all commands listed below the image. The image is attached to see the correct response for your commands.

Visual studio linux development

sudo apt install -y build-essential             Install essential packages to build Linux apps.
sudo apt install -y gdb                               Install GDB debugger program
sudo apt install -y gdbserver                     Install GDB server

Build-essential package contains following programs. make, libc6, gcc, g++, dpkg-dev. Basic package to build(C++ compiler g++, C compiler gcc, c library and header files, etc) and link your program. The gdb is debugger and gdbserver is way how to communicate with this debugger and see the information directly in Visual Studio. 
The visual studio communicates with Linux through SSH and shared file system. The same way how Putty communicates with your Linux instance in cloud machine. We need OpenSSH-server and configure ssh for PasswordAuthentification. 

sudo apt install -y openssh-server             Install openssh server
sudo nano /etc/ssh/sshd_config                 sshd configuration file and modify as follows:
Set PasswordAuthentication to Yes   “PasswordAuthentication” setting and make sure it’s set to “yes”:

sudo ssh-keygen -A                                      Generate new host keys for SSH
You can expect this as the response "ssh-keygen: generating new host keys: DSA"

sudo service ssh start                                    Start ssh service
You can expect this as the response "Starting OpenBSD Secure Shell server sshd".
The ssh configuration, key generation is the most problematic part. The problem's solution is not trivial for many people. Hopefully, now everything is prepared to communicate with Linux from Visual Studio.

Install and build opencv 4 in Linux from GIT

This part is easy for little bit experienced person. Infollowing text is described process of download latest GIT repository with Opencv 4. Build the opencv library by CMAKE and install Opencv shared library into the system.
Install opencv on ubuntu linux

I have one /home/opencv directory. Here just issue following commands:
git clone https://github.com/opencv/opencv.git
git clone https://github.com/opencv/opencv_contrib.git

This will create two folders opencv and opencv_contrib with download source code from github repository.  By following commands is installed everything what you need to build opencv. Even the packages already installed. 

[we have this]             sudo apt-get install build-essential
[additional required]   sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
[optional]                    sudo apt-get install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev


Next to opencv and opencv_contrib create the new directory build by mkdir and enter the directory.

mkdir build
cd build

The next step runs the configuration by cmake. Cmake creates your make file, all the parameters for compiling opencv project and link with installed packages. 

cmake -D CMAKE_BUILD_TYPE=Release -D -D CMAKE_INSTALL_PREFIX=/usr/local ..

Do not forgot the .. double dots. to compile with opencv_contrib specify the path to contrib modules by parameter -D OPENCV_EXTRA_MODULES_PATH="PATH/opencv_contrib/modules/"
The cmake finish with all configuration you need to build opencv by make. This takes a bit of time. The final command that install your library under system directory /usr/local/lib as on picture below commands. 

make -j7
sudo make install


Connect Visual Studio with Linux by cross-platform connection manager

To perform this configuration successfully the Prepare Linux Environment to communicate with Visual Studio chapter needs to be set up correctly. The first this is to create new project in Visual Studio 2017. I pick up Console Application (Linux).

Visual Studio console linux application

Now is the time to connect my project with a remote system. This means with my localhost Linux subsystem under windows 10. In main Visual studio, menu go to Tools-Options-Cross Platform. You will perform the operation under Add- and fille Connect to Remote System menu. 
Host name                   - probably localhost or ip
Port                              - You are running OpenSSH with most probably port 22
user name                    - Your user name to Linux
Authentification type  -Password
Password                     -Your password for the specified user (This option is enabled by sshd_config “PasswordAuthentication” setting and make sure it’s set to “yes” described in prepare Linux environment section ).

Tools/options/crossplatform
Visual studio connect to linux subsystem

The Remote IntelliSence is a very useful feature. Update button for selected already defined remote machine download header file into the local cache. When you are writing the code for Linux and using Linux or opencv headers visual studio knows the input parameters, format and mark them as known. This is a huge feature. Otherwise, most of your code will be marked as unknown. 


Headers from remote machine visual studio
Now you can add some source code and just verify your connection by print something to console. Maybe some header like #include <cstdio> or #incude <iostream> are required.

#include <cstdio>
#include <iostream>
int main()
{
    printf("hello from OpencvLinux!\n");
    return 0;
}




Configure Linux Opencv 4 projects in Visual Studio

This is not an easy step. The project configuration is complex and usually done by CMAKE. Cmake prepares all the compiler options, C++ language standard and linking dependencies options for you. In visual studio is used  g++compiler and linker. I configure a simple project in pure Linux, run cmake configuration and transfer all the compiler and linker option into Visual Studio. First, Look at full compiler and linker Configuration in the following sections. The explanation follows under compiler and linker full options. 

Opencv visual studio g++ compiler  configuration

This is the full compiler options 

"g++" -W"switch" -W"no-deprecated-declarations" -W"empty-body" -W"conversion" -W"return-type" -W"parentheses" -W"no-pointer-sign" -W"no-format" -W"uninitialized" -W"unreachable-code" -W"unused-function" -W"unused-value" -W"unused-variable" -std=c++11 -w -fno-strict-aliasing -I "/usr/local/include/opencv4/opnecv2" -g0 "g++" -O2 "3600000" -fno-threadsafe-statics -D "NDEBUG" -W"switch" -W"no-deprecated-declarations" -W"empty-body" -W"conversion" -W"return-type" -W"parentheses" -W"no-format" -W"uninitialized" -W"unreachable-code" -W"unused-function" -W"unused-value" -W"unused-variable" -fno-rtti -fno-omit-frame-pointer -std=c11 -fno-exceptions "1" -o  "C:\Users\Vlada\source\repos\OpencvLinux\OpencvLinux\obj\x64\Release\%(filename).o" 

Opencv visual studio g++ linker configuration 

This  is the full linker options

-o"C:\Users\Vlada\source\repos\OpencvLinux\OpencvLinux\bin\x64\Release\OpencvLinux.out" -Wl,-rpath-link="/lib/x86_64-linux-gnu/" -Wl,-rpath-link="/usr/local/lib" -Wl,-rpath-link="/usr/lib/x86_64-linux-gnu/" -Wl,-rpath-link="/usr/local/lib/" -Wl,-rpath-link="/usr/lib/gcc/x86_64-linux-gnu/7" "3600000" -Wl,-z,norelro -Wl,--no-undefined "g++" -l"opencv_core" -l"opencv_imgcodecs" -l"opencv_imgproc" -l"opencv_video" -l"opencv_videoio" -l"pthread" -l"jpeg" -l"gcc_s" -l"stdc++" -l"tbb" -l"rt" -l"dl" -l"m" -l"z" -l"png16" -l"c" 


Opencv 4 compiler and linker options

The first thing is to add in property page of C/C++ project all g++, include directory, C++11 standard in case of opencv 4 and other things from the picture below. The windows below include all the option and you can mimic this all configuration in a same window other than in separate tabs.

Linker opencv 4 configuration part is more tricky

Library dependencies -l"opencv_core" -l"opencv_imgcodecs" -l"opencv_imgproc" -l"opencv_video" -l"opencv_videoio" -l"pthread" -l"jpeg" -l"gcc_s" -l"stdc++" -l"tbb" -l"rt" -l"dl" -l"m" -l"z" -l"png16" -l"c"  

You are just filling following values into Library dependencies option without -l 
opencv_core
opencv_imgcodecs
opencv_imgproc
opencv_video
opencv_videoio
pthread
jpeg
gcc_s
stdc++
tbb
rt
dl
m
z
png16
c
opencv 4  linker

The shared library search path option is important to locate library specified in previous step. My shared libraries like c,z,tbb,stdc++ .... are located in this shared lib locations:

/lib/x86_64-linux-gnu/

/usr/local/lib
/usr/lib/x86_64-linux-gnu/
/usr/local/lib/
/usr/lib/gcc/x86_64-linux-gnu/7

(you can have different hard to say), You can compile your c++ project directly in Linux by Cmake and look what shared libraries are used by your program. By command, LDD OpencvLinux is possible list your shared dependencies library with version and location. Build a simple opencv 4 project by cmake and include all in visual studio.

Opencv 4 shared library

All linker configuration can be mimic by follow all the setting in this table.

Opencv Yolo darknet program loading image from windows

The following program is stupid, not parametric externally, but reads images from windows file system and perform some actions. I will rewrite this program and explain in more details in following tutorials how to prepare data for yolo darknet. The scope of this article is little bit different.

Yolo darknet dataset preparation program description 

Program reads images from positive image library /mnt/c/npos" (C:/npos) and negative image library /mnt/c/neg (C:/neg). The positive image is inserted into a negative one on random location. You can see an inserted person in the center of the image. This result is stored as input of yolo darknet. 1.jpg sample with 1.txt file 1 0.490885 0.811523 0.031684 0.0651042 with class, and person normalized location in the image.


One last output is txt file of all the input image for train.txt and test.txt.
/mnt/c/result/0.jpg
/mnt/c/result/1.jpg
/mnt/c/result/2.jpg
/mnt/c/result/3.jpg
/mnt/c/result/4.jpg
/mnt/c/result/5.jpg
/mnt/c/result/6.jpg
/mnt/c/result/7.jpg
/mnt/c/result/8.jpg
/mnt/c/result/9.jpg

Sample code in progress

#include "opencv2/imgcodecs.hpp"
#include "opencv2/imgcodecs/imgcodecs.hpp"
#include "opencv2/core/core.hpp"
#include "opencv2/highgui/highgui.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include "opencv2/videoio/videoio.hpp"
#include "opencv2/video/video.hpp"
#include <opencv2/imgproc.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/videoio/videoio.hpp>
#include <opencv2/imgcodecs/imgcodecs.hpp>
#include <vector>
#include <cstdio>
#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>
#include <string>

#include <time.h>
#include <ctime>
#include <random>
#include <iostream>
#include <sys/types.h>
#include <dirent.h>
                        using namespace std;
    using namespace cv;

    void read_directory(const std::string &name, vector<String> &v)
    {
        cout << "read dir" << endl;
        DIR *dirp = opendir(name.c_str());
        struct dirent *dp;
        while ((dp = readdir(dirp)) != NULL)
        {
            v.push_back(dp->d_name);
            cout << dp->d_name << endl;
        }
        closedir(dirp);
        cout << "closing" << endl;
    }

    int main()
    {

        vector<String> neg;
        read_directory("/mnt/c/neg", neg);
        vector<String> pos;
        read_directory("/mnt/c/npos", pos);
        cout << "dir read ends" << endl;
        long lowest = 0;
        if (pos.size() > neg.size())
        {
            lowest = neg.size();
        }
        else
        {
            lowest = pos.size();
        }
        cout << lowest << endl;
        std::random_device rd;
        std::mt19937 gen(rd());
        std::ofstream outtest("/mnt/c/test.txt");
        std::ofstream outtrain("/mnt/c/train.txt");
        cout << "step 2" << endl;
        for (int i = 2; i < lowest; i++)
        {
            cout << pos.at(i) << endl;
            cout << neg.at(i) << endl;
        }
        long iter = 2;
        for (int i = 3; i < lowest; i++)
        {
            cout << "xxxxxxxxxxxxxxxxx" << endl;
            cout << pos.at(i) << endl;
            string pom = "/mnt/c/npos/" + pos.at(i);
            Mat po = imread(pom, IMREAD_COLOR);
            string nem = "/mnt/c/neg/" + neg.at(i);
            cout << nem << endl;
            Mat ne = imread(nem, IMREAD_COLOR);
            cout << "xxxxxxxxxxxxxxxxx" << endl;
            cout << "after reading" << endl;
            std::uniform_int_distribution<> dis(1ne.cols - po.cols + 1);
            std::uniform_int_distribution<> dis2(1ne.rows - po.rows + 1);
            int a = dis(gen);
            int b = dis2(gen);
            cout << a << endl;
            cout << b << endl;
            cout << po.cols << endl;
            cout << po.rows << endl;
            po.copyTo(ne(Rect(a, b, po.colspo.rows)));
            cout << "writing result" << endl;
            string base = "/mnt/c/result/";
            stringstream builder;
            builder << base << iter << ".jpg";
            imwrite(builder.str(), ne);
            stringstream builder2;
            string base2 = "/mnt/c/result/";
            builder2 << base << iter << ".txt";
            std::ofstream outfile(builder2.str());
            stringstream builder4;
            builder4 << base2 << iter << ".jpg" << endl;
            outtest << builder4.str();
            outtrain << builder4.str();
            //outfile << base2 << " " << a << " " << b << " " << a+po.cols << " "<< b+po.rows  << std::endl;
            outfile << 1 << " " << (float)a / ne.cols << " " << (float)b
             / ne.rows << " " << (float)po.cols / ne.cols << " " <<
              (float)po.rows / ne.rows << std::endl;
            outfile.close();
            iter = iter + 1;
        }
        outtest.close();
        outtrain.close();
        cout << "all is done" << endl;
        return 0;
    }

Common problems with opencv like an undefined reference to cv::imwrite


1) Restart SSH if Visual studio stop communicates with linux subsystem:sudo service ssh restart


2)Problems with a linker, by cmake all is configured ok, by make file and own g++ setting like in case of Visual Studio the problem can be C++11 standard in opnecv 4 (std::__cxx11), include files but most probably the settings of the shared libraries. 

Symptoms:
undefined reference to cv::imwrite(std::basic_string

undefined reference to `cv::imread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)' undefined reference to 


cv::imwrite(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cv::_InputArray const&, std::vector<int, std::allocator<int> > const&)

Comments

Post a Comment