The Remote autonomous controller with Computer vision, Deep learning and Sensor fusion for my WiFi Robot (ongoing)

Posted on Edited on In Views:

Description:

This is a ongoing project. The idea is to build a remote Wifi Robot controller based on OpenCV 3.4.0, cuDNN, CUDA, Deep learning and Sensorfusion.
The sensor data and video are streamed to the PC for computer vision functions and Sensorfusion solutions realization. Then the controller sends back the control command back to the Wifi robot to execute.
You can find it here: https://github.com/forwardkth/BBRobotController_CV_DL_SF
It can be seen as a mini Autonomous Vehicle project with a RaCAM （forward looking Radar and Camera.
I just started to work on this project from time to time since 2017. So the code on Github has not been completed yet.

Historical updates:

2018.5 Project moved to Linux platform
2018.6 Integrate YoloV3 into BBRobot controller project with C++ Wrapper and shared library
BBRobot controller C++ project built up with Eclipse setup

Some tips:
1.in the makefile for generating the shared lib of Darknet, the nvcc configuration should be the path of your local cuda bin folder.

2. Suggest to copy the shared lib to /usr/lib folder to avoid the “can not find xxxx.so” problem.

3. The OpenCV lib dependancies should be manually added to the project build configuration in Eclipse.
4. You can find the Darknet shared lib builder here: https://github.com/forwardkth/DarkLibBuilder

Some old Pictures:






BBRobot project:

The source code of the Wifi Robot project can be found here: https://github.com/forwardkth/BBRobot

My Modification to the original Blacklib recently are:

  1. Modification to the class BalckPWM
  2. Implement the BlackServo class for easier Servo control like “PyBBIO”
  3. Add Servo control example code (example_Servo.h) under /BlackLib/examples folder
  4. Add my WIFI robot code (example_wifirobot.h) as a example under /BlackLib/examples folder

You can run the examples with the ExampleMain.cpp or run the formal project with WiFiRobotMain.cpp

Historical project updates:

2015

  1. Modification to the class of BalckPWM
  2. Add Servo motor control class “BlackServo”
  3. Add Servo control demo code under /BlackLib/examples folder
  4. Add demo WiFi robot code (example_wifirobot.h) as a example under /BlackLib/examples folder
  5. Add Video streaming code

2016

  1. New remote control UI with C sharp and WPF
  2. iOS APP for WiFi Robot remote control

2017

  1. Start working with the Computer Vision based remote controller for the WiFiRobot (ongoing…)

2018

  1. Add formal WiFi robot project code under “BBRobot/WiFiRobot/“
  2. Add Json support for Multiple sensor data transmission

The picture below shows the visualGDB configuration with VS2017 for cross complie and remote debug and deploy
https://ww3.sinaimg.cn/large/74505a4cgw1f1mhjohmbej21b70rcdwl.jpg




The new DEMO video on Youtube:
https://youtu.be/q4rHsRQ4LsQ
https://youtu.be/8c8KAbVZPCA

The old outdoor DEMO video on Youtube:
https://www.youtube.com/v77tkjFmZqY

In order to continue my project (remote contoller for my wifiRobot based on computer vision and sensor fusion), I bought a powerful Desktop PC from HP with the Nvidia Geforce GTX1070 Video Card. It looks quite nice and very easy for maintenance. But I am too old to have interest in playing games. Just use it as a working horse…




It took a lot of time to figure out the solutions for different issues before making the development enviroment ready.

  1. You need to disable the default driver for the video card and install the nvidia one. It is also OK to directly install the CUDA Tookit which has the official driver included.

  2. Disable the secure boot in BIOS. Otherwise the Nvidia driver will not be loaded. And It will not take effect.

  3. It also takes me a lot of time to figure out how to install tensorflow with CUDA9.2.
    actually you need to compile the tensorflow source code with the CUDA9.2 configuration by yourself. this blog might help: https://www.python36.com/install-tensorflow141-gpu/

I installed both the lastest versions of CUDA and CUDNN. But the TensorRT does not support CUDA9.2 for now. Later I will try to install multi versions of CUDA and try to switch among them.


  1. The version of Phython is also a something needs to be noticed. It is very important to know under which version of python the tensorflow is installed. And you need to do corresponding python interpreter configuration in the Pycharm IDE when working with tensorflow

  2. OpenCV is quite easy to install without any issue.

  3. Another thing is that when you compile the Darknet Yolo with Debug = 1, you need to add the following lines in the makefile. Unless the make process will not be successful.

Then I am happy to see the YoloV3 model can run more in real-time on this PC with maximum rate of 14FPS!

You can see the Video of testing here:

Finally I can really continue my project now: https://github.com/forwardkth/BBRobotController_CV_DL_SF

I will have the both versions of project for Linux and Windows.