Machine Vision

Motivation

An area I want to experiment with is machine vision. I believe the sonars are good enough to detect obstacles without the need to resort to LIDAR. However, if there are no fences around the garden, how do you know when you have left the property? I could put a GPS boundry in later as the differential GPS I have experimented with is definately accurate enough, but that is a job for another day. Instead, I intend to use a camera and try and recognize grass so if the mower tries to drive onto the footpath, it will stop and maybe turn around.

Goal

In order to process images or a video stream, I will need some more processing power than an Arduino can provide. I have a Raspberry Pi 3 that I should be able to press into service, especially as it has a very simple camera interface. Rather than use the Pi as the controller for the mower, my plan is to make a smart sensor whose output can be read by the arduino. So I need to be able to classify grass. Namely answer the question "is this grass?"

If the answer is 'yes', then we mow it. If the answer is 'no', then we fall back to other sensors and make more deterministic mechanisms. so to move forward with Moana, I need to do the following:

Machine Vision Roadmap

• Mount a camera and collect video footage of grass & things that are not grass.
• Download the video and pull all the images.
• Resize the images to something we can work with.
• The 'image processor' will be a raspberry pi.
• We also need to normalise a pre-process all these training images.
• Create a framework to train the network to classify the images.
• Start with simple classifiers.
• Move onto CNNs or similar if deemed necessary.
• Need to mount a Pi on a platform with a Pi camera.
• Use Pi to capture videos or images.
• Write code to pull images from video to directories.
• Resize images and pre proceess them.
• Create a training set
• Create a validation set.
• Train model and persist it.
• Write code to load trained model and integrate it with image capture.
• Make predictions and feed these back to the Arduino for action.
• Action will initially be logging or some physical manoever

A lot to do...

I have been digging into the raspberry Pi and its camera. You can set the resolution on the camera and the minimum value is 64 by 64 pixels which is in the right ballpark for what I need. The lower the resolution, the less processing power required. I was thinking of 32 by 32 but 64 by 64 seems a good place to start.

I wrote a simple program that can take and store a photo every 0.5 seconds. Each image is approx 30k in size so I can capture several thousand on a pi, download and use them to train the model. Apart from brightness normalization and gray scaling, hopefully there is little preprocessing required. So the plan is to put the raspberry pi and the camera on the mower, drive it around and collect the training data. The training will be done externally on a laptop to create the model. Using the same camera setup for training and running should yeild benefits.

So we need to have a new plan:

• write a program to capture images and store them on the pi
• write a simple locomotion routine for moana that does not use its many senses.
• Mount the pi on the mower.
• mount the camera on the mower
• create a means to power the pi.

Powering the Pi

I spent some time creating a power board for Moana which has 3-3v, 5 V and 9 v taps. The pi is powered via a micro-Usb port. I could make up a custom wire but instead I think I'll add a 5V USB socket to the power board and can the use a USB-to-micro-Usb cable.

Controlling the camera

Use a GPIO pin on the Pi which can be read from the image capture program. If the arduino sets the pin high when it starts to move the mower, we can start taking photos. When we stop, the pin will be pulled low and photos will pause.

Mounting the Pi

I have disabled the sonars and added the Pi camera to the fron of the sonar mast, pointing down towards the front of the mower. I made a small platform behind the mast on which I mounted the Pi, allowing it to connect the the raspberry Pi camera. This is a fairly temporary arrangement for now, but I need it to collect training data.

This is just a quick note to capture the camera angle used when taking training photos.

This image shows the Pi mounted behind the sonar tower

I've added to the MO1 command (stipes1 routine) to turn the ENABLE_CAMERA (pin 34) high. When the MO1 routine ends, the enable camera pin changes state (goes low) and turns the camera off.

To gain easy access to the pi, I have enabled ssh with static IP 192.168.1.74. Provided I'm within Wifi range it is possible to login and have a poke around (script tests connection every 10 mins so keep Pi running until it comes back online). The python program has been modified to store a photoid so as not to overwrite. The program then needs to integrate with arduino and run on startup. I can then run around various grass patches and set it running. When back in lab, I can pull the photos off via scp.

This is the script that runs on the Pi to restart the network when the mower comes back into range of the wifi network. This gets called from cron and is triggered every 10 minutes.

        #!/bin/bash

        ping -c2 192.168.1.1 > /dev/null

        if [ $? != 0 ]
        then
          echo "No network connection, restarting wlan0"
          /sbin/ifdown 'wlan0'
          sleep 5
          /sbin/ifup --force 'wlan0'
        fi
        

Note: On upgrading to the Buster Operating System, I found the Pi reconnected automatically and I did not need to run this script any longer.

February 2021