OpenCV Python and colour tracking

Despite our Unity asset being based on the OpenCV libraries (and supposedly OpenCV/java compatible) there are slight differences in the syntax that make it quite difficult to get working by porting online OpenCV examples from other languages.

So before we battled getting our Unity asset working, we thought we'd just try to get some OpenCV object detection working on a more common platform; Python.

There are loads of functions for object detecting, but they're pretty CPU intensive. We did - eventually - get HoughCircle detection working, albeit quite crudely. A frame rate of 1fp3s was about possible, and the detection routines missed some of the biggest, most obvious circles in the image!

One of the fastest routines we managed to get working was object detecting by colour.
To do this we created a threshold "hue" range that our colour should fall inside, applied this as a mask to our original webcam stream, then used the createContours function to find the mid-point of each coloured shape in the image. Finally, we chose only the largest shape from the image (to avoid picking up extraneous background noise as an object) and plotted the centrepoint back onto the image.

The first step was to create a "colour threshold".
To do this is a two-step process; first change the RGB image into HSV.

Then, we  create a range of HSV values to look for in the image.
This is done by creating two arrays of values.
To pick out, in this case, just blue colours, we look for pixels that match a specific range of hues, while at the same time, looking within a set of saturation (amount of colour compared to black & white) and value (brightness) ranges.

To find the HSV values for a given shade of blue, we used OpenCV's own function cvtColour in the Python IDE editor. This could easily be converted into a script if necessary, but we were just making a quick-n-dirty example, so entered commands into the Python interpreter directly.

Don't forget to import the libraries numpy and cv2 before starting!

Then define your colour to convert as a numpy array.

Note the colour is in BGR (not RGB) format.

my_colour = np.unit8([[[ b, g, r ]]])

Convert the array of colours from BGR to HSV using

hsv_colour = cv2.cvtColour(my_colour, cv2.COLOR_BGR2HSV)

And finally display the colour as HSV with the command

print hsv_colour

The result is an array of h, s and v values to put into the detection script.
Since we're only interested in the "hue" of the colour, we can ignore the s and v values- in our detection script, we'll set these as 100 at the lowest end of the range,  and 255 at the highest. That means we're looking for any shade of our selected colour from "a little bit darker" all the way up to "almost white". (if your image contained the colour you were looking for in a dark environment, you might want to change these to something like 50 at the lowest and 150 at the upper end of the range).
We created a hue range starting at h-10, up to h+10.

So in our example, the HSV value was 105, 228, 186.
So our HSV range in the detection script was from  90,100,100 to 110,255,255.

This results in a single black and white image as each pixel in the original is compared to the "threshold" value(s). Pixels either pass the threshold test (and turn white) or fail (and turn black). This is our mask.

OpenCV has a natty little routine that finds the contours of shapes in an image. You can think of it like "flood fill" in reverse - instead of blocking an area of the same colour, up to a boundary, it finds the boundary of an area of common colour

After running this function, we end up with an array of shapes that OpenCV found in the image. So it's a simple case of looping through all shapes, finding the largest shape, and placing a dot in the middle of it.

note that the script has placed the red dot in an area of black on the final merged image. This is quite acceptable - the shape of the blue region is two of Superman's legs, joined across the top. The centre-point of this entire region (from top-left to bottom-right) may well fall in an area that isn't actually blue in colour - if the two legs  were separate, the script would place the red dot in the centre of the largest individual leg.

Because the contours function works on the entire perimeter of the shape, it allows relatively complex shapes to be detected well - even if they have "gaps" or holes in the colour mask.

Here's the final Python script:

import cv2
import numpy as np

cap = cv2.VideoCapture(0)
kernel = np.ones((5,5),np.uint8)


while(1):

     # Take each frame
     _, frame = cap.read()

     # resample (to reduce the amount of work that has to be done)
     frame = cv2.resize(frame, (0,0), fx=0.5, fy=0.5)     

     # Convert BGR to HSV     
     hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
     
     # define range of blue color in HSV
     lower_blue = np.array([90,100,100])
     upper_blue = np.array([120,255,255])
     
     # Threshold the HSV image to get only blue colors
     mask = cv2.inRange(hsv, lower_blue, upper_blue)     

     # these two lines swell then contract the region around
     # each contour/shape, then contract then swell it; the idea
     # is to remove areas of noise and little tiny shapes
     mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel)
     mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, kernel)

     # now we find each shape in the image
     _, contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

     # Bitwise-AND mask and original image
     res = cv2.bitwise_and(frame,frame, mask= mask)

     # find the largest matching object
     max_area = 0
     r=0

     for cnt in contours:
          cnt_area = cv2.contourArea(cnt)
          if cnt_area > max_area:
               max_area = cnt_area
               best_cnt = cnt
               r=1
          # end if
     # next

     # and stick a dot in the centre of it
     if r==1:
          moments = cv2.moments(best_cnt)                               # Calculate moments
          if moments['m00']!=0:
               cx = int(moments['m10']/moments['m00'])           # cx = M10/M00
               cy = int(moments['m01']/moments['m00'])           # cy = M01/M00
               #moment_area = moments['m00']                         # Contour area from moment               
               # cv2.drawContours(res,[best_cnt],0,(0,255,0),1) # draw contours on final image
               cv2.circle(res,(cx,cy),5,(0,0,255),-1)      # draw centroids in red color
          # end if
     # end if


     cv2.imshow('frame',frame)
     cv2.imshow('mask',mask)
     cv2.imshow('res',res)



     k = cv2.waitKey(5) & 0xFF
     if k == 27:
          break

cv2.destroyAllWindows()