A few weeks ago I published a tutorial on how to get started with the Google Coral USB Accelerator. That tutorial was meant to help you configure your device and run your first demo script.
Today we are going to take it a step further and learn how to utilize the Google Coral in your own custom Python scripts!
Inside today’s tutorial you will learn:
- Image classification with the Coral USB Accelerator
- Image classification in video with the Google Coral Accelerator
- Object detection with the Google Coral
- Object detection in video with the Coral USB Accelerator
After reading this guide, you will have a strong understanding of how to utilize the Google Coral for image classification and object detection in your own applications.
To learn how to perform image classification and object detection with the Google Coral USB Accelerator, just keep reading!
Object detection and image classification with Google Coral USB Accelerator
For this guide I will be making the following assumptions:
- You already own a Google Coral USB Accelerator.
- You have followed my previous tutorial on how to install and configure Google Coral.
If you haven’t followed by install guide, please refer to it before continuing. Finally, I’ll note that I’m connecting my Google Coral USB Accelerator to my Raspberry Pi to gather results — I’m doing this for two reasons:
- I’m currently writing a book on using the Raspberry Pi for Computer Vision which will also cover the Google Coral.
- I cover the Raspberry Pi quite often on the PyImageSearch blog and I know many readers are interested in how they can leverage it for computer vision.
If you don’t have a Raspberry Pi but still want to use your Google Coral USB Accelerator, that’s okay, but make sure you are running a Debian-based OS.
Again, refer to my previous Google Coral getting started guide for more information.
Project structure
Let’s review the project included in today’s “Downloads”:
$ tree --dirsfirst . ├── inception_v4 │ ├── imagenet_labels.txt │ └── inception_v4_299_quant_edgetpu.tflite ├── mobilenet_ssd_v2 │ ├── coco_labels.txt │ └── mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite ├── mobilenet_v2 │ ├── imagenet_labels.txt │ └── mobilenet_v2_1.0_224_quant_edgetpu.tflite ├── classify_image.py ├── classify_video.py ├── detect_image.py ├── detect_video.py ├── janie.jpg └── thanos.jpg 3 directories, 12 files
Today we will be reviewing four Python scripts:
classify_image.py– Classifies a single image with the Google Coral.classify_video.py– Real-time classification of every frame from a webcam video stream using the Coral.detect_image.py– Performs object detection using Google’s Coral deep learning coprocessor.detect_video.py– Real-time object detection using Google Coral and a webcam.
We have three pre-trained TensorFlow Lite models + labels available in the “Downloads”:
- Classification (trained on ImageNet):
inception_v4/– The Inception V4 classifier.mobilenet_v2/– MobileNet V2 classifier.
- Object detection (trained on COCO):
mobilenet_ssd_v2/– MobileNet V2 Single Shot Detector (SSD).
If you are curious about how to train your own classification and object detection models, be sure to refer to Deep Learning for Computer Vision with Python.
For both classify_image.py and detect_image.py , I’ve provided two testing images in the “Downloads”:
janie.jpg– My adorable beagle.thanos.jpg– Character from Avengers: End Game.
For the classify_video.py and detect_video.py scripts, we’ll be capturing frames directly from a camera connected to the Raspberry Pi. You can use one of the following with today’s example scripts:
- PiCamera V2 – The official Raspberry Pi Foundation camera.
- USB Webcam – Any USB camera that supports V4L will work, such as a Logitech branded webcam.
Image classification with the Coral USB Accelerator
Let’s get started with image classification on the Google Coral!
Open up the classify_image.py file and insert the following code:
# import the necessary packages
from edgetpu.classification.engine import ClassificationEngine
from PIL import Image
import argparse
import imutils
import time
import cv2
# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-m", "--model", required=True,
help="path to TensorFlow Lite classification model")
ap.add_argument("-l", "--labels", required=True,
help="path to labels file")
ap.add_argument("-i", "--image", required=True,
help="path to input image")
args = vars(ap.parse_args())
We start of by importing packages. Most notably, we are importing ClassificationEngine from edgetpu on Line 2.
From there we’ll parse three command line arguments via Lines 10-17:
--model: The path to our TensorFlow Lite classifier.--labels: Class labels file path associated with our model.--image: Our input image path.
Using these three command line arguments, our script will be able to handle compatible pre-trained models and any image you throw at it all from the command line. Command line arguments are one of the number one problems people e-mail me about, so be sure to review my tutorial on argparse and command line arguments if you need a refresher.
Let’s go ahead and load the labels :
# initialize the labels dictionary
print("[INFO] parsing class labels...")
labels = {}
# loop over the class labels file
for row in open(args["labels"]):
# unpack the row and update the labels dictionary
(classID, label) = row.strip().split(" ", maxsplit=1)
labels[int(classID)] = label.strip()
Lines 21-27 facilitate loading class labels from a text file into a Python dictionary. Later on, the Coral API will return the predicted classID (an integer). We can then take that integer class label and lookup the associated label value in this dictionary.
Moving on, now let’s load our classification model with the edgetpu API:
# load the Google Coral classification model
print("[INFO] loading Coral model...")
model = ClassificationEngine(args["model"])
Our pre-trained TensorFlow Lite classification model is instantiated via the ClassificationEngine class (Line 31) where we pass in the path to our model via command line argument.
Let’s go ahead and load + preprocess our image :
# load the input image image = cv2.imread(args["image"]) image = imutils.resize(image, width=500) orig = image.copy() # prepare the image for classification by converting (1) it from BGR # to RGB channel ordering and then (2) from a NumPy array to PIL # image format image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) image = Image.fromarray(image)
Our image is loaded (Line 34) and then preprocessed (Lines 35-42).
Take note that we made an original copy of the image — we’ll be annotating this copy of the image with the output predictions later in the script.
How easy is it to perform classification inference on an image with the Google Coral Python API?
Let’s find out now:
# make predictions on the input image
print("[INFO] making predictions...")
start = time.time()
results = model.ClassifyWithImage(image, top_k=5)
end = time.time()
print("[INFO] classification took {:.4f} seconds...".format(
end - start))
On Line 47, we make classification predictions on the input image using the ClassifyWithImage function (a super easy one-liner function call). I really like how the edgetpu API allows us to specify that we only want the top results with the top_k parameter.
Timestamps are sandwiched around this classification line and the elapse time is then printed via Lines 49 and 50.
From here we’ll process the results :
# loop over the results
for (i, (classID, score)) in enumerate(results):
# check to see if this is the top result, and if so, draw the
# label on the image
if i == 0:
text = "Label: {}, {:.2f}%".format(labels[classID],
score * 100)
cv2.putText(orig, text, (10, 30), cv2.FONT_HERSHEY_SIMPLEX,
0.8, (0, 0, 255), 2)
# display the classification result to the terminal
print("{}. {}: {:.2f}%".format(i + 1, labels[classID],
score * 100))
# show the output image
cv2.imshow("Image", orig)
cv2.waitKey(0)
Looping over the results (Line 53) we first find the top result and annotate the image with the label and percentage score (Lines 56-60).
For good measure, we’ll also print the other results and scores (but only in our terminal) via Lines 63 and 64.
Finally, the annotated original (OpenCV format) image is displayed to the screen (Lines 67 and 68).
That was straightforward. Let’s put our classification script to the test!
To see image classification in action with the Google Coral, make sure you use the “Downloads” section of this guide to download the code + pre-trained models — from there, execute the following command:
$ python classify_image.py --model inception_v4/inception_v4_299_quant_edgetpu.tflite --labels inception_v4/imagenet_labels.txt --image janie.jpg [INFO] parsing class labels... [INFO] loading Coral model... W0507 08:04:36.445022 5885 package_registry.cc:65] Minimum runtime version required by package (5) is lower than expected (10). [INFO] making predictions... [INFO] classification took 1.2446 seconds... 1. beagle: 97.27%
The output of the image classification script can be seen in Figure 1 at the top of this section.
Here you can see that Janie, my dog, is correctly classified as “beagle”.
Image classification in video with the Google Coral Accelerator
In the previous section, we learned how to perform image classification to a single image — but what if we wanted to perform image classification to a video stream?
I’ll be showing you how to accomplish exactly that.
Open up a new file, name it classify_video.py and insert the following code:
# import the necessary packages
from edgetpu.classification.engine import ClassificationEngine
from imutils.video import VideoStream
from PIL import Image
import argparse
import imutils
import time
import cv2
# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-m", "--model", required=True,
help="path to TensorFlow Lite classification model")
ap.add_argument("-l", "--labels", required=True,
help="path to labels file")
args = vars(ap.parse_args())
There are two differences in our first code block for real-time classification compared to our previous single image classification script:
- On Line 2 we’ve added the
VideoStreamimport for working with our webcam. - We no longer have a
--imageargument since by default we will be using our webcam.
Just as before, let’s load the labels and model , but now we also need to instantiate our VideoStream :
# initialize the labels dictionary
print("[INFO] parsing class labels...")
labels = {}
# loop over the class labels file
for row in open(args["labels"]):
# unpack the row and update the labels dictionary
(classID, label) = row.strip().split(" ", maxsplit=1)
label = label.strip().split(",", maxsplit=1)[0]
labels[int(classID)] = label
# load the Google Coral classification model
print("[INFO] loading Coral model...")
model = ClassificationEngine(args["model"])
# initialize the video stream and allow the camera sensor to warmup
print("[INFO] starting video stream...")
vs = VideoStream(src=0).start()
#vs = VideoStream(usePiCamera=False).start()
time.sleep(2.0)
Lines 19-31 are identical to our previous script where we load our class labels and store them in a dictionary.
On Line 35 we instantiate our VideoStream object so that we can read frames in our webcam (covered in the next code block). A 2.0 second sleep is added so our camera has time to warm up (Line 37).
Note: By default, this script will use a USB webcam. If you would like to use a Raspberry Pi camera module, simply comment out Line 35 and uncomment Line 36.
Let’s begin our loop:
# loop over the frames from the video stream while True: # grab the frame from the threaded video stream and resize it # to have a maximum width of 500 pixels frame = vs.read() frame = imutils.resize(frame, width=500) orig = frame.copy() # prepare the frame for classification by converting (1) it from # BGR to RGB channel ordering and then (2) from a NumPy array to # PIL image format frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) frame = Image.fromarray(frame)
We start looping on Line 40.
Line 43 grabs a frame from the threaded video stream.
We go ahead and preprocess it exactly as we did in the previous script (Lines 44-51).
With the frame in the correct PIL format, now we can make predictions and draw our annotations:
# make predictions on the input frame
start = time.time()
results = model.ClassifyWithImage(frame, top_k=1)
end = time.time()
# ensure at least one result was found
if len(results) > 0:
# draw the predicted class label, probability, and inference
# time on the output frame
(classID, score) = results[0]
text = "{}: {:.2f}% ({:.4f} sec)".format(labels[classID],
score * 100, end - start)
cv2.putText(orig, text, (10, 30), cv2.FONT_HERSHEY_SIMPLEX,
0.5, (0, 0, 255), 2)
# show the output frame and wait for a key press
cv2.imshow("Frame", orig)
key = cv2.waitKey(1) & 0xFF
# if the `q` key was pressed, break from the loop
if key == ord("q"):
break
# do a bit of cleanup
cv2.destroyAllWindows()
vs.stop()
Just as before, Line 55 performs inference.
From there, the top result is extracted and the classification label + score is annotated on the orig frame (Lines 59-66).
The frame is displayed on the screen (Line 69).
If the "q" key is pressed, we’ll break from the loop and clean up (Lines 70-78).
Let’s give image classification in video streams with the Google Coral a try!
Make sure you use the “Downloads” section of this guide to download the code + pre-trained models — from there, execute the following command:
$ python classify_video.py --model mobilenet_v2/mobilenet_v2_1.0_224_quant_edgetpu.tflite --labels mobilenet_v2/imagenet_labels.txt [INFO] parsing class labels... [INFO] loading Coral model... W0507 07:52:49.077803 2830 package_registry.cc:65] Minimum runtime version required by package (5) is lower than expected (10). [INFO] starting video stream...
An example of real-time image classification can be seen above in Figure 2.
Using the Google Coral USB Accelerator, the MobileNet classifier (trained on ImageNet) is fully capable of running in real-time on the Raspberry Pi.
Object detection with the Google Coral
We’ve already learned how to apply image classification with the Google Coral — but what if we not only wanted to classify an object in an image but also detect where in the image the object is?
Such a task is called object detection, a technique I’ve covered quite a few times on the PyImageSearch blog (refer to this deep learning-based object detection guide if you are new to the concept).
Open up the detect_image.py file and let’s get coding:
# import the necessary packages
from edgetpu.detection.engine import DetectionEngine
from PIL import Image
import argparse
import imutils
import time
import cv2
# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-m", "--model", required=True,
help="path to TensorFlow Lite object detection model")
ap.add_argument("-l", "--labels", required=True,
help="path to labels file")
ap.add_argument("-i", "--image", required=True,
help="path to input image")
ap.add_argument("-c", "--confidence", type=float, default=0.3,
help="minimum probability to filter weak detections")
args = vars(ap.parse_args())
Our packages are imported on Lines 2-7. For Google Coral object detection with Python, we use the DetectionEngine from the edgetpu API.
Our command line arguments are similar to the classify_image.py script with one exception — we’re also going to supply a --confidence argument representing the minimum probability to filter out weak detections (Lines 17 and 18).
Now we’ll load the labels in the same manner as in our classification scripts:
# initialize the labels dictionary
print("[INFO] parsing class labels...")
labels = {}
# loop over the class labels file
for row in open(args["labels"]):
# unpack the row and update the labels dictionary
(classID, label) = row.strip().split(maxsplit=1)
labels[int(classID)] = label.strip()
And from there we’ll load our object detection model :
# load the Google Coral object detection model
print("[INFO] loading Coral model...")
model = DetectionEngine(args["model"])
We can now load our input image and perform preprocessing:
# load the input image image = cv2.imread(args["image"]) image = imutils.resize(image, width=500) orig = image.copy() # prepare the image for object detection by converting (1) it from # BGR to RGB channel ordering and then (2) from a NumPy array to PIL # image format image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) image = Image.fromarray(image)
After preprocessing, it is time to perform object detection inference:
# make predictions on the input image
print("[INFO] making predictions...")
start = time.time()
results = model.DetectWithImage(image, threshold=args["confidence"],
keep_aspect_ratio=True, relative_coord=False)
end = time.time()
print("[INFO] object detection took {:.4f} seconds...".format(
end - start))
Lines 49 and 50 use Google Coral’s object detection API to make predictions.
Being able to pass our confidence threshold (via the threshold parameter), is extremely convenient in this API. Honestly, I wish OpenCV’s DNN API would follow suit. It saves an if-statement later on as you can imagine.
Let’s process our results :
# loop over the results
for r in results:
# extract the bounding box and box and predicted class label
box = r.bounding_box.flatten().astype("int")
(startX, startY, endX, endY) = box
label = labels[r.label_id]
# draw the bounding box and label on the image
cv2.rectangle(orig, (startX, startY), (endX, endY),
(0, 255, 0), 2)
y = startY - 15 if startY - 15 > 15 else startY + 15
text = "{}: {:.2f}%".format(label, r.score * 100)
cv2.putText(orig, text, (startX, y),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)
# show the output image
cv2.imshow("Image", orig)
cv2.waitKey(0)
Looping over the results (Line 56), we first extract the bounding box coordinates (Lines 58 and 59). Conveniently, the box is already scaled relative to our input image dimensions (from any behind the scenes resizing the API does to fit the image into the CNN).
From there we can easily extract the class label via Line 60.
Next, we draw the bounding box rectangle (Lines 63 and 64) and draw the predicted object text on the image (Lines 65-68).
Our orig image (with object detection annotations) is then displayed via Lines 71 and 72.
Let’s put object detection with the Google Coral USB Accelerator to the test!
Use the “Downloads” section of this tutorial to download the source code + pre-trained models.
From there, open up a terminal and execute the following command:
$ python detect_image.py \ --model mobilenet_ssd_v2/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite \ --labels mobilenet_ssd_v2/coco_labels.txt --image thanos.jpg [INFO] parsing class labels... [INFO] loading Coral model... W0507 08:00:58.843066 4919 package_registry.cc:65] Minimum runtime version required by package (5) is lower than expected (10). [INFO] making predictions... [INFO] object detection took 0.2318 seconds...
Just for fun, I decided to apply object detection to a screen capture of Avengers: Endgame movie (don’t worry, there aren’t any spoilers!)
Here we can see that Thanos, a character from the film, is detected (Figure 3)…although I’m not sure he’s an actual “person” if you know what I mean.
Object detection in video with the Coral USB Accelerator
Our final script will cover how to perform object detection in real-time video with the Google Coral.
Open up a new file, name it detect_video.py , and insert the following code:
# import the necessary packages
from edgetpu.detection.engine import DetectionEngine
from imutils.video import VideoStream
from PIL import Image
import argparse
import imutils
import time
import cv2
# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-m", "--model", required=True,
help="path to TensorFlow Lite object detection model")
ap.add_argument("-l", "--labels", required=True,
help="path to labels file")
ap.add_argument("-c", "--confidence", type=float, default=0.3,
help="minimum probability to filter weak detections")
args = vars(ap.parse_args())
To start, we import our required packages and parse our command line arguments (Lines 2-8) Again, we’re using VideoStream so we can access our webcam (since we’re performing object detection on webcam frames, we don’t have a --image command line argument).
Next, we’ll load our labels and instantiate both our model and video stream:
# initialize the labels dictionary
print("[INFO] parsing class labels...")
labels = {}
# loop over the class labels file
for row in open(args["labels"]):
# unpack the row and update the labels dictionary
(classID, label) = row.strip().split(maxsplit=1)
labels[int(classID)] = label.strip()
# load the Google Coral object detection model
print("[INFO] loading Coral model...")
model = DetectionEngine(args["model"])
# initialize the video stream and allow the camera sensor to warmup
print("[INFO] starting video stream...")
vs = VideoStream(src=0).start()
#vs = VideoStream(usePiCamera=False).start()
time.sleep(2.0)
From there, we’ll loop over frames from the video stream:
# loop over the frames from the video stream while True: # grab the frame from the threaded video stream and resize it # to have a maximum width of 500 pixels frame = vs.read() frame = imutils.resize(frame, width=500) orig = frame.copy() # prepare the frame for object detection by converting (1) it # from BGR to RGB channel ordering and then (2) from a NumPy # array to PIL image format frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) frame = Image.fromarray(frame) # make predictions on the input frame start = time.time() results = model.DetectWithImage(frame, threshold=args["confidence"], keep_aspect_ratio=True, relative_coord=False) end = time.time()
Our frame processing loop begins on Line 41. We proceed to:
- Grab and preprocess our frame (Lines 44-52).
- Perform object detection inference with the Google Coral (Lines 56 and 57).
From there we’ll process the results and display our output:
# loop over the results
for r in results:
# extract the bounding box and box and predicted class label
box = r.bounding_box.flatten().astype("int")
(startX, startY, endX, endY) = box
label = labels[r.label_id]
# draw the bounding box and label on the image
cv2.rectangle(orig, (startX, startY), (endX, endY),
(0, 255, 0), 2)
y = startY - 15 if startY - 15 > 15 else startY + 15
text = "{}: {:.2f}%".format(label, r.score * 100)
cv2.putText(orig, text, (startX, y),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)
# show the output frame and wait for a key press
cv2.imshow("Frame", orig)
key = cv2.waitKey(1) & 0xFF
# if the `q` key was pressed, break from the loop
if key == ord("q"):
break
# do a bit of cleanup
cv2.destroyAllWindows()
vs.stop()
Here we loop over each of the detected objects, grab the bounding box + class label, and annotate the frame (Lines 61-73).
The frame (with object detection annotations) is displayed via Line 76.
We’ll continue to process more frames unless the "q" (quit) key is pressed at which point we break and clean up (Lines 77-85).
Let’s put this Python + Coral object detection script to work!
To perform video object detection with the Google Coral, make sure you use the “Downloads” section of the guide to download the code + pre-trained models.
From there you can execute the following command to start the object detection script:
$ python detect_video.py \ --model mobilenet_ssd_v2/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite \ --labels mobilenet_ssd_v2/coco_labels.txt [INFO] parsing class labels... [INFO] loading Coral model... W0507 07:43:19.420830 377 package_registry.cc:65] Minimum runtime version required by package (5) is lower than expected (10). [INFO] starting video stream...
For our final example of applying real-time object detection with the Google Coral, I decided to let Janie in my office for a bit as I recorded a demo (and even decided to sing her a little song) — you can see the result in Figure 4 above.
The problem with the Raspberry Pi 3B+ and Google Coral USB Accelerator
You might have noticed that our inference results are pretty similar to what we obtain with the Movidius NCS — doesn’t Google advertise the Coral USB Accelerator as being faster than the NCS?
What’s the problem here?
Is it the Google Coral?
Is it our code?
Is our device configured incorrectly?
Actually, it’s none of the above.
The problem here is the Raspberry Pi 3B+ only supports USB 2.0.
The bottleneck is the I/O taking place from the CPU, to USB, to the Coral USB Accelerator, and back.
Inference speed will dramatically improve once the Raspberry Pi 4 is released (which will certainly support USB 3, giving us the fastest possible inference speeds with the Coral USB Accelerator).
What's next? I recommend PyImageSearch University.
30+ total classes • 39h 44m video • Last updated: 12/2021
★★★★★ 4.84 (128 Ratings) • 3,000+ Students Enrolled
I strongly believe that if you had the right teacher you could master computer vision and deep learning.
Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?
That’s not the case.
All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.
If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.
Inside PyImageSearch University you'll find:
- ✓ 30+ courses on essential computer vision, deep learning, and OpenCV topics
- ✓ 30+ Certificates of Completion
- ✓ 39h 44m on-demand video
- ✓ Brand new courses released every month, ensuring you can keep up with state-of-the-art techniques
- ✓ Pre-configured Jupyter Notebooks in Google Colab
- ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
- ✓ Access to centralized code repos for all 500+ tutorials on PyImageSearch
- ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
- ✓ Access on mobile, laptop, desktop, etc.
Summary
In this tutorial, you learned how to utilize your Google Coral USB Accelerator for:
- Image classification
- Image classification in video
- Object detection
- Object detection in video
Specifically, we used pre-trained deep learning models, including:
- Inception V4 (trained on ImageNet)
- MobileNet V4 (trained on ImageNet)
- MobileNet SSD V2 (trained on COCO)
Our results were far, far better than trying to use the Raspberry Pi CPU alone for deep learning inference.
Overall, I was very impressed with how easy it is to use the Google Coral and the edgetpu library in my own custom Python scripts.
I’m looking forward to seeing how the package develops (and hope they make it this easy to convert and run custom deep learning models on the Coral).
To download the source code and pre-trained to this post (and be notified when future tutorials are published here on PyImageSearch), just enter your email address in the form below!
Download the Source Code and FREE 17-page Resource Guide
Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!
Hi Adrian,
Great work. I have been following your tutorials and they really are helpful to me.
I went thorugh your Dlib tracker tutorial and was wondering if it is possible to run that tracker on Google Coral.
Also, recommend if Yolov3-tiny is better for real time tasks or Mobilenet SSD ?
Keep up the good work !
Regards,
Naufil
The tracker itself will not be able to run on the Coral USB Accelerator as dlib’s correlation tracker is not a deep learning model.
As far as YOLO versus MobileNet + SSD goes, that really depends on your application. I personally find that MobileNet + SSD tends to perform better than YOLO (less false-positives). I’ve also found that MobileNet + SSD tends to be a bit easier to train. I don’t typically use YOLO unless I have a very specific reason to do so.
Hi Adrian,
Just wanted to push my thoughts into it. What about writing the dlib’s correlation tracker into TensorFlow in order to run it on Coral? Also as dlib’s GPU compile is available, so we can run this tracker on GPU.
Dlib’s correlation tracker is not a deep learning model. As far as I understand, it was never meant to run on a GPU.
Hello Adrian ,
I am using the teachable machines site. and I trained a module and export it to use with Tenserflow Lite and Edge TPU option .
When I ran your script it gave me the following error :
“Dectection model should have 4 output tensors!This model has 1.”
can you please tell me how to compile or use the options in the teachable machine site to export correctly?
Many thanks and great work!
Hey James — I would recommend you read Raspberry Pi for Computer Vision which shows you how to train models and deploy them to the Coral with TensorFlow Lite.
Hello andrian , i have been following your articles from past one year and that has helped me in various ways THANKS for your such well decribed articles.
Also can you please help me out .Am stuck in the custom model creation for edgetpus .My model gets converted by using the edgetpu compiler but it still shows me segmentation fault when i try to inference from it. Am trying to use the transfer learning and on that top my own custom model.
Thanks again
I’m sorry to hear about the segmentation fault issue. It’s hard to say what exactly is causing the issue though. Have you tried walking through the code using Python’s “pdb” debugger? That should give you some additional insight.
Nice article! It seems quite simple to implement tf models. I’m curious about the FPS compared to the movidius NCS1
Thanks Martin. I’ll be doing a full comparison between the embedded devices, including the NCS, Coral, and Nano, at a later date.
Hi Adrian,
Thanks for another great post, but I was asking my self, there are only three pre-trained models. can you use other pre-trained models or train your own and then download them into the board?
Thanks.
Great question. I’ll be covering how to train your own custom models and then convert + deploy them to the Google Coral USB Accelerator inside my upcoming book, Raspberry Pi for Computer Vision.
Another very timely tutorial for me!
I added your fps counter from other tutorials and ran this:
python3 detect_video.py –model mobilenet_ssd_v2/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite –labels mobilenet_ssd_v2/coco_labels.txt
on my Odroid XU-4 running Ubuntu Mate16, The Odroid has USB3 ports.
Doing everything via ssh -X wtih an HP 1080p USB webcam I got:
[INFO] starting video stream…
[INFO] starting the FPS counter …
[INFO] Run elapsed time: 44.93
[INFO] AI processing approx. FPS: 19.83
[INFO] Frames processed by AI system: 891
Repeating the test on my Pi3B+, same modified code, Coral stick and USB webcam, again via ssh -X I got:
[INFO] starting video stream…
[INFO] starting the FPS counter …
[INFO] Run elapsed time: 114.73
[INFO] AI processing approx. FPS: 6.52
[INFO] Frames processed by AI system: 748
The Odroid gets ~3X the performance of the the Pi3B+ on this code.
I think things like the Odroid XU-4 (octacore, with GigE, and USB3) should raise the bar for what we could hope to expect from the Pi4.
The Odroid XU-4 (straight from Korea) runs about ~$25-30 more (board, case, power supply) than the Pi3B+ if you ignore the shipping cost (~$20) which is zero here for the Pi as I can pick them up at my local “maker” supply store.
But even so 2X the cost for 3X the performance is usually a win.
Thanks for sharing all this, Wally!
Another followup to get an idea of the speed of the Coral, I ran this tutorial code on an i3-4025 “Industrial Mini PC” running Ubuntu Mate 16.04
With everything run as ssh -X I get:
[INFO] starting video stream…
[INFO] starting the FPS counter …
[INFO] Run elapsed time: 87.51
[INFO] AI processing approx. FPS: 46.36
[INFO] Frames processed by AI system: 4057
I’m impressed!
Running the openvino_real_time_object_detection.py tutorial code using Movidius NCS2 on this system I get:
[INFO] starting video stream…
[INFO] elapsed time: 174.52
[INFO] approx. FPS: 19.30
Using the original NCS:
[INFO] starting video stream…
[INFO] elapsed time: 84.82
[INFO] approx. FPS: 9.82
I think the Coral TPU is the bang/buck leader at the moment.
Running this sample code on my i7-8750H notebook I get 70 fps with the same webcam I used on the i3-4025 system above.
On this system the OpenVINO with NCS2 gets 23.7 fps on their Security Barrier demo, but 101 fps with CPU instead of MYRIAD.
Your OpenVINO tutorial on this system gets 20.2 fps with NCS2
The Coral on a <$100 Odroid XU-4 is getting about the same frame rate as the NCS2 on a $300+ i3 system!
Very nice post,
thanks again 🙂
maybe next step could be using jetson nano and coral usb accelerator (there is an usb-3 …)
Thanks Jerome, although if you are using a Jetson Nano there wouldn’t be much point of using the Coral USB Accelerator since the Nano has a GPU for inference.
It could make sense, depending on the situation. The Coral accelerator is faster for object detection.
Also there are other non-deep learning algorithms that use CUDA such as pose tracking/SLAM. That is one thing I will be trying, so the GPU on the Nano will be all used for mapping while the TPU is used for object detection.
Or just multiple deep learning models at once.
I think there still could be a point to running both.
I’ve verified that OpenVINO and Coral USB accelerators can be run simultaneously, so Coral and Jetson or Movidius and Jetson could be useful if, like me, you haven’t yet gotten into training models and are thus confined to using only publicly available models — not all models are available for every TPU, at least for now.
Its easy to envision a pipeline of detection and classification models using the “best” TPU for each model.
I think the hope is the Jetson Nano GPU will do better with YOLO models that Movidus and Coral.
at the right time the right information, as always. good work, keep it up.
maybe its of help, to resize images not only to 500px, instead directly to the tensor-size of the coral-coproc?
like its done here: https://github.com/freedomtan/edge_tpu_python_scripts/blob/master/object_detection_coral.py
check it out, if you can improve your examples…
thanks, keep on going..
Are the pretrained models able to detect drones as a unique object class or will it classify them as aircraft or birds or something else?
There is not a drone/quadcopter object class. It would either classify them as something else or fail to detect them entirely.
It’s a nice post!
Have you ever had experience in detecting camera occlusionr or have some resources worthy of reference?
I have some problem in this regard and don’t know how to do
Hi Adrian
Could you post how you record your video clips ? That would be very useful.
Also, with regard to the PI, have you had any success running them headless and sending video to another desktop over VNC ? vino only works if there is a running X server and desktop on the Pi and I’ve not had any luck with other alternatives on a truly headless Pi.
The video clip demos you mean? If so, I use VNC to stream to my Mac where I use Camtasia to record my screen.
I think I just fell in love with the Coral.
I was struggling to get the people counter program to work for my application, which is to watch over my 93 year old dad who still lives alone. I wanted to recognize a certain amount of movement within a certain period, by him crossing an imaginary (horizontal) line. If he does, fine, if he doesn’t, I will get an SMS as a warning.
I’m using an RPi M3+, and I just could not get the frame rate high enough to have a reliable recognition/tracking.
So I purchased the Coral USB stick.
It arrived today, and I quickly went through the demo programs. They all worked very well, as usual. I was a little apprehensive that I would be disappointed, but my hopes were getting up.
When I modified the detect video demo program to make it do what I needed, I was amazed about the throughput.
No more detection every 40 frames, centroids and complicated tracking to get a reasonable fps (between 6 and 7) at best. And still have major stuttering and still loosing track of objects.
Even with recognition now going at every frame, I get between 9 and 10 fps, and with that, the video is not stuttering at all and tracking is perfect. I’m delighted!
Thank you for bringing this subject to the masses!
Keep it up!
Paul
Thanks for the comment Paul, I’m happy the Coral is working out for you!
Hey Adrian,
Not sure if this has been mentioned, apologies if it has. But do you plan on testing the google tpu Dev Board? Interested to see if there is better performance there.
Thanks!
I hope to but I’m not sure when that may be.
What do you mean? Available time or access to the hardware? I got the dev board a few weeks ago and plan to do a bit of testing with it. If there are any suggestions you might have please let me know. I can publish the results here. Thanks!
Mainly just time right now. I’m working on the Raspberry Pi for Computer Vision book and haven’t quite made my way to the TPU chapters yet 🙂
Could you please add steps to this tutorial to install OpenCV within the coral virtual environment? I tried one of your previous tutorials for OpenCV 4.0 and failed to import package cv2. I might be using the wrong paths for RPi.
Make sure you are in the “coral” Python virtual environment and then install OpenCV:
There will be a few other packages you may need to install so make sure you refer to this tutorial.
Thank you very much for your detailed tutorials!
BTW, I think in the following code in the source codes ‘False’ should be changed to ‘True’ to use the Pi Camera. Please confirm about it 🙂
vs = VideoStream(usePiCamera=False).start()
Today, I’ve tested the Edge TPU USB with Raspberry Pi 3. It just works great!
I’m using the PiCamera for detect_video.py, and the frame rate is around 9.5~9.7 FPS for detecting objects using Mobilenet SSD V2.
FYI, “maximum frequency” setting for the EdgeTPU libraries seems make no difference for the inference speed in this case.
Thanks for sharing, Jaewoo!
Raspberry Pi 4B has been released!
https://www.raspberrypi.org/products/raspberry-pi-4-model-b/
Big news for us:
2 USB3 ports
Gigabit Ethernet
Ram options 1, 2, or 4GB.
4K display support.
Biggest negatives
USB C power port (you’ll need a new power supply 15W)
mini HDMI connectors — to make room for dual monitor support.
I hope to grab one from my local Maker Supplier as soon as they have them in stock.
Price, $35, 45, 55 depending on RAM option, very nice!
The RPi 4 looks awesome, I can’t wait to get my hands on it!
I think the Pi4B is a winner! I got the 1GB version, same price as the Pi3B+ and managed to get the Coral TPH API installed on the new Raspbian 10 “Buster” it requires, along with the OpenVINO OpenCV API.
Running this tutorial code on it I got ~30.3 fps
Recall from other posts, My Pi3B+ only got ~ 6.5 fps.
Other systems I’d tested
Odroid XU4 ~19.8 fps (Mate16, 32-bit, USB3)
i3-4025 ~46.4 fps (Mate16, 32-bit, USB3)
Jetson Nano ~44.5 fps (Ubuntu18, 64-bit, USB3)
With OpenVINO and the original NCS stick and your OpenVINO version of this tutorial, the Pi4B gets ~8.5 fps whereas it was ~6.5 fps on the Pi3B+.
My only NCS2 is in another system running stress tests I don’t want to interrupt for at least another day unless it hits a problem.
I think the Pi4 is in a sweet spot as a Coral host now as competitors like the XU-4 can’t keep up at higher expense and the Jetson nano ends up costing nearly three times a much for a 50% frame rate boost.
Amazing, thanks for sharing Wally!
Hi … your article is very interesting :). I’m trying to port code in C ++ but I found some problems with the installation of tensorflow lite. I can’t create the tensor library flow-lite.so.
Note that I am using Ubuntu 16 as the operating system and the installation instructions on the coral blog refer to other systems like IOS, Android and raspberry.
Can you help me solve the problem?
thanks a lot !!
Hey John — I only cover Python on this blog, no C++ implementations or support is provided. I wish you the very best of luck with it!
Hello Adrian:
Your tutorials are really helpful, thanks a lot.
but I have a question:
1.
In your code, the image seems keep the uint8 x 3 (RGB) format,
that means the model take (height, width, 3) uint8 as input, not float32?
2.
How to create a custom convolution model, which can run on Coral USB Accelerator?
I try several days, but still doesn’t work. Could you please give me some hints?
thank you!
Hey there, Raspberry Pi for Computer Vision covers how to train your own custom CNNs on the Google Coral. I would definitely point you there.
Hey Adrian,
Really impressed with the work you are doing. I am following your tutorials and that has helped a lot in building my knowledge and career. I have a question for you.
Have you tried Coral dev board integrated with its own Camera and try doing Object Detection? If so, can you please provide us the link? I have tried searching for python package which lets me to access this camera, to which I have failed in getting one. Please do this favor.
This will be of a great help!
Thanks in advance!
Cheers..
Hey Adrian,
I’m really sorry for troubling you. We are in need of this badly. Please provide us some solution.
Thanks!
Hey Krishna — I cover object detection with the Google Coral inside Raspberry Pi for Computer Vision. If you need help with object detection on the Coral definitely refer there.
Hi Adrian,
Thanks for these posts which introduced me to the coral. I am running it on Raspberry Pi 4B and so far they are pretty impressive combined.
My questions are – what is the reason for loading it into PIL and what format does PIL convert it into. Also does the image need to be 500 width for it to work.
Dani
PIL is just a simplified image processing library. We end up using the image as a NumPy array.
Hi Adrian, excellent tutorial. I wonder if it is possible to run a regression model in USB coral accelerator ? I don’t see an API for that ? is it possible ? Thanks in advance.
I’ll be covering how to use your own custom models on the Coral inside Raspberry Pi for Computer Vision. If you’re interested in running your mown models, definitely refer there.
Hi Adrian,
Thanks for posting such helpful and detailed tutorial. I was following your tutorial to run object detection. When I run the inference, it only shows the scores and results for the label that is at 0 index (in case of key-value pair, the value or label_id corresponding to where we write the key as 0).
I have been trying top figure out why that is happening but am not able to figure it out. While retraining the model, I used just 1 class at a time(eg. flower) and retrained it for that class, but while running inference, I am using the classes the model was retrained on (i.e. ‘Abyssinian’, ‘american_bulldog’ and ‘flower’).
Since, we are not using the config file while running inference, can it be the parameters while training that are affecting this like number of classes?
Since, I am not able to read the tflite file, I am not for the life of me able to figure it out!
I apologize for the verbose question but I would really appreciate any kind of help! Thank you so much for your time Adrian, I reallty appreciate it!!
Thanks!
Hi i just download your coral-classify-detection zip file and the codes were really outstanding!!
i really appreciated.
Buy the way, i have a question about the code. since i ran code, i found that the images can read up to three objects. is there any way to read more than three objects? Thanks!
Thanks, I’m glad the code was helpful to you.
As for your question, try adjusting the minimum probability required for a positive detection.
Thanks, but how can i adjust the minimum probability??
When you loop over the prediction results from the model check the probability and make sure it’s greater than N% (you will define N yourself manually through experimental tuning).
Thanks!! I solved the problem~I really appreciated by your help
Awesome, I’m glad that worked!
FYI.
I managed to get this detect_video.py tutorial running on my Coral Development board. Building OpenCV was the major issue. 4.1.0 failed, but 4.0.0 eventually succeeded following these instructions https://medium.com/@balaji_85683/installing-opencv-4-0-on-google-coral-dev-board-5c3a69d7f52f after I made the swapfile 2Gb instead of the suggested 1 Gb.
It only gets ~18.6 fps whereas I got ~30fps on the new Pi4B and ~19.8 on the Odroid XU-4.
Running my security DVR AI code on it I get essentially the same frame rate as on the Pi4B. the bottleneck it not pushing data in and out of the TPU. but getting images from multiple cameras into the AI subsystem and detection results out.
My take is the Coral Development board is only worthwhile if you will be running multiple pipelined inferences per image.
I think the Pi4B 2GB + Coral TPU stick is a bit less expensive and a whole lot easier to setup!
Downside to the Pi4B for any serious use needs a fan, the Coral Dev board has one built in, as does the Odroid, the Jetson has a massive heat sink that is nearly the same volume as the Coral Dev board 🙂
Thanks Wally!
Hi, Adrian.
I don’t have a Raspberry Pi. Could I do the sample using my cellphone camera? Like an App that I just open and point at things? Everything running my cellphone?
Great content!
Absolutely. I suggest following this tutorial on Keras and CoreML.
Hey Adrian. I just wanted to thank you for these tutorials. I find them very helpful. I have a question: I want to build an app that can detect my cat. Unfortunately, because he is black, he will not show up on most object detection programs, including ones specifically designed for pets. Do you have any advice or recommendations for me?
Thanks,
Tyler
A black cat shouldn’t cause any issue with object detection. Have you tried existing pre-trained models?
Hi Adrian,
thanks for the great tutorial. Ihave one question. I have an edge TPU board and I would like to run an object detection on it using my own video file. I am struggling with the input format of the video, does it mean, that the inference can be run only on the video with certain resolution?Do you have any experience with it?
Thanks a lot
Jan
Interesting, what is the input resolution/file format of your video?
Hi Adrian,
Thanks for sharing interesting blog and helping a community in the learning curve.
Could you please provide me any pointer that how can I convert tf object detection model to tflite model?
I want to deploy tflite model on a raspberry pi. But facing a problem with converting tf object detection model to tflite model.
Thanking you,
Saurabh
Converting models to TensorFlow Lite is covered inside Raspberry Pi for Computer Vision.
Thank you for your continuous guidance and sharing interesting blogs.
Happy New Year 2020!
Hi Adrian,
I am thinking of doing deep dream with tensorflow as per https://github.com/vasi1796/deep_dream_challenge/blob/master/deep_dream.py. Would it be possible to use the coral edge tpu to accelerate the processing. From what I can see you would need access to tensorflow commands such as graph and interactive sessions which may not be available with tensorflow lite.
Any pointers would be helpful
Good question. I’ve never tired to apply the deep dream algorithm with Google Coral’s TPU. I’d have to do some research there so I unfortunately don’t have a direct answer to that question.
Otherwise, my only other recommendation would be to read Deep Learning for Computer Vision with Python where I show how to implement deep dream by hand.
I hope that helps!
A question regarding the Coral USB (and the compute stick as well) — will they only accelerate the GPU when using classification models and the like, or is there a way they can also accelerate other processes like object tracking in openCV on the raspberry pi?
No, they will only enhance the performance of DL models running on the accelerator itself. It will not help speed up OpenCV or other processes running on the RPi.
Hi Adrian!
What if I wanted to limit the model to only detect people?
Thanks!
There are a few ways to accomplish this:
1. Fine-tune the model to only recognize people
2. Or, more easily, loop over the detected objects and discard any class/detection that is not a person
The latter is easier.
Hello ,
I would like your opinion on how to approach a project .
we are going to use pi 4 for real time face detection .
it is important for us that it will work fast and will be able to discover many persons at once .
is using google usb accelerator really faster then just using the open CV library directly ?
we also would like to know the distance of the object detected and would thank you if you could give us a good advice .
many thanks and keep up the great work you are doing!
Hi Adrian
I’ve recently stumbled across your tutorials and they look fantastic! I have not had the chance to try out this tutorial yet, but was wondering what is required to run object detection on videos already captured (i.e. not realtime)? I assume it is a simple modification to the input by changing “import VideoStream” to the directory where the videos are located?
You probably already have a tutorial about this on your website that I haven’t found yet…
Thanks!
Awsome tutorial, would it be a lot harder to get video detection via a rtsp stream instead of a local camera? Seems like it would be vastly more usefull that way 🙂
Code example would be much appreciated! 😀
I don’t like using RTSP. I would suggest using ImageZMQ instead.
That’s very helpful if you want to stream video across a network from a local camera (if i understand it correctly) but i have a bunch of unifi g3 cameras that speak RTSP very well and i’de simply like to hook those up and do near-realtime object detection and/or classification.
Thanks for your tutorials, very educative 🙂