In today’s tutorial, you’ll learn how to stream live video over a network with OpenCV. Specifically, you’ll learn how to implement Python + OpenCV scripts to capture and stream video frames from a camera to a server.
Every week or so I receive a comment on a blog post or a question over email that goes something like this:
Hi Adrian, I’m working on a project where I need to stream frames from a client camera to a server for processing using OpenCV. Should I use an IP camera? Would a Raspberry Pi work? What about RTSP streaming? Have you tried using FFMPEG or GStreamer? How do you suggest I approach the problem?
It’s a great question — and if you’ve ever attempted live video streaming with OpenCV then you know there are a ton of different options.
You could go with the IP camera route. But IP cameras can be a pain to work with. Some IP cameras don’t even allow you to access the RTSP (Real-time Streaming Protocol) stream. Other IP cameras simply don’t work with OpenCV’s cv2.VideoCapture function. An IP camera may be too expensive for your budget as well.
In those cases, you are left with using a standard webcam — the question then becomes, how do you stream the frames from that webcam using OpenCV?
Using FFMPEG or GStreamer is definitely an option. But both of those can be a royal pain to work with.
Today I am going to show you my preferred solution using message passing libraries, specifically ZMQ and ImageZMQ, the latter of which was developed by PyImageConf 2018 speaker, Jeff Bass. Jeff has put a ton of work into ImageZMQ and his efforts really shows.
As you’ll see, this method of OpenCV video streaming is not only reliable but incredibly easy to use, requiring only a few lines of code.
To learn how to perform live network video streaming with OpenCV, just keep reading!
Live video streaming over network with OpenCV and ImageZMQ
In the first part of this tutorial, we’ll discuss why, and under which situations, we may choose to stream video with OpenCV over a network.
From there we’ll briefly discuss message passing along with ZMQ, a library for high performance asynchronous messaging for distributed systems.
We’ll then implement two Python scripts:
- A client that will capture frames from a simple webcam
- And a server that will take the input frames and run object detection on them
Will be using Raspberry Pis as our clients to demonstrate how cheaper hardware can be used to build a distributed network of cameras capable of piping frames to a more powerful machine for additional processing.
By the end of this tutorial, you’ll be able to apply live video streaming with OpenCV to your own applications!
Why stream videos/frames over a network?
There are a number of reasons why you may want to stream frames from a video stream over a network with OpenCV.
To start, you could be building a security application that requires all frames to be sent to a central hub for additional processing and logging.
Or, your client machine may be highly resource constrained (such as a Raspberry Pi) and lack the necessary computational horsepower required to run computationally expensive algorithms (such as deep neural networks, for example).
In these cases, you need a method to take input frames captured from a webcam with OpenCV and then pipe them over the network to another system.
There are a variety of methods to accomplish this task (discussed in the introduction of the post), but today we are going to specifically focus on message passing.
What is message passing?
Message passing is a programming paradigm/concept typically used in multiprocessing, distributed, and/or concurrent applications.
Using message passing, one process can communicate with one or more other processes, typically using a message broker.
Whenever a process wants to communicate with another process, including all other processes, it must first send its request to the message broker.
The message broker receives the request and then handles sending the message to the other process(es).
If necessary, the message broker also sends a response to the originating process.
As an example of message passing let’s consider a tremendous life event, such as a mother giving birth to a newborn child (process communication depicted in Figure 2 above). Process A, the mother, wants to announce to all other processes (i.e., the family), that she had a baby. To do so, Process A constructs the message and sends it to the message broker.
The message broker then takes that message and broadcasts it to all processes.
All other processes then receive the message from the message broker.
These processes want to show their support and happiness to Process A, so they construct a message saying their congratulations:
These responses are sent to the message broker which in turn sends them back to Process A (Figure 3).
This example is a dramatic simplification of message passing and message broker systems but should help you understand the general algorithm and the type of communication the processes are performing.
You can very easily get into the weeds studying these topics, including various distributed programming paradigms and types of messages/communication (1:1 communication, 1:many, broadcasts, centralized, distributed, broker-less etc.).
As long as you understand the basic concept that message passing allows processes to communicate (including processes on different machines) then you will be able to follow along with the rest of this post.
What is ZMQ?
ZeroMQ, or simply ZMQ for short, is a high-performance asynchronous message passing library used in distributed systems.
Both RabbitMQ and ZeroMQ are some of the most highly used message passing systems.
However, ZeroMQ specifically focuses on high throughput and low latency applications — which is exactly how you can frame live video streaming.
When building a system to stream live videos over a network using OpenCV, you would want a system that focuses on:
- High throughput: There will be new frames from the video stream coming in quickly.
- Low latency: As we’ll want the frames distributed to all nodes on the system as soon as they are captured from the camera.
ZeroMQ also has the benefit of being extremely easy to both install and use.
Jeff Bass, the creator of ImageZMQ (which builds on ZMQ), chose to use ZMQ as the message passing library for these reasons — and I couldn’t agree with him more.
The ImageZMQ library
Jeff Bass is the owner of Yin Yang Ranch, a permaculture farm in Southern California. He was one of the first people to join PyImageSearch Gurus, my flagship computer vision course. In the course and community he has been an active participant in many discussions around the Raspberry Pi.
Jeff has found that Raspberry Pis are perfect for computer vision and other tasks on his farm. They are inexpensive, readily available, and astoundingly resilient/reliable.
At PyImageConf 2018 Jeff spoke about his farm and more specifically about how he used Raspberry Pis and a central computer to manage data collection and analysis.
The heart of his project is a library that he put together called ImageZMQ.
ImageZMQ solves the problem of real-time streaming from the Raspberry Pis on his farm. It is based on ZMQ and works really well with OpenCV.
Plain and simple, it just works. And it works really reliably.
I’ve found it to be more reliable than alternatives such as GStreamer or FFMPEG streams. I’ve also had better luck with it than using RTSP streams.
You can learn the details of ImageZMQ by studying Jeff’s code on GitHub.
Jeff’s slides from PyImageConf 2018 are also available here.
In a few days, I’ll be posting my interview with Jeff Bass on the blog as well.
Let’s configure our clients and server with ImageZMQ and put it them to work!
Configuring your system and installing required packages
Installing ImageZMQ is quite easy.
First, let’s pip install a few packages into your Python virtual environment (assuming you’re using one). If you need to set up pip and virtual environments, please refer to my pip install opencv tutorial first.
Then use the following commands:
$ workon <env_name> # my environment is named py3cv4 $ pip install opencv-contrib-python $ pip install imagezmq $ pip install imutils
You must install these packages on both the clients and server. Provided you didn’t encounter any issues you are now ready to move on.
Note: On your Raspberry Pi, we recommend installing this version of OpenCV: pip install opencv-contrib-python==4.1.0.25.
Preparing clients for ImageZMQ
ImageZMQ must be installed on each client and the central server.
In this section, we’ll cover one important difference for clients.
Our code is going to use the hostname of the client to identify it. You could use the IP address in a string for identification, but setting a client’s hostname allows you to more easily identify the purpose of the client.
In this example, we’ll assume you are using a Raspberry Pi running Raspbian. Of course, your client could run Windows Embedded, Ubuntu, macOS, etc., but since our demo uses Raspberry Pis, let’s learn how to change the hostname on the RPi.
To change the hostname on your Raspberry Pi, fire up a terminal (this could be over an SSH connection if you’d like).
Then run the raspi-config command:
$ sudo raspi-config
You’ll be presented with this terminal screen:
raspi-config. Shown is the raspi-config home screen.Navigate to “2 Network Options” and press enter.
raspi-config network settings page.Then choose the option “N1 Hostname”.
You can now change your hostname and select “<Ok>”.
You will be prompted to reboot — a reboot is required.
I recommend naming your Raspberry Pis like this: pi-location . Here are a few examples:
pi-garagepi-frontporchpi-livingroompi-driveway- …you get the idea.
This way when you pull up your router page on your network, you’ll know what the Pi is for and its corresponding IP address. On some networks, you could even connect via SSH without providing the IP address like this:
$ ssh pi@pi-frontporch
As you can see, it will likely save some time later.
Defining the client and server relationship
Before we actually implement network video streaming with OpenCV, let’s first define the client/server relationship to ensure we’re on the same page and using the same terms:
- Client: Responsible for capturing frames from a webcam using OpenCV and then sending the frames to the server.
- Server: Accepts frames from all input clients.
You could argue back and forth as to which system is the client and which is the server.
For example, a system that is capturing frames via a webcam and then sending them elsewhere could be considered a server — the system is undoubtedly serving up frames.
Similarly, a system that accepts incoming data could very well be the client.
However, we are assuming:
- There is at least one (and likely many more) system responsible for capturing frames.
- There is only a single system used for actually receiving and processing those frames.
For these reasons, I prefer to think of the system sending the frames as the client and the system receiving/processing the frames as the server.
You may disagree with me, but that is the client-server terminology we’ll be using throughout the remainder of this tutorial.
Project structure
Be sure to grab the “Downloads” for today’s project.
From there, unzip the files and navigate into the project directory.
You may use the tree command to inspect the structure of the project:
$ tree . ├── MobileNetSSD_deploy.caffemodel ├── MobileNetSSD_deploy.prototxt ├── client.py └── server.py 0 directories, 4 files
Note: If you’re going with the third alternative discussed above, then you would need to place the imagezmq source directory in the project as well.
The first two files listed in the project are the pre-trained Caffe MobileNet SSD object detection files. The server (server.py ) will take advantage of these Caffe files using OpenCV’s DNN module to perform object detection.
The client.py script will reside on each device which is sending a stream to the server. Later on, we’ll upload client.py onto each of the Pis (or another machine) on your network so they can send video frames to the central location.
Implementing the client OpenCV video streamer (i.e., video sender)
Let’s start by implementing the client which will be responsible for:
- Capturing frames from the camera (either USB or the RPi camera module)
- Sending the frames over the network via ImageZMQ
Open up the client.py file and insert the following code:
# import the necessary packages
from imutils.video import VideoStream
import imagezmq
import argparse
import socket
import time
# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-s", "--server-ip", required=True,
help="ip address of the server to which the client will connect")
args = vars(ap.parse_args())
# initialize the ImageSender object with the socket address of the
# server
sender = imagezmq.ImageSender(connect_to="tcp://{}:5555".format(
args["server_ip"]))
We start off by importing packages and modules on Lines 2-6:
- Pay close attention here to see that we’re importing
imagezmqin our client-side script. VideoStreamwill be used to grab frames from our camera.- Our
argparseimport will be used to process a command line argument containing the server’s IP address (--server-ipis parsed on Lines 9-12). - The
socketmodule of Python is simply used to grab the hostname of the Raspberry Pi. - Finally,
timewill be used to allow our camera to warm up prior to sending frames.
Lines 16 and 17 simply create the imagezmq sender object and specify the IP address and port of the server. The IP address will come from the command line argument that we already established. I’ve found that port 5555 doesn’t usually have conflicts, so it is hardcoded. You could easily turn it into a command line argument if you need to as well.
Let’s initialize our video stream and start sending frames to the server:
# get the host name, initialize the video stream, and allow the # camera sensor to warmup rpiName = socket.gethostname() vs = VideoStream(usePiCamera=True).start() #vs = VideoStream(src=0).start() time.sleep(2.0) while True: # read the frame from the camera and send it to the server frame = vs.read() sender.send_image(rpiName, frame)
Now, we’ll grab the hostname, storing the value as rpiName (Line 21). Refer to “Preparing clients for ImageZMQ” above to set your hostname on a Raspberry Pi.
From there, our VideoStream object is created to connect grab frames from our PiCamera. Alternatively, you can use any USB camera connected to the Pi by commenting Line 22 and uncommenting Line 23.
This is the point where you should also set your camera resolution. We are just going to use the maximum resolution so the argument is not provided. But if you find that there is a lag, you are likely sending too many pixels. If that is the case, you may reduce your resolution quite easily. Just pick from one of the resolutions available for the PiCamera V2 here: PiCamera ReadTheDocs. The second table is for V2.
Once you’ve chosen the resolution, edit Line 22 like this:
vs = VideoStream(usePiCamera=True, resolution=(320, 240)).start()
Note: The resolution argument won’t make a difference for USB cameras since they are all implemented differently. As an alternative, you can insert a frame = imutils.resize(frame, width=320) between Lines 28 and 29 to resize the frame manually.
From there, a warmup sleep time of 2.0 seconds is set (Line 24).
Finally, our while loop on Lines 26-29 grabs and sends the frames.
As you can see, the client is quite simple and straightforward!
Let’s move on to the actual server.
Implementing the OpenCV video server (i.e., video receiver)
The live video server will be responsible for:
- Accepting incoming frames from multiple clients.
- Applying object detection to each of the incoming frames.
- Maintaining an “object count” for each of the frames (i.e., count the number of objects).
Let’s go ahead and implement the server — open up the server.py file and insert the following code:
# import the necessary packages
from imutils import build_montages
from datetime import datetime
import numpy as np
import imagezmq
import argparse
import imutils
import cv2
# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-p", "--prototxt", required=True,
help="path to Caffe 'deploy' prototxt file")
ap.add_argument("-m", "--model", required=True,
help="path to Caffe pre-trained model")
ap.add_argument("-c", "--confidence", type=float, default=0.2,
help="minimum probability to filter weak detections")
ap.add_argument("-mW", "--montageW", required=True, type=int,
help="montage frame width")
ap.add_argument("-mH", "--montageH", required=True, type=int,
help="montage frame height")
args = vars(ap.parse_args())
On Lines 2-8 we import packages and libraries. In this script, most notably we’ll be using:
build_montages: To build a montage of all incoming frames.imagezmq: For streaming video from clients. In our case, each client is a Raspberry Pi.imutils: My package of OpenCV and other image processing convenience functions available on GitHub and PyPi.cv2: OpenCV’s DNN module will be used for deep learning object detection inference.
Are you wondering where imutils.video.VideoStream is? We usually use my VideoStream class to read frames from a webcam. However, don’t forget that we’re using imagezmq for streaming frames from clients. The server doesn’t have a camera directly wired to it.
Let’s process five command line arguments with argparse:
--prototxt: The path to our Caffe deep learning prototxt file.--model: The path to our pre-trained Caffe deep learning model. I’ve provided MobileNet SSD in the “Downloads” but with some minor changes, you could elect to use an alternative model.--confidence: Our confidence threshold to filter weak detections.--montageW: This is not width in pixels. Rather this is the number of columns for our montage. We’re going to stream from four raspberry Pis today, so you could do 2×2, 4×1, or 1×4. You could also do, for example, 3×3 for nine clients, but 5 of the boxes would be empty.--montageH: The number of rows for your montage. See the--montageWexplanation.
Let’s initialize our ImageHub object along with our deep learning object detector:
# initialize the ImageHub object
imageHub = imagezmq.ImageHub()
# initialize the list of class labels MobileNet SSD was trained to
# detect, then generate a set of bounding box colors for each class
CLASSES = ["background", "aeroplane", "bicycle", "bird", "boat",
"bottle", "bus", "car", "cat", "chair", "cow", "diningtable",
"dog", "horse", "motorbike", "person", "pottedplant", "sheep",
"sofa", "train", "tvmonitor"]
# load our serialized model from disk
print("[INFO] loading model...")
net = cv2.dnn.readNetFromCaffe(args["prototxt"], args["model"])
Our server needs an ImageHub to accept connections from each of the Raspberry Pis. It essentially uses sockets and ZMQ for receiving frames across the network (and sending back acknowledgments).
Our MobileNet SSD object CLASSES are specified on Lines 29-32. If you aren’t familiar with the MobileNet Single Shot Detector, please refer to this blog post or Deep Learning for Computer Vision with Python.
From there we’ll instantiate our Caffe object detector on Line 36.
Initializations come next:
# initialize the consider set (class labels we care about and want
# to count), the object count dictionary, and the frame dictionary
CONSIDER = set(["dog", "person", "car"])
objCount = {obj: 0 for obj in CONSIDER}
frameDict = {}
# initialize the dictionary which will contain information regarding
# when a device was last active, then store the last time the check
# was made was now
lastActive = {}
lastActiveCheck = datetime.now()
# stores the estimated number of Pis, active checking period, and
# calculates the duration seconds to wait before making a check to
# see if a device was active
ESTIMATED_NUM_PIS = 4
ACTIVE_CHECK_PERIOD = 10
ACTIVE_CHECK_SECONDS = ESTIMATED_NUM_PIS * ACTIVE_CHECK_PERIOD
# assign montage width and height so we can view all incoming frames
# in a single "dashboard"
mW = args["montageW"]
mH = args["montageH"]
print("[INFO] detecting: {}...".format(", ".join(obj for obj in
CONSIDER)))
In today’s example, I’m only going to CONSIDER three types of objects from the MobileNet SSD list of CLASSES . We’re considering (1) dogs, (2) persons, and (3) cars on Line 40.
We’ll soon use this CONSIDER set to filter out other classes that we don’t care about such as chairs, plants, monitors, or sofas which don’t typically move and aren’t interesting for this security type project.
Line 41 initializes a dictionary for our object counts to be tracked in each video feed. Each count is initialized to zero.
A separate dictionary, frameDict is initialized on Line 42. The frameDict dictionary will contain the hostname key and the associated latest frame value.
Lines 47 and 48 are variables which help us determine when a Pi last sent a frame to the server. If it has been a while (i.e. there is a problem), we can get rid of the static, out of date image in our montage. The lastActive dictionary will have hostname keys and timestamps for values.
Lines 53-55 are constants which help us to calculate whether a Pi is active. Line 55 itself calculates that our check for activity will be 40 seconds. You can reduce this period of time by adjusting ESTIMATED_NUM_PIS and ACTIVE_CHECK_PERIOD on Lines 53 and 54.
Our mW and mH variables on Lines 59 and 60 represent the width and height (columns and rows) for our montage. These values are pulled directly from the command line args dictionary.
Let’s loop over incoming streams from our clients and processing the data!
# start looping over all the frames
while True:
# receive RPi name and frame from the RPi and acknowledge
# the receipt
(rpiName, frame) = imageHub.recv_image()
imageHub.send_reply(b'OK')
# if a device is not in the last active dictionary then it means
# that its a newly connected device
if rpiName not in lastActive.keys():
print("[INFO] receiving data from {}...".format(rpiName))
# record the last active time for the device from which we just
# received a frame
lastActive[rpiName] = datetime.now()
We begin looping on Line 65.
Lines 68 and 69 grab an image from the imageHub and send an ACK message. The result of imageHub.recv_image is rpiName , in our case the hostname, and the video frame itself.
It is really as simple as that to receive frames from an ImageZMQ video stream!
Lines 73-78 perform housekeeping duties to determine when a Raspberry Pi was lastActive .
Let’s perform inference on a given incoming frame :
# resize the frame to have a maximum width of 400 pixels, then
# grab the frame dimensions and construct a blob
frame = imutils.resize(frame, width=400)
(h, w) = frame.shape[:2]
blob = cv2.dnn.blobFromImage(cv2.resize(frame, (300, 300)),
0.007843, (300, 300), 127.5)
# pass the blob through the network and obtain the detections and
# predictions
net.setInput(blob)
detections = net.forward()
# reset the object count for each object in the CONSIDER set
objCount = {obj: 0 for obj in CONSIDER}
Lines 82-90 perform object detection on the frame :
- The
framedimensions are computed. - A
blobis created from the image (see this post for more details about how OpenCV’s blobFromImage function works). - The
blobis passed through the neural net.
From there, on Line 93 we reset the object counts to zero (we will be populating the dictionary with fresh count values shortly).
Let’s loop over the detections with the goal of (1) counting, and (2) drawing boxes around objects that we are considering:
# loop over the detections
for i in np.arange(0, detections.shape[2]):
# extract the confidence (i.e., probability) associated with
# the prediction
confidence = detections[0, 0, i, 2]
# filter out weak detections by ensuring the confidence is
# greater than the minimum confidence
if confidence > args["confidence"]:
# extract the index of the class label from the
# detections
idx = int(detections[0, 0, i, 1])
# check to see if the predicted class is in the set of
# classes that need to be considered
if CLASSES[idx] in CONSIDER:
# increment the count of the particular object
# detected in the frame
objCount[CLASSES[idx]] += 1
# compute the (x, y)-coordinates of the bounding box
# for the object
box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])
(startX, startY, endX, endY) = box.astype("int")
# draw the bounding box around the detected object on
# the frame
cv2.rectangle(frame, (startX, startY), (endX, endY),
(255, 0, 0), 2)
On Line 96 we begin looping over each of the detections . Inside the loop, we proceed to:
- Extract the object
confidenceand filter out weak detections (Lines 99-103). - Grab the label
idx(Line 106) and ensure that the label is in theCONSIDERset (Line 110). For each detection that has passed the two checks (confidencethreshold and inCONSIDER), we will:- Increment the
objCountfor the respective object (Line 113). - Draw a
rectanglearound the object (Lines 117-123).
- Increment the
Next, let’s annotate each frame with the hostname and object counts. We’ll also build a montage to display them in:
# draw the sending device name on the frame
cv2.putText(frame, rpiName, (10, 25),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)
# draw the object count on the frame
label = ", ".join("{}: {}".format(obj, count) for (obj, count) in
objCount.items())
cv2.putText(frame, label, (10, h - 20),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255,0), 2)
# update the new frame in the frame dictionary
frameDict[rpiName] = frame
# build a montage using images in the frame dictionary
montages = build_montages(frameDict.values(), (w, h), (mW, mH))
# display the montage(s) on the screen
for (i, montage) in enumerate(montages):
cv2.imshow("Home pet location monitor ({})".format(i),
montage)
# detect any kepresses
key = cv2.waitKey(1) & 0xFF
On Lines 126-133 we make two calls to cv2.putText to draw the Raspberry Pi hostname and object counts.
From there we update our frameDict with the frame corresponding to the RPi hostname.
Lines 139-144 create and display a montage of our client frames. The montage will be mW frames wide and mH frames tall.
Keypresses are captured via Line 147.
The last block is responsible for checking our lastActive timestamps for each client feed and removing frames from the montage that have stalled. Let’s see how it works:
# if current time *minus* last time when the active device check
# was made is greater than the threshold set then do a check
if (datetime.now() - lastActiveCheck).seconds > ACTIVE_CHECK_SECONDS:
# loop over all previously active devices
for (rpiName, ts) in list(lastActive.items()):
# remove the RPi from the last active and frame
# dictionaries if the device hasn't been active recently
if (datetime.now() - ts).seconds > ACTIVE_CHECK_SECONDS:
print("[INFO] lost connection to {}".format(rpiName))
lastActive.pop(rpiName)
frameDict.pop(rpiName)
# set the last active check time as current time
lastActiveCheck = datetime.now()
# if the `q` key was pressed, break from the loop
if key == ord("q"):
break
# do a bit of cleanup
cv2.destroyAllWindows()
There’s a lot going on in Lines 151-162. Let’s break it down:
- We only perform a check if at least
ACTIVE_CHECK_SECONDShave passed (Line 151). - We loop over each key-value pair in
lastActive(Line 153):- If the device hasn’t been active recently (Line 156) we need to remove data (Lines 158 and 159). First we remove (
pop) therpiNameand timestamp fromlastActive. Then therpiNameand frame are removed from theframeDict.
- If the device hasn’t been active recently (Line 156) we need to remove data (Lines 158 and 159). First we remove (
- The
lastActiveCheckis updated to the current time on Line 162.
Effectively this will help us get rid of expired frames (i.e. frames that are no longer real-time). This is really important if you are using the ImageHub server for a security application. Perhaps you are saving key motion events like a Digital Video Recorder (DVR). The worst thing that could happen if you don’t get rid of expired frames is that an intruder kills power to a client and you don’t realize the frame isn’t updating. Think James Bond or Jason Bourne sort of spy techniques.
Last in the loop is a check to see if the "q" key has been pressed — if so we break from the loop and destroy all active montage windows (Lines 165-169).
Streaming video over network with OpenCV
Now that we’ve implemented both the client and the server, let’s put them to the test.
Make sure you use the “Downloads” section of this post to download the source code.
From there, upload the client to each of your Pis using SCP:
$ scp client.py pi@192.168.1.10:~ $ scp client.py pi@192.168.1.11:~ $ scp client.py pi@192.168.1.12:~ $ scp client.py pi@192.168.1.13:~
In this example, I’m using four Raspberry Pis, but four aren’t required — you can use more or less. Be sure to use applicable IP addresses for your network.
You also need to follow the installation instructions to install ImageZMQ on each Raspberry Pi. See the “Configuring your system and installing required packages” section in this blog post.
Before we start the clients, we must start the server. Let’s fire it up with the following command:
$ python server.py --prototxt MobileNetSSD_deploy.prototxt \ --model MobileNetSSD_deploy.caffemodel --montageW 2 --montageH 2
Once your server is running, go ahead and start each client pointing to the server. Here is what you need to do on each client, step-by-step:
- Open an SSH connection to the client:
ssh pi@192.168.1.10 - Start screen on the client:
screen - Source your profile:
source ~/.profile - Activate your environment:
workon py3cv4 - Install ImageZMQ using instructions in “Configuring your system and installing required packages”.
- Run the client:
python client.py --server-ip 192.168.1.5
As an alternative to these steps, you may start the client script on reboot.
Automagically, your server will start bringing in frames from each of your Pis. Each frame that comes in is passed through the MobileNet SSD. Here’s a quick demo of the result:
A full video demo can be seen below:
What's next? I recommend PyImageSearch University.
30+ total classes • 39h 44m video • Last updated: 12/2021
★★★★★ 4.84 (128 Ratings) • 3,000+ Students Enrolled
I strongly believe that if you had the right teacher you could master computer vision and deep learning.
Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?
That’s not the case.
All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.
If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.
Inside PyImageSearch University you'll find:
- ✓ 30+ courses on essential computer vision, deep learning, and OpenCV topics
- ✓ 30+ Certificates of Completion
- ✓ 39h 44m on-demand video
- ✓ Brand new courses released every month, ensuring you can keep up with state-of-the-art techniques
- ✓ Pre-configured Jupyter Notebooks in Google Colab
- ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
- ✓ Access to centralized code repos for all 500+ tutorials on PyImageSearch
- ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
- ✓ Access on mobile, laptop, desktop, etc.
Summary
In this tutorial, you learned how to stream video over a network using OpenCV and the ImageZMQ library.
Instead of relying on IP cameras or FFMPEG/GStreamer, we used a simple webcam and a Raspberry Pi to capture input frames and then stream them to a more powerful machine for additional processing using a distributed system concept called message passing.
Thanks to Jeff Bass’ hard work (the creator of ImageZMQ) our implementation required only a few lines of code.
If you are ever in a situation where you need to stream live video over a network, definitely give ImageZMQ a try — I think you’ll find it super intuitive and easy to use.
I’ll be back in a few days with an interview with Jeff Bass as well!
To download the source code to this post, and be notified when future tutorials are published here on PyImageSearch, just enter your email address in the form below!
Download the Source Code and FREE 17-page Resource Guide
Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!
How does this compare with MQTT, performance wise?
I use paho-mqtt MQTT in Python and mosquitto MQTT broker running locally on the Pi to pass images among separate processes or to processes running on different machines.
MQTT has the “virtue” that you can buy “cloud broker” services from IBM, HiveMQ, etc. and get remote access without needing to run your own servers exposed to the internet
In any event thanks for posting this, looking at the imagezmq git hub, it looks to be potentially veryuseful. I’d encourage Jeff to get it pip install-able for wider usage.
If I can free up some time, I might try to take this sample and modify it to use MQTT for comparison.
ImageZMQ was designed with ZMQ in mind. It would definitely require some updating to get it to run on MQTT and gather comparisons.
I have MQTT sending and receiving jpg images. I was thinking of “dropping” my MQTT code into the timing test code on the ImageZMQ GitHub to compare.
I’ve downloaded the ImageZMQ, just need to find some time to do it.
Sorry I wasn’t clearer, your previous OpenVINO tutorial solved some problems for me that has kept me busy implementing the solution and testing.
Hi Wally,
I did some performance tests this morning comparing MQTT (that I use for the same purpose as you, sending images across my network) with imageZMQ, in this case using send_jpg instead of send_image since I needed to send the image as a buffer
In the tests I sent the same 95k jpg image ten times as fast as possible from one Pi to another over my wireless lan connections for both
The results shows that there was no signinficant difference at all, it took around 3 seconds for both technologies
mqtt DONE! 2.3724958896636963
mqtt DONE! 3.370811939239502
mqtt DONE! 4.014646530151367
mqtt DONE! 2.674704074859619
mqtt DONE! 3.1588287353515625
zmq DONE! 2.7648768424987793
zmq DONE! 5.127021312713623
zmq DONE! 3.3753623962402344
zmq DONE! 2.6726326942443848
zmq DONE! 3.2702481746673584
Thank you for sharing, Walter!
same concepts. mqtt can transport whatever you want. the core part is the broker (there also a lot of mqtt borkers out there). you could use an mqtt broker instead of zmq. apache has many adapters to support any kind of protocol for the broker.
Thanks for your great work!
Awesome content Adrian. What machine did you use for processing on the server side? Was it GPU enabled because running object detection on frames from 4 cameras can be quite computationally expensive.
It was actually a MacBook Pro running YOLO via the CPU!
I know you asked Adrian, but wanted to share this is working on a Raspberry Pi CM3 on ethernet as well as a Raspberry Pi 3A+ over wifi. I’m getting about 1 frame every 4 seconds with each pi camera streaming at max resolution like Adrian has.
It’ll max out all 4 cores of the cpu and use about 400MB of RAM.
Thanks for sharing, Marshall!
Great post, Adrian,
I loved the Jeff Bass’ presentation at PyImageConf.
One of these days I am going to finally get furiously angry with opencv’s USB Camera interface and start using libuvc. The Pi Camera interface by comparison is very clean and straightforward and consistent.
Thanks David. I’ll have an interview publishing with Jeff on Wednesday as well 🙂
Is it possible to send it via http to see it on a web page?
I’ll be covering that in my upcoming Raspberry Pi for Computer Vision book, stay tuned!
Hi i want to ask you a question say i implemented this project now what i want to do is store the output to a database or simply say i am running open cv face recognition on client and when a person is recognized i want a json of his name in real time streaming. So what should i do?
You could simply encode the name of the person as JSON string. That’s not really a computer vision question though. I would instead suggest you read up on Python programming basics, including JSON serialization and database fundamentals. That will better enable you to complete your project.
Hi,
Very nice project. but I have an error message : module ‘imagezmq’ has no attribute ‘ImageHub’ when I launch the server.
Have you an idea.
Tahnks
It sounds like you have not properly installed the “imagezmq” library. Double-check your sym-links or simply put the “imagezmq” Python module in the working directory of your project.
I was having the same issue running the server. Putting the imagezmq folder that contains __init__.py, __version__.py and imagezmq.py in the working folder solved the issue.
please let me knoe how to install and configure in imagezmq
I had the same problem, I solved it in the following way.
1.- copy the imagezmq folder inside the project directory
2.- in the server.py file change:
import imagezmq
by
from imagezmq import imagezmq
I hope to be helpful
regards
I had the same issue and this fixed mine as well.
Alba – did you resolve this ? I have the same issue.
Its ok — Hans M’s solution worked for me.
Awesome, glad to hear it Geoff!
Awesome Adrian. It’s great
I have two questions about.
1. It’s possible to share multiple ip camera by one Ras PI? my memory is only 1GB.
2. Can I put a password for my app and share them on the Internet?
1. You mean have two cameras on a single Pi? Yes, absolutely. You would just have two client scripts running on the Pi, each accessing its respective camera.
2. Yes, but you would need to code any authentication yourself.
I can confirm this works, I am streaming from a picam and a USB camera.
I should add that I changed the client code very slightly to:
while True:
# read the frame from the camera and send it to the server
frame = vs.read()
if frame is not None:
sender.send_image(rpiName, frame)
else:
print(“no frame to send {}”.format(time.time()))
time.sleep(1.0)
I added a 2nd USB camera along with the picamera, 3 cameras in total which was mostly fine but run into what was either specific to the USB camera I was using, or USB bus limits. I suspect it was the former. By confirming there is a frame, the client doesn’t crash when there is no frame ready.
I also added time to the frame overlay on the server so it is more obvious when new frames come in on static scenes.
I suspect if hosting multiple cameras, it’d be better to read and send them consecutively from a single process rather than have multiple instances running. However, this is a great tutorial to build from.
Hi Adrian and thanks for your Awesome website, I have a question:
Is it possible to reduce the bandwidth of streaming video by doing some compression algorithms on it? I mean choosing for example MJPEG or MPEG4 or some other formats for our streaming video.
thanks a lot
Yes, the smaller your frames are (in terms of spatial dimensions) combined with your compression will reduce network load and therefore reduce latency. You just need to be careful that the compression can still happen in real-time on the client.
I have MQTT sending and receiving jpg images. I was thinking of “dropping” my MQTT code into the timing test code on the ImageZMQ GitHub to compare.
I’ve downloaded the ImageZMQ, just need to find some time to do it.
Sorry I wasn’t clearer, your previous OpenVINO tutorial solved some problems for me that has kept me busy implementing the solution and testing.
Thanks for the great article! I’m looking at using WebRTC and the open source Kurento server to stream content from a laptop camera, apply an OpenCV filter on the server side, then stream the results out to a browser endpoint. You mentioned you had some trouble with RTSP? Was this just due to some camera’s not being able to publish on that protocol? Are there other hurdles with RTSP / RTC that are important to consider?
Hey Brett — see my reply to Heughens Jean.
Any suggestions on making the pi weatherproof for outdoor applications?
Jeff discusses his weather/waterproofing in this interview.
I tried to install imagezmq on your colleague’s site at https://github.com/jeffbass/imagezmq using the git command.
and got the following error
Any idea?
Thank you,
Anthony of Sydney
Are you using Windows? If so, you can find the solution here.
This is very useful tool!
One question. If the processing speed on server is slow, will it take the latest frame from the queue on the second loop? or the second one?
Sorry, I’m not sure what you mean? Could you elaborate?
I do agree with you after my work of ffmpeg and cgstreamer.
The transmission on internet of opencv frame is very pretty work.It will great decrease hardware cost.
Thank you so much.Great Adrian
You are welcome, I’m glad you enjoyed the tutorial!
Dear Adrian,
This works basically as you have presented it…but the client.py/server.py is really loading the Pi heavily, the CPU load goes up in the roof
And still, OpenCV seems not able to keep the frame rate at the same level as the other video software (Motion) I’m currently using in my solution. With Motion the CPU load in the Pi is just around 15% with a frame rate of 10 fps including motion detection enabled
Motion -> MQTT -> my python dnn analyzer server
To send images via MQTT, I just send them as byte arrays and it seems to be very fast
I’m wondering if there is a performance advantage in using imageZMQ instead of pure MQTT. I noticed that zmq uses Cython that I believe should be great for performace reasons, not sure if MQTT does the same
Anyway, my best regards & thanks for all great writing & sharing
Walter
Hey Walter, I see you already replied to the thread with Wally regarding MQTT. Thanks for doing that.
As far as CPU usage, are you sure it’s just not the threading of the VideoStream? You can reduce load by using “cv2.VideoCapture” and only polling frames when you want them.
Why not use IP cameras? They are cheap, compact and don’t require RasPI on far end
I’ve just checked my IP-cam module (ONVIF) with your code from face-detection post.
It works like a sharm.
Thank you.
Take a look at the intro of the post. If you can use an IP camera, great, but sometimes it’s not possible.
I’ve been trying to do the same. Got it to work but the images being displayed on the web just freezes and updates after minutes. Please how were you able to update your web display if you could..
Hey Adrian,
What were some of the issues you were seeing when you tried streaming with rtsp? I’m assuming this should also work on a tx2 or nano.
Yes, this will also work on the TX2, Nano, Coral, etc.
As for RTSP, gstreamer, etc., you end up going down a rabbit hole of trying to get the correct parameters to work, having them work one day, and fail the next. I hated debugging it. ImageZMQ makes it far easier and reliable.
Hi Adrian, thanks for this post. But how about to forward montage or cv2.imshow output include detected rectangle from server to website (like custom html with dashboard and video player) ?
I’m covering that exact topic inside my Raspberry Pi for Computer Vision book. If you’re interested, you can use the Kickstarter page to pre-order at reduced prices.
Inspiring post.
I want to get images from camera located at a remote site over a GSM enabled router. Will it work? What is needed?
Kindly shed more light on the configuration of the server side.
Which python night vision library can you advice?will you be covering in “wildlife monitoring ” section of the book?
1. Yes, I will be covering wildlife detection inside the book, including wildlife at night. If you haven’t pre-ordered your copy yet, you can use this page to do so.
2. As for the GSM router, as long as you have a publicly access IP address it will work.
Hey Adrian! Thank you for continuing to write articles like this!
Do you think it would be possible for a Raspberry Pi with the Movidius to have enough power to process the inputs from the streaming cameras?
In your article on OpenVino, it seems you are processing a similar payload as would be received in this article.
https://www.pyimagesearch.com/2019/04/08/openvino-opencv-and-movidius-ncs-on-the-raspberry-pi/
That really depends on:
1. How computationally expensive the model is
2. The # of incoming streams
If you’re going to use the Raspberry Pi + NCS I guess my question would be why not just run inference there on the Pi instead of sending it over the network?
any idea how to stop the server hanging at (rpiName, frame) = hub.recv_image()
Double-check that your server is running ZMQ. Secondly, make sure your IP address and port numbers are correct.
Hello, I’ve been reading your tutorials an they really helped me learning to use opencv. Could you suggest a way of streaming the captured video to a webpage? Thanks in advance.
I’m actually covering that exact topic in my Raspberry Pi for Computer Vision book. You can find the Kickstarter page for the book here if you would like to pre-order a copy.
Thanks for this. It is interesting to me not just for the video streaming part, but I had never heard of zeromq and libraries like it before.
Whenever I wanted to do something like this, my preferred solution was to use Redis, which is also very simple and reliable. Not sure if it is as fast.
Also, the smoothest, high fps video for opencv setup I have ever tried was with Webrtc.
However to take advantage of hardware acceleration, etc, it was a very ugly, complicated set up using JavaScript in the browser, on both ends a node server for Webrtc, a node server to receive frames on the server side out of the browser on the server end, and a redis server. Very ugly and brittle, but the video was super smooth.
I am reading up as much as possible on zeromq and nanomsg, thanks to you, because tcp can be a pain to work with directly.
Redis is amazing, I’m a HUGE fan of Redis. You can even design entire message passing/broker libraries around Redis as well.
Hello Adrian,
Thanks for the post, very helpful.
Is it possible to send video from laptop camera to AWS EC2 for processing and show the processed video back on the same laptop.
Thanks.
Yes, you can use this exact method, actually. Just supply the IP address of your AWS server and ensure the ZMQ server is running on it.
Awesome work Adrian! My project is the same but I’m using an IP camera? May I use the same code of Raspberry Pi and if yes what changes must be done
How do you typically access your IP camera? Do you/have you used OpenCV to access it before?
I failed at the sym-link option
cd ~/.virtualenvs/lib/python3.5/site-packages
No such file or directory
My mistake,
I needed to add the “cv” (my name for my virtual environment) in the path
/home/pi/.virtualenvs/cv/lib/python3.5/site-packages
…still learning…
Congrats on resolving the issue, Kurt!
Please explain in detail as I am stuck with the same error.
this is really cool. thanks for the article. i am also wondering what is the best way to push from zmq to web browsers? any recommendations? thank you!
I’m actually covering that exact project in my Raspberry Pi for Computer Vision book. You can find the Kickstarter page for the book here.
hi Adrian. Thx for the tutorial. I’m planning to do something similar instead stream iPhone video at very high fps to my laptop so I can faster prototype algorithms. Do you have any experience or comparison with LCM (https://github.com/lcm-proj/lcm). It seems it targets realtime scenarios and using UDP multicast so theoretically should have lower latency.
Sorry, I do not have any experience with that library so I can’t really comment there.
Thanks for your great work!
I tried this work and the result was great
But implemented only in one device in the sense implemented by the client and the server in one computer. The process did not work with me in the raspberry pi problem. I experimented even with ubuntu computer and windows computer did not fix.
But if you do them together they work both in ubuntu or in the windows
I have a problem with communication.
Hi Adrian,
I’ve managed to get the RPi client and Ubuntu 18.04 server communicating properly, but my Pi camera is mounted upside down and I’m not sure if I need to import the PiCamera module and use the –vflip switch. Would this be a way to flip the video image? If so, where in the script should I place the code? Thanks.
You could control it directly via the “vflip” switch OR you could use the “cv2.flip” function. Either will work.
where do i find the imutils.video stuff?
You install it via pip:
$ pip install imutilsHi Adrian,
Thanks for your tutorial! I made it success in my Env.(Raspberry Pi 3 B+ and up-board square), I would like to ask you several questions.
1. what is the bottle neck of the low frame rate?
2. if I use an AI accelerator(like NCS or Edge TPU) for inference, can I get more frame rate?
3. Is Jetson nano a good platform for the host?
Thanks and regards,
Rean
The bottleneck here is running the object detector on the CPU. If your network is poor then latency could become the bottleneck. If you’re streaming to a central device you wouldn’t use a NCS, Edge TPU, or Nano. You would just perform inference on the device itself.
Great tutorial
but I have a question
what is the video format (codec) used by ?
Hey Adrian, as always, a great tutorial!
I am working on a project in which I need to live stream feed from a rtsp ip cam to the browser. I am also performing some recognition and detection on the feed through opencv on the backend. The problem is I have to put this code on a VM and show these feeds to client on their browser. I tried different players but none is able to recognize the format of the stream. The only option I found was to convert the stream using ffmpeg and then display it on the browser. But doing this process simultaneously is something I am stuck at. Do you have any suggestions?
Hey Sam — I’m covering how to stream from a webcam to the browser inside my book, Raspberry Pi for Computer Vision.
Your tutorial is awesome. Thanks Adrian, you are a prince. This one helped me tremendously in getting my automated aircraft annotater up and working in only a week.. probably would still be figuring out step 1 inf it wasn’t for your help.
https://github.com/froohoo/skyWatcher
Holy cow. This is one of the neatest projects I have seen. Congratulations on a successful project!
Hi Adrian,
Having some trouble running the code. I first run the server.py as instructed followed by running the client script on the Pi. I am using a Windows 10 machine as the server. The problem is that after running the client script, nothing is being returned on the server side.
The client.py script appears to hang at the command: “sender.send_image(rpiName, frame)”
I have checked that the server IP is entered correctly. Any suggestions on how to debug?
Thanks in advance!
Hi. Just wondering if there is a way to add basic error handling to these scripts. I’ve noticed that if the client is started before the server it will simply block on send_image() until the server is up and will work fine.
However if the server is killed (for whatever reason) and then restarted it no longer works without killing all the clients and restarting them as well.
I guess I can have a script running on the server – monitoring the status of the script from this blog post and if something goes wrong ssh into each client and kill it and restart.
It seems to be an issue in imageHub.rev_image() that stops the server working after a restart.?!? Any ideas on how to make this a robust solution?
Hey Jason — we’re actually including a few updates to the ImageZMQ library inside Raspberry Pi for Computer Vision to make it a bit more robust.
Hi ! Awesome tutorial!
What is the difference between this and using the Motion project ? does that still use message passing?
Which motion project are you referring to?
Hi Adrian,
Is it possible to use Raspberry Pi Zero W as the CAM?
or, what is the minimum demand for a video streamer in Pi series?
Thanks and Regards,
Rean
You can technically use a Pi Zero W but streaming would be a bit slower. I recommend a 3B+.
understood, thanks!
Thanks for that great work, could I use that code inside commercial product ?
Yes, but make sure you read the ImageZMQ license first.
I have been having this issue no matter what I try to do with sending video from one machine to another machine, and I have not seen this asked in the comments section. Whenever I run the client and server on the same machine I have no problems the image frame opens up and displays the video. However every time I try to host the server on one machine and run the client on another machine the image frame never shows up, and in this case the server doesn’t even see the client connecting.
I am entering the correct IP of the server in the arguments, what am I doing wrong?
Thank you in advance.
Are both machines on the same network? And have you launched ImageZMQ on the machines?
Hello, Adrian. It’s OK? I really liked this project. One question, if I wanted to use an IP camera, how would I configure this client code? Another question, is it possible to use the people counting system in this example too? Thank you.
I don’t have any examples of IP cameras yet. But you’ll be able to use people counting inside my book, Raspberry Pi for Computer Vision.
How do I do not let zero the counter of people and objects?
I personally do not like the low framerate associated with ZeroMQ.
In my own work, I’ve found that ROS+raspicam_node has been a very frictionless system that offers pretty low latency, and will support native 30fps framerates. The only headache I guess will be using ROS if you’re not already familiar.
ROS can be quite a pain to work with. Do you happen to know how ROS is working with the RPi camera? Is it a native integration or something involving a service like gstreamer?
Great proyect my friend, this help me out of my issue with recognizing faces faster with the raspberry pi 3+b thank you! My best wishes!
Thanks Jahir, I’m glad it helped!
Hi Adrian, thanks for the tutorial, I have a problem sending the frames from the pi to my server, when i first run my server displays in the terminal “starting video stream…”, then i run the client.py on the pi but my server never says “receiving video from…”. it works on windows, but not in Ubuntu 18.04. D:
Try checking the IP address of the server as it sounds like you’re using two different systems.
Hello, thank you for a great solution. I wish to askm how can one stream these videos to an online web page, shared hosting server or something where someone can login to see live video feeds of some remote place by opening a web page displaying the video feeds.
Any ideas please?
I am covering that exact equation inside Raspberry Pi for Computer Vision — I suggest you start there.
hello adrian thank you for your tutorial.i have a face recognition project and i want to use 15 cameras for this purpose.my system info is: gpu 1080ti, cpu core i7, 16gb memory.
My goal is to have at least 10 cameras for real time face recognition, and the speed of recognize in this project is very important.
Is this sufficient for this project? What is the minimum system I need?
Is zmq and 10 raspberry pi’s able to give me a good performance?
Im using ip cameras.
Hey Jenny — I’d be happy to help and provide suggestions for this project, but I would definitely suggest you pickup a copy of Raspberry Pi for Computer Vision first. That book will teach you how to build such a project and it will enable me to provide you with targeted help on your project. Thank you!
i just can to add 4cameras and cpu usage is 80%,Do I need to add a cpu number or is my programming problem ? Please help me.
Hi Adrian,
I’ve found your post and what you’ve done is amazing! I’m researching in the field of computer Vision and looking for some way to use OpenPose for tracking body movements. I have facing some problems regarding processing the images using this library, as it takes 20 sec to process each of the frames. I would like to set up a GPU in the cloud and then be able to run it real time. Is that possible using imagezmq?
Cheers Fabio
Yes, basically you would use ImageZMQ to stream frames from your webcam to a system running a GPU. The GPU machine will then run the OpenPose models.
hey, Adrian is it possible to sent result back from the server to a client (suppose I want to send a number of persons available in the room back towards the client) how can I do that?
i have a question. i am trying to integrate this within the Kivy toolset, i know how to get a screen to display an openCV video stream however when trying to work with this library i get a bunch of errors. is there any advice you might be able to offer with this?
Sorry, I don’t do much work with Kivy. Good luck resolving the issue!
Dear Adrian,
Thanks a lot for making this tutorial. I am able to use this program without any problem. I have a few questions to ask.
i) How can I set up the server and client if they are not connected to the same wifi network? For example, my client is RPi3 and server is AWS EC2. Which IP of my AWS instance will go into –server-ip.
ii) Do I need to open port 5555?
Thanks.
1. You’ll want to check your EC2 dashboard to find the IP address of your EC2 instance.
2. Make sure the port is available for use. Check your security configurations and make sure port 5555 is accessible.
Hi adrian , i have done both as well. still couldn’t get data streams over the networks. what wrong have I done ?.
Dear Adrian,
I have another question.
Suppose for some reason, I stop my server in between but my client is running. When I restart the server, it cannot receive frames from the client. Do you have any idea about that? If I restart the server, I also have to restart the client.
How to deal with this problem?
Thank you in advance.
I solved this problem as mentioned here:
https://github.com/jeffbass/imagezmq/issues/18
If anyone is looking for a way to send images to a web page / server rather than to a python client (as I was for an application I was working on), then I highly recommend checking out the Starlette python web framework (https://www.starlette.io).
This framework is asynchronous, which makes it very simple to create a frame generator (such as receiving jpegs from imagezmq’s ImageHub) and pass it to Starlette’s StreamingResponse module to feed the ‘src’ tag of an html img with the frames from any imagezmq client.
Lots of possibilities!
Thanks for sharing!
Hello Adrian,
Interesting post, just what I needed, Thanks much. I was using regular VideoCapture and when number of cameras increased, I could feel the pain. They are IP cameras and I use rtsp protocol to get the feed. Let me try using ImageMQ and see how it goes, will share what I find soon.
Hi,
Thanks for the detailed post!
Two questions, if I may ask.
1. Did you test the latency for your set-up?
2. I want to do a similar project but I want the server to be outside the “home network”, so that anyone in the world with an internet connection can see the streaming. What would I need to change?
Thanks!
Hi Adrian,
Thanks for such a good post.
I have question about comparison of gRPC and ZMQ, and which one you prefer gRPC or ZMQ?
Hi Adrian,
thanks for this great tutorial.
Any tips on sharing data over a shared-internet (USB) connection? I’ve posted this question on https://raspberrypi.stackexchange.com/questions/103958/pi-zero-connected-to-pc-via-usb-shared-internet-pc-pi-communication-via-ima
best, Andy.
Sorry, I haven’t tried using a shared USB connection. As long as you know the IP address of both machines it should still work without an issue. Just make sure the shared connection has its own IP address on the network.
Hi,
Thanks for your great work!
I tried this code using the (logitech c920) external webcam and it worked fine! Then unfortunately it was decided to work with the laptop built-in camera, ignoring Logitech.
Do you have any idea how to force her to work with Logitech only?
Best, Hind
Inspired by this article, I implemented similar functionality with a few changes:
* MQTT – for broad IOT compatibility
* Browser viewing using streamlit
The repo is on https://github.com/robmarkcole/mqtt-camera-streamer
Thanks for sharing, Robin!
in this source code ,,,compulsory required raspberry pi are not ,,,please let me know …..
No, you could use whatever OS you like provided you have ImageZMQ properly installed.
Hi Adrian,
This is a great resource.
1 question, Instead of Raspberry Pi as client and Mac as server, can this be run on a Windows client (laptop) and Windows server (Virtual Machine) ?
Yes, provided you have ImageZMQ installed you can use whatever OS you want.
hi guys. i’m having a little problem when i run the client.py script.
It says: AttributeError : module ‘imagezmq.imagezmq’ has no attribute ‘ImageSender’.
can someone help me resolve this please?!
Hey Meg, refer to the comments to this post as that’s already been addressed 🙂
This is great, and would be even better if we could use regular webpages instead of raspberries, as that would mean being able to send data from any webpage or even cell phones to be analised on real time and obtain via zmq back a bounding box.
Hi Adrian,
Its such an awesome work. Just keen to know how can I show the received montage on a webpage in the server.
Follow this tutorial.
Hi Adrian,
Have you tested IP cameras with OpenCV that have actually worked? If so, could you provide the names and models of those cameras?
Thanks,
I don’t use IP cameras that often. I prefer using ImageZMQ as it gives you much more control.
I enjoyed this great presentation of how Jess uses his learnings and skills in implementing a monitoring solution for his farm. https://www.pyimageconf.com/static/talks/jeff_bass.pdf
I recommend everyone to have a look specially about his learnings and how he use various enclosures, etc in his work.
Hi Adrian,
Could we use ImageZMQ programming for jetson nvidia device? or it would be better we stick to the only ZMQ ?
Can you be a bit more specific? What are you intending to do with ImageZMQ? Is the Jetson running ImageZMQ? Or some other device?
Hi Adrian,
Fantastic as usual. I seem to be confusing myself trying a slightly different implementation. I’m very used to regular ZMQ and I use it for all my scripts to communicate between themselves and even scripts on other R Pis on the network. But whereas this tutorial is a “many to one” example, I need much of my image usage to be of a “one-to-many” form; I have many scripts that all do different things (face recognition, human activity recognition, object detection, etc) and I need them to be able to connect to the frame stream without “owning” it. In ZMQ, my “serving” side binds to a port, and my receiving “clients” connect to that port. If a particular client is slow, it just misses a few sends, no impact on the overall system. This doesn’t seem to work quite the same way with ImageZMQ non-blocking PUB/SUB setup. what am I missing?
PUB Side (sender of image stream):
def __init__(self, connect_to=’tcp://192.168.1.201:5557′, REQ_REP = False):
SUB Side (multiple receivers of image stream on same Pi):
def __init__(self, open_port=’tcp://192.168.1.201:5557′, REQ_REP = True):
Can you point me to a simple code example that does what I’m trying to do…
Thanks,
Dave
Never mind, I think I’ve got it…. sorry. Long day.
Congrats on resolving the issue, Dave! Also, you should take a look at the updated ImageZMQ. It’s now pip installable and cleans up the package a bit.
Hi Adiran,
Thanks for your post.
I would like to use 2 cameras for capture the persons face, age, emotion, name and the daily report status of persons.
Is this above example helpful for me and what are the requirements to implement?
Kindly provide us the steps / any references to implement the solution.
Your valuable suggestions are very helpful to me.
Thanks.
I would suggest you read Deep Learning for Computer Vision with Python. That book will teach you how to perform age and emotion recognition. Face recognition is covered inside the PyImageSearch Gurus course as well as these tutorials.
Hi Adrian, great tutorial, what if I don’t want to count the person but just display the bounding box with a label, what part of the code should I modify?
Hey Adrian, is there a repo I can fork? Wanted to add some simple parameterization for client.py to specify a specific camera at runtime.
Yes, you can fork Jeff’s official repo.
How can we retrieve the real time object count from the server to the client(Raspberry pi) to perform say some automative stuffs.
Thanks
Hi Adrian,
Does ImageZMQ support streaming video and audio?
Just video/images.
Thanks for your response
Hi Adrian,
I’m looking for a video streaming solution that supports this scenario:
Streaming from a raspberry pi or an Ubuntu machine such as a robot to an Ubuntu server to do deep learning processing, also there is a mobile application or web application that needs to receive the stream from the server or from the streaming client (i.e. raspberry pi, or Ubuntu machine).
Does ImageZMQ help me to accomplish this or do you suggest any other Open Source solution?
Thank you alot
Yes, this seems like a good application of ImageZMQ. Give it a try.
Thank you a lot, I will give it a try