Over the next few weeks, I’ll be doing a series of blog posts on how to improve your frames per second (FPS) from your webcam using Python, OpenCV, and threading.
Using threading to handle I/O-heavy tasks (such as reading frames from a camera sensor) is a programming model that has existed for decades.
For example, if we were to build a web crawler to spider a series of webpages (a task that is, by definition, I/O bound), our main program would spawn multiple threads to handle downloading the set of pages in parallel instead of relying on only a single thread (our “main thread”) to download the pages in sequential order. Doing this allows us to spider the webpages substantially faster.
The same notion applies to computer vision and reading frames from a camera — we can improve our FPS simply by creating a new thread that does nothing but poll the camera for new frames while our main thread handles processing the current frame.
This is a simple concept, but it’s one that’s rarely seen in OpenCV examples since it does add a few extra lines of code (or sometimes a lot of lines, depending on your threading library) to the project. Multithreading can also make your program harder to debug, but once you get it right, you can dramatically improve your FPS.
We’ll start off this series of posts by writing a threaded Python class to access your webcam or USB camera using OpenCV.
Next week we’ll use threads to improve the FPS of your Raspberry Pi and the picamera module.
Finally, we’ll conclude this series of posts by creating a class that unifies both the threaded webcam/USB camera code and the threaded picamera code into a single class, making all webcam/video processing examples on PyImageSearch not only run faster, but run on either your laptop/desktop or the Raspberry Pi without changing a single line of code!
Use threading to obtain higher FPS
The “secret” to obtaining higher FPS when processing video streams with OpenCV is to move the I/O (i.e., the reading of frames from the camera sensor) to a separate thread.
You see, accessing your webcam/USB camera using the cv2.VideoCapture function and the .read() method is a blocking operation. The main thread of our Python script is completely blocked (i.e., “stalled”) until the frame is read from the camera device and returned to our script.
I/O tasks, as opposed to CPU bound operations, tend to be quite slow. While computer vision and video processing applications are certainly quite CPU heavy (especially if they are intended to run in real-time), it turns out that camera I/O can be a huge bottleneck as well.
As we’ll see later in this post, just by adjusting the the camera I/O process, we can increase our FPS by as much as 379%!
Of course, this isn’t a true increase of FPS as it is a dramatic reduction in latency (i.e., a frame is always available for processing; we don’t need to poll the camera device and wait for the I/O to complete). Throughout the rest of this post, I will refer to our metrics as an “FPS increase” for brevity, but also keep in mind that it’s a combination of both a decrease in latency and an increase in FPS.
In order to accomplish this FPS increase/latency decrease, our goal is to move the reading of frames from a webcam or USB device to an entirely different thread, totally separate from our main Python script.
This will allow frames to be read continuously from the I/O thread, all while our root thread processes the current frame. Once the root thread has finished processing its frame, it simply needs to grab the current frame from the I/O thread. This is accomplished without having to wait for blocking I/O operations.
The first step in implementing our threaded video stream functionality is to define a FPS class that we can use to measure our frames per second. This class will help us obtain quantitative evidence that threading does indeed increase FPS.
We’ll then define a WebcamVideoStream class that will access our webcam or USB camera in a threaded fashion.
Finally, we’ll define our driver script, fps_demo.py, that will compare single threaded FPS to multi-threaded FPS.
Note: Thanks to Ross Milligan and his blog who inspired me to do this blog post.
Increasing webcam FPS with Python and OpenCV
I’ve actually already implemented webcam/USB camera and picamera threading inside the imutils library. However, I think a discussion of the implementation can greatly improve our knowledge of how and why threading increases FPS.
To start, if you don’t already have imutils installed, you can install it using pip :
$ pip install imutils
Otherwise, you can upgrade to the latest version via:
$ pip install --upgrade imutils
As I mentioned above, the first step is to define a FPS class that we can use to approximate the frames per second of a given camera + computer vision processing pipeline:
# import the necessary packages import datetime class FPS: def __init__(self): # store the start time, end time, and total number of frames # that were examined between the start and end intervals self._start = None self._end = None self._numFrames = 0 def start(self): # start the timer self._start = datetime.datetime.now() return self def stop(self): # stop the timer self._end = datetime.datetime.now() def update(self): # increment the total number of frames examined during the # start and end intervals self._numFrames += 1 def elapsed(self): # return the total number of seconds between the start and # end interval return (self._end - self._start).total_seconds() def fps(self): # compute the (approximate) frames per second return self._numFrames / self.elapsed()
On Line 5-10 we define the constructor to our FPS class. We don’t require any arguments, but we do initialize three important variables:
_start: The starting timestamp of when we commenced measuring the frame read._end: The ending timestamp of when we stopped measuring the frame read._numFrames: The total number of frames that were read during the_startand_endinterval.
Lines 12-15 define the start method, which as the name suggests, kicks-off the timer.
Similarly, Lines 17-19 define the stop method which grabs the ending timestamp.
The update method on Lines 21-24 simply increments the number of frames that have been read during the starting and ending interval.
We can grab the total number of seconds that have elapsed between the starting and ending interval on Lines 26-29 by using the elapsed method.
And finally, we can approximate the FPS of our camera + computer vision pipeline by using the fps method on Lines 31-33. By taking the total number of frames read during the interval and dividing by the number of elapsed seconds, we can obtain our estimated FPS.
Now that we have our FPS class defined (so we can empirically compare results), let’s define the WebcamVideoStream class which encompasses the actual threaded camera read:
# import the necessary packages from threading import Thread import cv2 class WebcamVideoStream: def __init__(self, src=0): # initialize the video camera stream and read the first frame # from the stream self.stream = cv2.VideoCapture(src) (self.grabbed, self.frame) = self.stream.read() # initialize the variable used to indicate if the thread should # be stopped self.stopped = False
We define the constructor to our WebcamVideoStream class on Line 6, passing in an (optional) argument: the src of the stream.
If the src is an integer, then it is presumed to be the index of the webcam/USB camera on your system. For example, a value of src=0 indicates the first camera and a value of src=1 indicates the second camera hooked up to your system (provided you have a second one, of course).
If src is a string, then it assumed to be the path to a video file (such as .mp4 or .avi) residing on disk.
Line 9 takes our src value and makes a call to cv2.VideoCapture which returns a pointer to the camera/video file.
Now that we have our stream pointer, we can call the .read() method to poll the stream and grab the next available frame (Line 10). This is done strictly for initialization purposes so that we have an initial frame stored in the class.
We’ll also initialize stopped , a boolean indicating whether the threaded frame reading should be stopped or not.
Now, let’s move on to actually utilizing threads to read frames from our video stream using OpenCV:
def start(self): # start the thread to read frames from the video stream Thread(target=self.update, args=()).start() return self def update(self): # keep looping infinitely until the thread is stopped while True: # if the thread indicator variable is set, stop the thread if self.stopped: return # otherwise, read the next frame from the stream (self.grabbed, self.frame) = self.stream.read() def read(self): # return the frame most recently read return self.frame def stop(self): # indicate that the thread should be stopped self.stopped = True
Lines 16-19 define our start method, which as the name suggests, starts the thread to read frames from our video stream. We accomplish this by constructing a Thread object using the update method as the callable object invoked by the run() method of the thread.
Once our driver script calls the start method of the WebcamVideoStream class, the update method (Lines 21-29) will be called.
As you can see from the code above, we start an infinite loop on Line 23 that continuously reads the next available frame from the video stream via the .read() method (Line 29). If the stopped indicator variable is ever set, we break from the infinite loop (Lines 25 and 26).
Again, keep in mind that once the start method has been called, the update method is placed in a separate thread from our main Python script — this separate thread is how we obtain our increased FPS performance.
In order to access the most recently polled frame from the stream , we’ll use the read method on Lines 31-33.
Finally, the stop method (Lines 35-37) simply sets the stopped indicator variable and signifies that the thread should be terminated.
Now that we have defined both our FPS and WebcamVideoStream classes, we can put all the pieces together inside fps_demo.py :
# import the necessary packages
from __future__ import print_function
from imutils.video import WebcamVideoStream
from imutils.video import FPS
import argparse
import imutils
import cv2
# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-n", "--num-frames", type=int, default=100,
help="# of frames to loop over for FPS test")
ap.add_argument("-d", "--display", type=int, default=-1,
help="Whether or not frames should be displayed")
args = vars(ap.parse_args())
We start off by importing our necessary packages on Lines 2-7. Notice how we are importing the FPS and WebcamVideoStream classes from the imutils library. If you do not have imutils installed or you need to upgrade to the latest version, please see the note at the top of this section.
Lines 10-15 handle parsing our command line arguments. We’ll require two switches here: --num-frames , which is the number of frames to loop over to obtain our FPS estimate, and --display , an indicator variable used to specify if we should use the cv2.imshow function to display the frames to our monitor or not.
The --display argument is actually really important when approximating the FPS of your video processing pipeline. Just like reading frames from a video stream is a form of I/O, so is displaying the frame to your monitor! We’ll discuss this in more detail inside the Threading results section of this post.
Let’s move on to the next code block which does no threading and uses blocking I/O when reading frames from the camera stream. This block of code will help us obtain a baseline for our FPS:
# grab a pointer to the video stream and initialize the FPS counter
print("[INFO] sampling frames from webcam...")
stream = cv2.VideoCapture(0)
fps = FPS().start()
# loop over some frames
while fps._numFrames < args["num_frames"]:
# grab the frame from the stream and resize it to have a maximum
# width of 400 pixels
(grabbed, frame) = stream.read()
frame = imutils.resize(frame, width=400)
# check to see if the frame should be displayed to our screen
if args["display"] > 0:
cv2.imshow("Frame", frame)
key = cv2.waitKey(1) & 0xFF
# update the FPS counter
fps.update()
# stop the timer and display FPS information
fps.stop()
print("[INFO] elasped time: {:.2f}".format(fps.elapsed()))
print("[INFO] approx. FPS: {:.2f}".format(fps.fps()))
# do a bit of cleanup
stream.release()
cv2.destroyAllWindows()
Lines 19 and 20 grab a pointer to our video stream and then start the FPS counter.
We then loop over the number of desired frames on Line 23, read the frame from camera (Line 26), update our FPS counter (Line 35), and optionally display the frame to our monitor (Lines 30-32).
After we have read --num-frames from the stream, we stop the FPS counter and display the elapsed time along with approximate FPS on Lines 38-40.
Now, let’s look at our threaded code to read frames from our video stream:
# created a *threaded* video stream, allow the camera sensor to warmup,
# and start the FPS counter
print("[INFO] sampling THREADED frames from webcam...")
vs = WebcamVideoStream(src=0).start()
fps = FPS().start()
# loop over some frames...this time using the threaded stream
while fps._numFrames < args["num_frames"]:
# grab the frame from the threaded video stream and resize it
# to have a maximum width of 400 pixels
frame = vs.read()
frame = imutils.resize(frame, width=400)
# check to see if the frame should be displayed to our screen
if args["display"] > 0:
cv2.imshow("Frame", frame)
key = cv2.waitKey(1) & 0xFF
# update the FPS counter
fps.update()
# stop the timer and display FPS information
fps.stop()
print("[INFO] elasped time: {:.2f}".format(fps.elapsed()))
print("[INFO] approx. FPS: {:.2f}".format(fps.fps()))
# do a bit of cleanup
cv2.destroyAllWindows()
vs.stop()
Overall, this code looks near identical to the code block above, only this time we are leveraging the WebcamVideoStream class.
We start the threaded stream on Line 49, loop over the desired number of frames on Lines 53-65 (again, keeping track of the total number of frames read), and then display our output on Lines 69 and 70.
Threading results
To see the affects of webcam I/O threading in action, just execute the following command:
$ python fps_demo.py
As we can see, by using no threading and sequentially reading frames from our video stream in the main thread of our Python script, we are able to obtain a respectable 29.97 FPS.
However, once we switch over to using threaded camera I/O, we reach 143.71 FPS — an increase of over 379%!
This is clearly a huge decrease in our latency and a dramatic increase in our FPS, obtained simply by using threading.
However, as we’re about to find out, using the cv2.imshow can substantially decrease our FPS. This behavior makes sense if you think about it — the cv2.show function is just another form of I/O, only this time instead of reading a frame from a video stream, we’re instead sending the frame to output on our display.
Note: We’re also using the cv2.waitKey(1) function here which does add a 1ms delay to our main loop. That said, this function is necessary for keyboard interaction and to display the frame to our screen (especially once we get to the Raspberry Pi threading lessons).
To demonstrate how the cv2.imshow I/O can decrease FPS, just issue this command:
$ python fps_demo.py --display 1
Using no threading, we reach 28.90 FPS. And with threading we hit 39.93 FPS. This is still a 38% increase in FPS, but nowhere near the 379% increase from our previous example.
Overall, I recommend using the cv2.imshow function to help debug your program — but if your final production code doesn’t need it, there is no reason to include it since you’ll be hurting your FPS.
A great example of such a program would be developing a home surveillance motion detector that sends you a txt message containing a photo of the person who just walked in the front door of your home. Realistically, you do not need the cv2.imshow function for this. By removing it, you can increase the performance of your motion detector and allow it to process more frames faster.
What's next? I recommend PyImageSearch University.
30+ total classes • 39h 44m video • Last updated: 12/2021
★★★★★ 4.84 (128 Ratings) • 3,000+ Students Enrolled
I strongly believe that if you had the right teacher you could master computer vision and deep learning.
Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?
That’s not the case.
All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.
If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.
Inside PyImageSearch University you'll find:
- ✓ 30+ courses on essential computer vision, deep learning, and OpenCV topics
- ✓ 30+ Certificates of Completion
- ✓ 39h 44m on-demand video
- ✓ Brand new courses released every month, ensuring you can keep up with state-of-the-art techniques
- ✓ Pre-configured Jupyter Notebooks in Google Colab
- ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
- ✓ Access to centralized code repos for all 500+ tutorials on PyImageSearch
- ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
- ✓ Access on mobile, laptop, desktop, etc.
Summary
In this blog post we learned how threading can be used to increase your webcam and USB camera FPS using Python and OpenCV.
As the examples in this post demonstrated, we were able to obtain a 379% increase in FPS simply by using threading. While this isn’t necessarily a fair comparison (since we could be processing the same frame multiple times), it does demonstrate the importance of reducing latency and always having a frame ready for processing.
In nearly all situations, using threaded access to your webcam can substantially improve your video processing pipeline.
Next week we’ll learn how to increase the FPS of our Raspberry Pi using the picamera module.
Be sure to enter your email address in the form below to be notified when the next post goes live!
Download the Source Code and FREE 17-page Resource Guide
Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!
Thank you ! Great tuto !
I’m wondering if (in a production app) we should use a lock or something to synchronize access to the frame which is a shared resource, right ?
If it’s a shared resource, then yes, you should absolutely use a lock on the image data, otherwise you can run into a synchronization issue.
Same here – I assumed you should have a thread acquire and release so it isn’t reading the image wihile it is being written? Apparently assinging a value to be a numpy array is atomic – or it doesn’t really matter if it was the last frame, not the very latest?
Looks like if you have ANY processing you need to have it out of that fetching image thread, and it runs pretty fast.
Hi Adrian,
looks like the increase in fps is a fake; you get a frame immediately when required, but looks like it is still the former frame when the program body is executed faster than the physical frame rate of the camera. What do you think?
Jürgen
Very true — and at that point it depends on your physical camera. If your loop is faster than the physical frame rate of the camera, then the increase in FPS is not as realistic. This is further evidenced when we use the
cv2.imshowfunction to simulate a more “realistic” scenario. In either case though, threading should be used since it can increase FPS.This is 2 years later and I am using opencv 3.2. Seems to me that it is already using threading, at lest all cores are in use.
I agree with Jürgen, that the higher framerate is just fake.
In the 1st (“non-threaded”) case, the code assumedly waits for the next frame of the camera to arrive, then calculates, framerate is calculated from the resulting speed of the loop. So the framerate is limited by a) camera framerate b) calculation time.
If calculation time is much smaller than camera-framerate, you are only limited by a)
In the 2nd case, you decouple the two. that means your framerate only depends on b), but you still only get another new frame according to a) and so you keep re-calculating uselessly on the same frame.
I don’t see an advantage of threading the way it is done here, at least as long as your calculation time is shorter than the time between frames. (But I think that it *is* a good way to measure how long your image processing takes). You even lose some time, because you are (obviously) busy needlessly re-calculating at the time that the next frame comes in.
What happens if your calculation time is larger than the time between frames from the camera is not so clear. In your threaded case, you keep fetching images and throw away any previous ones… But I guess in the non-threaded case, the kernel does just the same? Else you’d fill kernel buffer space with frames if you cannot process fast enough (or wait long enough between fetches with waitKey).
You can, btw., set and query the framerate in opencv 3.2 with e.g. cap.set(cv2.CAP_PROP_FPS,150)
After setting, reading the value with cap.get only returns what you set, you have to measure fps in the read-loop, similar to what you did.
You probably also have to set smaller cv2.CAP_PROP_FRAME_WIDTH and cv2.CAP_PROP_FRAME_HEIGHT for higher fps rates or so (at least that is the case with the camera here).
Cheers,
I.
Hi Iridos — thanks for the comment. I’ve tried to make this point clear in the blog post and in the comments section, but what you’re actually measuring here is the frame processing throughput rate of your pipeline. There are physical limitations to the number of frames that can be read per second from a given camera sensor, so it’s often helpful to instead look at how fast our process pipeline (and see if we can speed it up). By using threading, we don’t have to worry about the video I/O latency. In the purposely simple examples in the blog post, the
whileloop runs faster than the next frame can be read. Keep in mind that for any reasonable video processing pipeline this wouldn’t be the case.Hi,
My query is. If I set it, will it be set for the session until cv object is obsolete / video is cleared. Will the “set” scope be limited to certain conditions? Like, the script that includes the cv2, or is there a way to globally set it to say 30 fps.
Right now,the camera runs 20 fps and I need it to work on 30. Can I permanently set it for always ?
Thanks,
ReemaRaven
A few years later 🙂
There are three reasons to use threading (that I use anyway): The first is software synchronisation and processing of multiple cameras – you need 2N threads for N cameras; a capture thread and a processing thread for each.
The second use case is syncing cameras that have multiple frame rates (e.g. one at 50fps one at 10fps) – you can wait for all cameras to signal that they’ve acquired and then pull the latest image from each camera.
Finally (and most importantly) if you have any long processing you should always service the camera whilst it’s happening, otherwise your video stream will always lag behind (by up to 32 frames in V4L2 as that’s the maximum buffer size). This answers your question about what happens if you don’t service the camera. That may be OK.
Really this code needs is a flag to identify when a new frame has been acquired; the flag then gets cleared when you call .latest() or whatever. You can also use an event loop like this to artificially reduce the frame rate of your camera by only signalling “new frame” at your desired frame rate. This is how I implemented it anyway. You can also do stuff like implement things like timer callbacks in the loop.
So it’s not about the camera processing being threaded (indeed on my desktop I get all 8 cores used) it’s that you want to dequeue images from the camera immediately and not leave them to stack up. That said, you can also offload processing onto the threads since you’ve got ~1/fps seconds to do stuff before the next frame arrives and it won’t affect your frame rate. Do note that on some platforms, image capture is single threaded.
Hi Adrian
This is great. I am myself experimenting with a multithreaded app that runs opencv and other libraries and I’m already using your video stream class.
Special note for OSX users: I’ve run into a limitation in opencv on OSX, the command cv2VideoCapture(0) can only be issued in the main thread, so take this into account when designing your app. See http://stackoverflow.com/questions/20445762/unable-to-start-camera-capture-in-python-opencv-cv2-using-a-non-main-thread for more info
Thanks for sharing David — that’s a great tip regarding
cv2.VideoCapturecan only be executed from the main thread.Awesome, Adrian!! Can’t wait to read the tutorial about fps increase for Raspberry Pi using the picamera module!
Great tutorial (as allways)! and good timing too…
I’m just trying to make the image gathering threaded for my raspberry pi project to improve the framerate. Without threading but with the use of a generator type of code for the image handling I improved the framerate by around 2 times but hopefully threading will do more. Another thing that is interesting is how to optimize the framerate vs the opencv computational time to reach a good balance. Jurgen mentioned that several frames could be similar and then it is no need to make calculation on that second frame (at least not in my case). On a raspberry pi 2 there is 4 cores and distributing the collection of frame data and calculations in a good way would improve the performance. Do you have any thoughts or advice about that?
If you’re using the Pi 2, then distributing the frame gathering to a separate thread will definitely improve performance. In fact, that’s exactly what next week’s blog post is going to be about 😉
Hi Adrian
In the single threaded case you’re limited to 30fps because that is the framerate of the camera in this case and you’re not really achieving 143fps in the multi-threaded case since you’re simply processing the same frame multiple times. The 143fps is really a measure of the amount of time the imutils.resize() takes, i.e. ~6.9ms. So the comparison between 30fps and 143fps isn’t really a fair and accurate comparison.
I recently had a project where we ended up using the same approach, i.e. grabbing the webcam frames on a secondary thread and doing the OpenCV processing on the main python thread. However this wasn’t in order to increase the fps processing rate, rather it was to minimize the latency of our image processing. We were measuring aircraft control positions by visually tracking targets on the aircraft controls to record their position during flight and needed to synchronize the recording of the control positions with another instrument we use to record the aircraft’s attitude etc.
So we needed as little latency as possible in order to keep the control positions synchronized with the other aircraft data we were recording. We didn’t need a high sampling rate, i.e. we were happy with 10Hz as long as the latency was as small as possible.
Our camera output at 1080@30Hz and our image processing (mainly Hough circles) took longer than the frame period of ~33ms and if we read the camera frames on the main thread the OS would buffer roughly 5 frames if we didn’t read them fast enough. So going with the multithreaded approach we could always retrieve the latest frame as soon as our image processing was complete, so at a lower rate than the camera rate but with minimizing latency.
Cheers
Indeed, you’re quite right. The 143 FPS isn’t a fair comparison. I was simply trying to drive home the point (as you suggested) of the latency. Furthermore, simply looping over a set of frames (without doing any processing besides resizing) isn’t exactly fair of what a real-world video processing would look like either.
Hi Adrian
But I think that overall you’ve made it more confusing mixing up fps and latency. If your main point that you were trying to drive home is the win in terms of latency then that should be in the title, in the examples your provide etc.
Sort of like mixing up a disk’s transfer rate and latency.
Cheers
I’ll be sure to make the point more clear in the next post on Raspberry Pi FPS and latency.
Hi Sean,
I have the same problem as yours. I need in my project the minimum latency as possible. Due to the opencv internal buffer I have to use threads. I am working with several 8Mp cameras, each of them with its own thread. But using threads then I face the “select timeout” problem. Did you have the same problem? By the way, did you use locks to access the variable “frame”?
Nice tutorial – thanks for the mention!
I was experiencing a subtly different problem with webcam frames in my apps, which led me to use threading. I was not so concerned with the speed of reading frames, more that the frames were getting progressively stale (after running app for a minute or so on Raspberry Pi, the frame served up was a number of frames behind the real world). Perhaps my app loop was too slow interrogating the webcam and was being served up a cached image? By using a thread I was able to interrogate the webcam constantly and keep the frame fresh.
Thanks for the tip Ross — you’re definitely the inspiration behind where this post came from.
Hi Ross
See my comment above, we saw the same issue as you on the ODROID we were using. On our system it looked like the OS/v4l/OpenCV stack was maintaining a buffer on the order of 5 frames if we didn’t retrieve frames as fast as the camera’s frame rate, which meant we ended up with an extra latency on the order of 5x33ms = 165ms.
So we ended up pulling the usb web camera images at the camera’s frame rate on a secondary thread so that we were always processing the latest web camera image even though overall our main video processing only ran at 10fps.
We initially tried to see if there was a way to prevent this buffering but weren’t able to find a way to disable it, so we ended up with the multi-threading approach.
Cheers
I installed imutils but still get this error when I run the program:
ImportError: No module named ‘imutils.video’
Running python 3.4 and opencv2
Make sure you have the latest version of
imutils:$ pip install --upgrade imutils --no-cache-dirI have the latest version installed, but I’m still getting the error. Please help if you can. All I need is something simple that can display an image on the screen from a USB webcam, and can start automatically at boot. I am running a Raspberry Pi Zero and Raspbian Jessie. The webcam is a rather cheap GE model, with YUYV 640×480 support. I have already tried multiple programs, but only luvcview gave a usable picture, and it broke itself when attempting at auto-start script. Any help at all would be useful! Thank you in advance!
I detail how to create an autostart script here. This should also take care of the error messages you’re getting since you’ll be accessing the correct Python virtual environment where ideally you have installed the imutils package.
Thank you for a great tutorial. I am working with applications like SURF, Marker detection etc. but i need to increase the FPS for the above mentioned applications. Will this approach work with OpenCV C++? If yes, how?
Yes, the same approach will work for C++. The exact code will vary, but you should do some research on creating and managing separate threads in C++.
I completed a C ++ version, but don’t know it is suitable with C++ mutex. Hope this is useful.
[mutexvideocapture.cpp](https://github.com/nimohunter/Calibration-Use-ChArucoBoard/blob/master/mutexvideocapture.cpp)
This is a classic producer/consumer problem.
Here we have a camera (the producer) that is delivering frames at a constant rate in real time
and a frame reading program (the consumer) that is not processing the frames in real time.
In this case we must have a frame queue with a frame count. If the frame count is > 0 then the consumer consumes a frame – reducing the frame count. If the frame count is zero then the consumer must wait until the frame count rises above zero before consuming a frame.
There can be any number of consumers but the producer must serialize access to the frame queue (using locks/semaphores).
I’m wondering if Adrian has fully covered this aspect in his book and tutes…
I don’t cover this in detail inside Practical Python and OpenCV, but it’s something that I do plan on covering in future blog posts.
Unless you want to process every frame that is read by the camera, I actually wouldn’t use a producer-consumer relationship for this. Otherwise, as you suggested, if your buffer is > 0, then you’ll quickly build up a backlog of frames that need to be processed, making the approach unsuitable for real-time applications.
Thank you sooo much for sharing your blog post. My use case is using Tensorflow model evaluation from a webcam stream. Each instance of model evaluation is taking about 80ms on Mobilenet based model. My question is, if i used the threaded stream reader, would i run into an issue wherein my model is evaluating older frames because it is slow while, the threaded stream reader is quickly building up backlog… If that happens the model would never be able to catch up. In that situation, i would have to figure out how to reduce the FPS to match the model’s evaluation time of 80ms.
The threaded reader isn’t building a backlog of frames, it’s grabbing the most recent frame.
I don’t see a performance increase on Windows 7. Process explorer (confirmed by cpu usage graph in Task Manager) confirms cv2 is already divided in multiple threads??
I’m not a Windows user, so I’m not entirely sure how Windows handles threading. The Python script won’t be divided across multiple process, but a new thread should be spawned. Again, I haven’t touched a Windows system in over 9+ years, so I’m probably not the right person to ask regarding this question.
Hello,
Thank you very much for sharing this information.
One drawback of this method is as mentioned in the comments that you kind of lose the timestamp / counter information when a frame was shot by the camera.
Now the funny thing is a timesteamp is provided if you use the capture_continous function and provide a string as argument.
So diving a bit deeper in the code of this function, don’t you think we could just add a timestamp / counter in the “else” condition? It might make the code that later processes these images a bit more efficient since a mechanism can be made to avoid processing the same frame twice.
Not sure if you have an opinion on this one / ever experimented with it 🙂 ?
http://picamera.readthedocs.io/en/release-1.10/_modules/picamera/camera.html#PiCamera.capture_continuous
hey is there a use for the variable “grabbed”? I can’t see it being used anywhere… I might be misunderstanding a lot though!
The
stream.readfunction returns a 2-tuple consisting of the frame itself along with a boolean indicating if the frame was successfully read or not. You can use thegrabbedboolean to determine if the frame was successfully read or not.Hello Adrian,
Thank you for the very useful blog post. You explain everything very clearly, especially to someone very new to python and image processing. Have you ever made a blog post regarding packages/module hierarchy? In some of your other blog posts you explicitly tell us how to name a file, but I’m confused about how the FPS and WebcamVideoStream classes are defined in the directory. More specifically, what names should those files have (is there a naming convention?) Where in the project are they typically located? How are they pulled “from imutils.video”? I know these are very basic questions, but I haven’t found a resource online that explains this clearly.
Thanks again for your work.
Hey Kevin — this is less of a computer vision question, but more of a Python question. I would suggest investing some time reading about Python project structure and taking a few online Python courses. I personally like the RealPython.com course. The course offered by CodeAcademy is also good.
hi!
Thanks for the great tutorial.
Is there any way to make the .read() method block until there is a new frame to return?
If not, is there any efficient way to determine if the old frame is the same as the new frame so I can ignore it?
thanks again
As far as I know, you can’t block the
.read()method until a new frame is read. However, a simple, efficient method to determine if a new frame is ready is to simply hash the frame and compare hashes. Here is an example of hashing NumPy arrays.Hi,
I followed up on the tutorial, after executing “python picamera_fps_demo.py” the results were:
[INFO] sampling frames from ‘picamera’ module…
[INFO] elapsed time: 3.57
[INFO] approx. FPS: 28.32
[INFO] sampling THREADED frames from ‘picamera’ module…
[INFO] elapsed time: 0.48
[INFO] approx. FPS: 208.95
but when I ran “python picamera_fps_demo.py –display 1” the results were:
[INFO] sampling frames from ‘picamera’ module…
[INFO] elapsed time: 8.54
[INFO] approx.FPS: 11.83
[INFO] sampling THREADED frames from ‘picamera’ module…
[INFO] elapsed time: 6.54
[INFO] approx. FPS: 15.29
So it ran slower, I’m not sure what is the real issue, but I’m using a Pi 3 fresh out of the box, using a 5V 1A power source, I’m connected the Pi 3 to the laptop using Xming and putty via LAN cable.
Looking at your results, it seems that in both cases the threaded version obtained faster FPS processing rate. 15.29 FPS is faster than 11.83 FPS and 208.95 FPS is faster than 28.32 FPS. So I’m not sure what you mean by running slower?
Thanks for the reply Adrian, when I see the video feed from the Pi camera on my laptop, there is a large delay, when I wave my hand over the camera it almost takes 2 to 3 seconds then I see my hand in the video feed, is this delay normal or is it an issue.
So you’re accessing the video feed via X11 forwarding or VNC? That’s the problem then. Your Pi is reading the frames just fine, it’s just the network overhead of sending the frames from the Pi to your VNC or X11 viewer. If you were to execute the script on your Pi with a keyboard + HDMI monitor, the frames would look much more fluid.
Hi Adrian,
I’m accessing the video feed via X11 forwarding, thanks for helping me identify what the problem really is. A lot of your tutorials has provided me with the basic foundation for my project, there is no other place I would recommend a beginner like me to start off learning image and video processing.
Great job resolving the issue Peni, I’m glad it was a simple fix. And thank you for the kind words, I’m happy I can help out 🙂
Hi Adrian, thank you for the post!
I have a question that is currently bothering me a bit, that is: when we use multi threads as this post’s approach, does that mean we are using multi cores? or are we just using single core with 2 threads? cuz I’m currently up to some real-time project and trying to do it using multi threads, what surprised me is that the number of frames i can process per second actually decreased a bit, cuz i thought if i’m using a different core for reading in the video i should at least save the reading time and be able to process a bit more frames per second right?
Thanks again for your help!
Hey Samuel — we are using a single core with 2 threads. Typically, when you work with I/O operations it’s advisable to utilize threads since most of your time is spent waiting for new frames to be read/written. If you’re looking to parallelize a computation, spread it across multiple cores and processors.
Hi.! How to use this python code with other image processing task. should we run parallel these two codes using terminal? how to do this? please help me.
Anyway your blog is the best
Hey Shirosh — I’m not sure what you mean by “use this Python code with other image processing task”. Can you please elaborate?
Hello Adrian,
I’ve noticed that cv2.imshow on OS X is muuuuch slower than its counterpart on windows.
The following benchmark runs in 15 seconds on a virtualised windows inside my mac, but it takes as long as 2 minutes to run on OS X itself!
https://db.tt/9FklKUpJ
Do you know what could be the reason and possible fix? Thanks a lot!
Best,
Cassiano
That’s quite strange, I’m not sure why that may be. I don’t use Windows, but I’ve never noticed an issue with
cv2.imshowbetween various operating systems.Hi, Adrian.
I have a question.
Where can I save ‘fps_demo.py’ ?
You can save it anywhere on your system that you would like. I would suggest using the “Downloads” section of this tutorial to download the code instead though.
hi adrian
thank you for this good project
when i run this project
after increse fps show a core dump segmentation fault error . can you help me ??..
If you’re getting a segmentation fault then the threading is likely causing an issue. My best guess is that the stream object is being destroyed while it’s in the process of reading the next frame. What OS are you using?
Hi,
I am trying to use Ps3 EYE Camera on ubuntu for my OpenCv project. This camera support 640×480 up to 60fps and 320×240 resolution up to 187fps. I am sure you know what I mention about. I can set each one of this values on windows with CodeLaboratories driver. But on ubuntu, I use ov534.c driver and QT v4L2 software. Even though I ‘m seeing all of configuration settings of this camera on v4L2, I can’t set over 60fps. I can set any value under 60fps. Do you have an idea about this problem. What can I do for setting at least 120fps?
Unfortunately, I don’t have any experience with the PS3 Eye Camera, so I’m not sure what the proper parameter settings/values are for it. I hope another PyImageSearch reader can help you out!
Hey Adrian,
I’m working on an image processing pipeline with live display that runs at 75FPS without cv2.imshow() and 13 FPS with no screen output (OS-X FPS #s). I need a live output to the screen, and I want to maximize framerate. I already tried using Pygame’s blit command as an imshow() replacement, but got about the same speed. Are you aware of a module/command that will get an efficient screen refresh? If I’m lucky there will be an approach that will transfer from OS X to Raspberry Pi without too many hitches.
Thanks.
Hey Jon — when it comes to displaying frames to screen I’ve always used
cv2.imshow. Unfortunately I don’t know of any other methods to speedup the process. Best of luck with the project!I found that PySDL2 as an interface to the SDL2 library fixes the problem on OS-X and may be viable on Raspberry Pi too. For any interested readers, this post has code that can be integrated into the code here to speed things up when displaying all frames to the screen.
http://stackoverflow.com/questions/18434348/converting-cv2-images-to-pysdl2-surfaces-for-blitting-to-screen#19554202
Thanks for a great resource Adrian!
Thanks for sharing Jon!
Hello adrian.. i recently mixing this code with image filtering.. but the image had a quite delay,, the image is like quite frezee.. not like cv2.videocapture(-1). What happen..? I dont know this..Can you explain me..?
and if i use only serial usart communication i/o to send x y coordinate out.. where the code part must be change.. ? Thanks
Hey Hanifudin — I’m honestly not sure what you are trying to ask. Can you please try to elaborate on your question? I don’t understand.
1. I send coordinate x and y target with serial pin. But when i using threading,, raspberry pi cannot send data via serial..so, how i activate serial tx rx in threading mode..?
2. In threading mode,, the image is quite freeze,, like paused and lagging (not smooth)..it doesnt like non threading mode.. so how i make it smooth like non threading mode..?
Im sorry i’ve bad english
Unfortunately, I don’t have much experience sending serial data via threading in this particular use case. I would try to debug if this is a computer vision problem by removing any computer vision code and ensure your data is being properly sent. As for the slow/freezing video stream, I’m not sure what the exact problem is. You might be overloading your hardware or there might be a bug in your code. It’s impossible for me to know without seeing your machine.
Oh,, i know how why captured image is quite freeze,, im using usb hub to connect webcam,, wifi adapter and wireless mouse.. when i connect without usb hub it solved..
And sending serial data in threading mode is waiting to solved
Hi Adrian,
Thanks for the codes!
I tried to run the code. it works perfectly without display but I get an error “Segmentation fault: 11” when sampling threaded frames with display on. what should I do?
And I’m also doing a project with python and opencv try to record 120fps video, my camera is good for the fps but the utmost I can get is 30fps, any recommendations?
Chris
Regarding the segmentation fault, that sounds like a threading error. Most likely either the OpenCV window is being closed before the thread is finished or vice versa. It’s hard to debug that error, especially without access to your machine.
Also, keep in mind that you cannot increase the physical FPS of your camera. The code used in this blog post shows how you can increase the frame processing rate of your pipeline. If your camera is limited to 30 FPS, you won’t be able to go faster than that. However, you will use this technique when you add more complex steps to your video processing pipeline and need to milk every last bit of performance out of it.
Hi Adrian, your post is impressive! I have similar problem right now but I’m working on 16 bits sensors/cameras instead. Can I apply this to 16 bits? Do you know how to use opencv to capture 16 bits frames from usb device? Thanks!
Hi Guille — I haven’t tried this code using a 16 bit raw sensor to OpenCV, so unfortunately I don’t have any insight here.
Hi,
Its a great tutorial. I am new to python and opencv and your explanation is great . Thanks for posting and explaining everything line by line. I understood the program, but if i want to store the frame using videowriter instead of viewing them, where do i do it. I am thinking wither in the webcamcapture class so that it is done in the thread itself or should i do it in the main thread where you use cv2.show.
Currently i am using a webcam to capture at 1280×720 @ 30 hz, but i only get 8Hz. I am trying to improve this and want 30 Hz. Is this possible.
If you would like to save the frames to disk, please see this blog post where I discuss
cv2.VideoWriter.Hi , Thank you for this awesome tutorials !
I’m using RPi3 with Raspbian Stretch.
When I run fps_demo.py, I get the error message as shown:
[INFO] sampling frames from webcam…
Traceback (most recent call last):
…
(h, w) = image.shape[:2]
AttributeError: ‘NoneType’ object has no attribute ‘shape’
I saw your OpenCV: Resolving Nonetype errors post but I’m still confused.
Could you help me solve this error?
Oh, figured out that it was for the webcam, not the picamera. I saw your increasing picamera fps and it was all good.
Is it possible to use this method if I unify picamera and cv2.VideoCapture as you posted in here ?https://www.pyimagesearch.com/2016/01/04/unifying-picamera-and-cv2-videocapture-into-a-single-class-with-opencv/
I would suggest simply using the
VideoClassdiscussed in the link you included. This is the best method to achieve computability with both the Raspberry Pi camera module and USB cameras.Hi Adrien,
Thank you for the wonderful tutorial. I’ve developed an app to get 4 camera streams and show and save one video with all 4 streams. I’ve also used your threaded streams approach for better performance. The problem is, I cannot figure out correct FPS to use for the output video (in the VideoWriter). Any thoughts on this ?
Please see this blog post, in particular the comments section. The gist is that you’ll likely need to tune the FPS experimentally.
instead cv2.VideoCapture(0)
I replaced it with WebcamVideoStream(src=0)
then @ ret,frame = cap.read()
ERROR :too many value to unpack
why it happen and how to fix it ?
Please refer to this blog post. Notice how the
.readmethod ofWebcamVideoStreamonly returns a single value:frame = cap.readby use this code, my program running noticeable faster , but~ still have some delay, what others I can do ? or do u have any tutorial about use GPU to make it faster ?
I’m not sure what you’re asking. What in particular are you trying to optimize via the GPU?
basically, you code made my program running much faster , thanks for that , but it still not fast enough, I wonder if any other ways can make it even better ?
What type of hardware are you using? Laptop? Desktop? Raspberry Pi? I would suggest trying to optimize the OpenCV library itself, as we do in this post.
Hi Adrian, Awesome tutorials. I have learned a ton. I wish all online tutorials were as thoughtful and well-designed. Thank you.
I was trying to time the latency of a few of my web cams. Basically, I took your program and revised it a bit. It starts off by reading about 100 frames to warm the camera up. After 100 frames, it starts a timer and then uses serial via an Arduino to light a blue LED in front of the lens. The program then counts the number of frames read after the LED is lit and the time that elapses. Using 640×480 and 60fps, without threading, it takes about .09 secs between turning on the LED and identifying the LED. (It does this in one frame, which I gather means there’s no buffer.) With threading, it takes about .06 seconds and reads about five frames. With 720p images, there’s almost no difference in latency bw threaded and unthreaded approaches. So, have three questions please:
1) If the frames come in faster than the program can process them, will a queue build up and will that queue get longer and longer as the program runs? Is there any way to clear it out?
2) I am using this for robotics, so I would actually want to read the newest frame and dump the older frames, even though they haven’t been read. (I actually want last in first out rather than first in first out.)
3) Why would threading not improve the latency of capturing video at higher resolutions?
Sorry for the long question, but I thought more detail was better.
Thanks Jim, I appreciate that.
1. That really depends on the data structure you use. With a traditional list, yes. Other data structures place limits on the number of items to store in the queue. It’s really implementation dependent.
2. If that’s the case, use a FIFO queue. The Python language has them built-in my default.
3. It’s an issue of more data to transfer. The less data we transfer, the faster the pipeline. The more data we transfer, the slower the pipeline. Then there are overheads on packing/unpacking the data which we can’t really get around.
Hi Adrian
Thank you for the tutorial! It helps a lot (i am actually a greenhorn in programming :-))
I have a quick question: I have a C920 USB Webcam attached to a raspberry 3b. Before i start the script, i can set the cameras resolution to 1920×1080 (v1l2-ctl –set-fmt-video=width=1920,height=1080,pixelformat=1). I check the settings with: vl2-ctl -d /dev/video0 -V. Then i can see that the resolution was changed to 1920×1080 and the codec to H264.
When i start your script, the resolution gets changed to 640×480 and the codec is changed to YUYV. I cannot set the resolution while your script is running (“Device is busy”). So i have to set it in your script. Do you have a clue where to set the resolution/codec?
Best regards,
Roman
Hey Roman — you would need to download a copy of the
WebcamVideoStreamclass and then modify theself.streamobject in the constructor after Line 9. The capture properties can be set via the.setfunction but they are a bit buggy depending on the camera and install. You can read up more on them here.Hi Adrian,
I use Logitech 920, and I put some more lines after self.stream = cv2.VideoCapture(src)
def __init__(self, sec=0):
self.stream = cv2.VideoCapture(src)
self.stream.set(cv2.CAP_PROP_FRAME_WIDTH, 1280)
self.stream.set(cv2.CAP_PROP_FRAME_HEIGHT, 720)
self.stream.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*”MJPG”))
but the resolution remains the same.
On the other hand, iit works without threading
webcam = cv2.VideoCapture(0)
webcam.set(cv2.CAP_PREP_FRAME_WIDTH, 1280)
webcam.set(cv2CAP_PREP_FRAME_HEIGHT, 720)
Is that a buggy thing you mentioned?
IF you’re modifying the actual imutils library itself then it should use the dimensions you supplied. Make sure you’re not accidentally editing a different imutils install on your system and make sure you’re importing the correct ones (a few “print” statements will help you debug that).
I was checking imutils code and the things is, that using it doesn’t give you a bigger framerate.
When you call read() it will return the last read frame. You can call it 5 times, and get the same frame back over and over. So this does not improve the framerate, instead if gives you repeated frames creating the illusion of a bigger framerate
Hey Javier — you are correct that .read() will return whatever the newest frame is in the buffer. If your processing pipeline is so incredibly fast that you can process a frame before the next one is ready you could process duplicate frames. For most processing pipelines that will not be the case. Perhaps I didn’t make it clear enough in this blog post (but I tried to in the follow up one) that we are increasing the frame rate of the processing pipeline, not necessarily the frame rate of the actual camera sensor.
Hello
Sir
I cant detect face using rpi.
Help me.
Which code are you using to detect faces? For what it’s worth, I demonstrate how to perform face detection on the Raspberry Pi inside Practical Python and OpenCV.
Hi Adrian,
I love your blog and all the posts in it.
I am working with Python 3.6 on a Windows 8.1 machine.
The code runs perfectly except that it doesn’t exit from command-line on completion.
Is it because some threads are still active in the background?
Please help me resolve this tiny problem.
Thank you for your help!
Keep posting 🙂
Hey “Menaka” — can you elaborate on what you mean by “doesn’t exit from command-line completion”? I’m not sure what you are referring to.
how to reduces frame per second in video file
Just to clarify, are you referring to writing a frames to an output file to disk?
Thanks for reply
Actually we are doing object detection in video using tensorflow. As we are using cpu the video file hardly runs. so we just tried to reduce the fps read by opencv
So the TensorFlow model is running too quickly on each of the input frames? In that case, just add a time.sleep call to the end of your loop that is responsible for grabbing new frames.
This is great tutorial, but I have a couple questions: 1) How can I make use of cv2.waitKey() on a headless system (ie: with no windowing system, accessed over SSH)? and 2) None of this works with the PiCamera module. What would be the equivalent code for that?
Oh wait, nevermind! I see there’s a separate tutorial for the PiCamera module (https://www.pyimagesearch.com/2015/12/28/increasing-raspberry-pi-fps-with-python-and-opencv/ ). Thanks!
Yep, no problem Phil. Let me know if you have any other questions.
Hi everybody !
I used this method to read a video not from my cam but from another stream. The problem is that if you read directly from the file, opencv will try to read it as fast as possible. If you want to stick with the right frame rate, add a sleep(1/fps) in the thread update method 🙂
Thank you Adrian for this great tuto again !
I am doing a project where my robot follows a person using a color tracking method,
i am using the camera https://www.amazon.in/Logitech-C270-HD-Webcam-Black/dp/B008QS9J6Y/ref=sr_1_2?ie=UTF8&qid=1506877018&sr=8-2&keywords=usb+webcams ,
using this camera I am getting 5-second delay to show the streaming using cv2.videocapture(0)
Can u tell me suggestion to reduce the delay?
i am using raspberry pi 3 and python code.
can u tell me to get a low latency video using cv2.videocapture()
A five second delay sounds incredibly high. Can you tell us more about the stream? You mentioned using a robot, but is the robot streaming the video stream to another system where the processing takes place?
Hi Adrian,
I love your blog, it is really usefull.
I have a technical question:
I am doing an pupil diameter project and with every action I do to measure the pupil (aka: morphological transformation, median filter, among others) I lose FPS, so I start reading the video at 30 FPS and at the end it is only 4 FPS.
This is do to all the action I do in the frame video? If I use a camera with a higher default FPS, will I get at the end more FPS?
The problem isn’t your camera, it’s your processing pipeline. Think of your pipeline as a staircase. The less steps there are (i.e., computer vision or programming operations), the less stairs on the staircase and the faster you can clime it. The more operations, the more stairs, and the longer it takes. You’re in the same boat here — you need to optimize your actual processing pipeline.
Adrian, wow. I can’t thank you enough for all of your tutorials on OpenCV with dnn + Python. My question to you is, I am looking to consume an rtsp stream from IP Cameras on my network. I’ve seen some basic examples of how to capture frames in to memory but not sure I am confident in integrating the two (Your real time object detection and the rtsp frame capture). Do you have any advice or know of a tutorial that showcases this? I guess it doesn’t necisarily have to be rtsp, even http (if that’s all the camera supports). My initial thought was to have a different script (or thread) capturing frames and feeding them through the script we wrote in your tutorial for “Object detection with deep learning and OpenCV”. Would love to know your thoughts.
Hey Mike, thanks for the comment! I’m so happy to hear you are enjoying the PyImageSearch blog 🙂
I do not have any tutorials on RTSP but I’ve added it to my queue for later this year. I’ll try to do a post on it sooner rather than later. As far as combining the two, is your goal to apply object detection to the frame after you grab it from an RTSP stream? Or are you trying to pull an image from your webcam, apply object detection, and then write it back out to the RTSP stream?
Thanks Adrian
I using the code “Real-time object detection with deep learning and OpenCV” for object detect from IPCAM hikvision throught RTSP stream.
Its great solution, but after a few minutes I have problems with the get frame: the error is:
————————————————————-
frame = imutils.resize(frame, width=400)
File “/usr/local/lib/python2.7/dist-packages/imutils/convenience.py”, line 45, in resize
(h, w) = image.shape[:2]
AttributeError: ‘NoneType’ object has no attribute ‘shape’
—————————————————————
I think the problem is the processing pipeline and frame delay. I will hope your blog about RTSP stream.
Cheers from Loja-Ecuador
Rodrigo
I think it may be an issue related to the RTSP stream — perhaps the stream is dropping. That said you can handle the error gracefully by checking if the frame is “None” before you continue processing it:
if frame is None: continueThat way if a frame drops during the stream you can still handle the issue and ideally the next frame will not be dropped.
Hi Adrian. If my VideoCapture source is an RTSP stream, does that make it pointless to use threading because there is no real processing of frames on my side?
I use your code for web cam capture and must say that it bounded by physical speed of sensor so in my case (c270 + ubuntu 16.04) the quality of the video the same with blocking capture and without it. But somehow guvcview retrieve delivers much better FPS/
As far as I understand it might be the case that when you read a frame you skip may over previous ones. I think you need a queue to make sure you’ve e.g. displayed everything you’ve read. It is spectacularly optimized but practically I am not sure if it’s that useful. :/ 🙂
* To add more, I think your code is physically indeed reading all these frames but if you decide to process of show each of them, it will fail as it’s always fetching the most recent read-in frame. It might have read 2-3 others in that time, which will remain hidden.
This is a feature not a bug. Some computer vision pipelines cannot run in true real-time. If you used a queue you would end up storing all frames that need to be processed. The queue would eventually build up and your script would never complete. The point of this script is to process the most recent frames only. If you want to process every single frame you can use the raw
cv2.VideoCapturefunction.Hi,i am using this in combination with an object detection api. It works well but it would be kind of you if you can let me know how do I manage skipping of frames.i mean I don’t want to skip too many frames and make it a bit faster. For example with yolov3 python wrapper I get fps of around 11 I don’t want a dramatic increase it would be nice if I can get it upto 15 fpsor so. Do i need to use a queue instead?
Instead of the obvious things like “check to see if the frame should be displayed to our screen”, i’d rather read why strange things like “& 0xFF” are there. Is it a Mac thing? If so, other code appears to use Windows style, then.
Code needed
You can use the “Downloads” section of this post to download the source code.
Great blog, thanks for the clear, thorough, and complete instructions and useful tips. You say in this post that using threading this way reduces “latency” because the program “always [has] a frame ready for processing.” I think that processing the same frame repeatedly is likely to *increase* latency (compared to not re-processing the same frame) rather than reduce it because processing a genuinely new frame will be delayed until after (re-)processing an old frame has finished. If processing takes, say, 90% of the frame period, the first frame would be processed twice and the second not until 80% of a period after it’s ready. The FPS numbers collected by re-processing the same frame are a bit meaningless. I’m new to python but this worked for me: add a threading.Condition() to the WebcamVideoStream; then wrap the block in the loop in update() in a “with self.cv:”; then call “self.cv.notify_all()” when the new frame is ready; then, in read(), call self.cv.wait() before returning the frame. This causes the caller to wait until a *new* frame is ready but to process it *immediately* when it is actually ready. In my 90% example, the second frame will be processed immediately and the result will be ready much sooner than with the re-processing method. If what matters is reacting to what the camera sees, this would seem to be a further improvement in (reduced) latency.
Hey Adrian, I don’t know if you still reply to old posts, but might as well give it a try.
I’m using your tutorial to thread my opencv image subtraction code for multi-object tracking. On my computer, I get well above 90fps, but on the pi, I can only get 0.5-0.9fps…
I assume it’s because of my code, but anyway I’m trying to improve it step by step…
During an image subtraction, what should the threads be? how to separate the capturing and the processing of the images?
The way I’m doing it is capturing two grayscale frames, absolute diff and thresholding, then find the contours, and draw the bounding boxes…
I just can’t seem to understand how and what to separate in different threads for a better fps
thanks a lot for your wonderful blogs!
Hello adrian, i implemented this kind of threading for an IP camera stream. But i see that the one using threading uses more CPU than the one that does not. Do you know what might cause this, and how i can alleviate this?
Thanks!
It’s because the
cv2.VideoCapturemethod is constantly running in the background. You might instead want to use a blocking operation or manually update the class to only read new frames every N seconds.what difference does it have between a threaded videocapture and just a videocapture in a while loop? can you elaborate a bit? Is it simply because in my videocapture while loop, the cv2.imshow is there, preventing the update from running at super high speed?
The “cv2.imshow” call is going to take up a lot of time. If you remove it you’ll see your pipeline speedup.
As for the threading, I’m not sure I understand your question. This tutorial shows you how to wrap cv2.VideoCapture in a threaded class to make sure it doesn’t block. A normal cv2.VideoCapture in a while loop would be a blocking operation but it may reduce CPU load as you don’t have a thread in the background constantly polling for new frames.
i see. The thread itself is the reason why the cpu load is high.
Also i noticed the grab and retrieve methods of a cv2.VideoCapture object, i would probably try to use grab in place of read for the thread, and only use retrieve when i explicitly need the frame so it only decode the image when needed.
Thank you adrian!