Being raised on the east coast of the United States I guess I should be used to the snow by now — but I’m not. As a kid, I loved the snow. I loved sledding, snowboarding, and snowball fights.
But now, as an adult, snow just means an inconvenience. It means travel is going to be terrible. It means not being able to drive to the gym. And it means being stuck in the apartment.
That said, I’m taking this snow day on the east coast and writing a blog post on capturing mouse click events with Python and OpenCV.
In this example we’ll click and drag a rectangular Region of Interest (ROI) and crop it from our image. This technique is especially helpful if you are labeling data as input to an image classification algorithm.
So if you’re trying to understand how mouse click events work with Python and OpenCV, then look no further! This blog post will show you exactly that!
OpenCV and Python versions:
In order to run this example, you’ll need Python 2.7 and OpenCV 2.4.X.
Capturing mouse click events with Python and OpenCV
Let’s go ahead and get this example started. Open up a new file, name it click_and_crop.py , and we’ll get to work:
# import the necessary packages
import argparse
import cv2
# initialize the list of reference points and boolean indicating
# whether cropping is being performed or not
refPt = []
cropping = False
def click_and_crop(event, x, y, flags, param):
# grab references to the global variables
global refPt, cropping
# if the left mouse button was clicked, record the starting
# (x, y) coordinates and indicate that cropping is being
# performed
if event == cv2.EVENT_LBUTTONDOWN:
refPt = [(x, y)]
cropping = True
# check to see if the left mouse button was released
elif event == cv2.EVENT_LBUTTONUP:
# record the ending (x, y) coordinates and indicate that
# the cropping operation is finished
refPt.append((x, y))
cropping = False
# draw a rectangle around the region of interest
cv2.rectangle(image, refPt[0], refPt[1], (0, 255, 0), 2)
cv2.imshow("image", image)
We’ll start by importing our two necessary packages: argparse for parsing command line arguments and cv2 for our OpenCV bindings.
We also define two global variables on Lines 7 and 8: refPt , which is a list of two (x, y)-coordinates specifying the rectangular region we are going to crop from our image, and cropping , a boolean indicating whether we are in cropping mode or not.
To process the mouse click events we define the click_and_crop callback function on Line 10. Anytime a mouse event happens, OpenCV will relay the pertinent details to our click_and_crop function.
In order for our function to handle the relay, we need to accept 5 arguments:
- event: The event that took place (left mouse button pressed, left mouse button released, mouse movement, etc).
- x: The x-coordinate of the event.
- y: The y-coordinate of the event.
- flags: Any relevant flags passed by OpenCV.
- params: Any extra parameters supplied by OpenCV.
From there we grab reference to our refPt list and cropping flag on Line 12.
We then make a check on Line 17 to see if our left mouse button was pressed. If it was, we record the (x, y)-coordinates of the event and indicate that we are now in “cropping mode”.
At this point we would drag out the rectangular region of the image that we want to crop. After we are done dragging out the region, we release the left mouse button — Line 22 handles when the left mouse button is released and updates the list of points containing our ROI. We also draw the rectangle representing the ROI on Lines 29 and 30.
Now, let’s see how we can use this function to crop an image:
# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required=True, help="Path to the image")
args = vars(ap.parse_args())
# load the image, clone it, and setup the mouse callback function
image = cv2.imread(args["image"])
clone = image.copy()
cv2.namedWindow("image")
cv2.setMouseCallback("image", click_and_crop)
# keep looping until the 'q' key is pressed
while True:
# display the image and wait for a keypress
cv2.imshow("image", image)
key = cv2.waitKey(1) & 0xFF
# if the 'r' key is pressed, reset the cropping region
if key == ord("r"):
image = clone.copy()
# if the 'c' key is pressed, break from the loop
elif key == ord("c"):
break
# if there are two reference points, then crop the region of interest
# from teh image and display it
if len(refPt) == 2:
roi = clone[refPt[0][1]:refPt[1][1], refPt[0][0]:refPt[1][0]]
cv2.imshow("ROI", roi)
cv2.waitKey(0)
# close all open windows
cv2.destroyAllWindows()
We start by parsing our command line arguments on Lines 33-35. We need only a single switch here, --image , which is the path to the image we want to crop. This image is then loaded and cloned on Lines 38 and 39. We make a deep copy of the image so we can draw the rectangular bounding box on Line 29 without destroying the original image.
Lines 40 and 41 handle registering our click_and_crop callback function. We first call cv2.namedWindow to create a window named “image”. And we then set the mouse callback by calling cv2.setMouseCallback , supplying our named “image” window and our click_and_crop callback function.
From here, the rest of this example just ties the pieces together.
We start a loop on Line 44 which (1) displays our image on screen and (2) waits for a keypress.
This is the point where we select the rectangular region we want to crop from the image.
If we press the r key after selecting the region to crop, we reset our cropping
And if we press the c key, then we break from the loop and perform the actual cropping on Lines 59-62.
Finally, we cleanup and close all open windows on Line 65.
OpenCV mouse events in action
Now that we have our example coded up, let’s try it out. Open up a terminal and execute the following command:
$ python click_and_crop.py --image jurassic_park_kitchen.jpg
You’ll first be presented with the image on your screen:
Select the region you want to crop by clicking, dragging, and releasing:
And finally press the cĀ Ā key to perform the crop:
As you can see, we have successfully cropped Tim’s face from the image.
Here is another example where we crop the Velociraptor:
So there you have it — a simple method to capture mouse events to crop an image using Python and OpenCV.
What's next? I recommend PyImageSearch University.
30+ total classes • 39h 44m video • Last updated: 12/2021
★★★★★ 4.84 (128 Ratings) • 3,000+ Students Enrolled
I strongly believe that if you had the right teacher you could master computer vision and deep learning.
Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?
Thatās not the case.
All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And thatās exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.
If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here youāll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.
Inside PyImageSearch University you'll find:
- ✓ 30+ courses on essential computer vision, deep learning, and OpenCV topics
- ✓ 30+ Certificates of Completion
- ✓ 39h 44m on-demand video
- ✓ Brand new courses released every month, ensuring you can keep up with state-of-the-art techniques
- ✓ Pre-configured Jupyter Notebooks in Google Colab
- ✓ Run all code examples in your web browser ā works on Windows, macOS, and Linux (no dev environment configuration required!)
- ✓ Access to centralized code repos for all 500+ tutorials on PyImageSearch
- ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
- ✓ Access on mobile, laptop, desktop, etc.
Summary
In this blog post we learned how to capture mouse events using OpenCV and Python. While this post did not provide a complete and exhaustive overview ofĀ all the mouse events you can capture, it laid the groundwork for what is possible.
This post also demonstrated how you can quickly crop and extract regions of an image, which is especially useful when creating training data for your own custom object detectors.
Download the Source Code and FREE 17-page Resource Guide
Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!
Another great post… and amazingly timely for me as you often are! This was a huge help for my current work. I did note, that if the bounding box was drawn starting from the right, moving left; it would not work. To eliminate this, with a little research, I added a line to draw the rectangle using the coordinates first ordered by the minimum x,y values- which would be the top left of the box, and got the lower right by subtraction. Now it doesn’t matter how it is drawn. The line is below.
“refPt = (min(ix,x), min(iy,y), abs(ix-x), abs(iy-y)) #set bounding box by mouse move”
Feeling emboldened by your challenge, I also added a cv2.EVENT_MOUSEMOVE (where that line sits), in hopes of seeing the box grow as it is drawn, I did this on a copy of a copy of the original image (didn’t want to use the original for any work). The best I could get here was to see a number of boxes growing larger as I moved the mouse… none of which disappeared, so I had a bunch of boxes on the finished image. The number of boxes depended on the speed I moved the mouse. The cropping still worked great, so the multiple boxes are a riddle for another day! I am working with the opencv grabcut code… which uses mouse movements to capture coordinates- and it draws a box nice and smooth. I cannot see what the difference is… so far!
Thanks again Adrian!
Hi MJ, thanks for the comment! And your code on setting the top-left coordinate of the box is fantastic — great addition! As for the multiple drawings of the bounding box, I ran into the same issue. I wanted to draw a nice smooth rectangle around the region, but I ended up with multiple rectangles drawn on the image, exactly like in your case. This problem will probably need some more investigation…
Hey Adrian, I have a workaround for this issue. Create a new global, in my case I called it sel_rect_endpoint. Then add this after the mouseup event:
elif event == cv2.EVENT_MOUSEMOVE and cropping: sel_rect_endpoint = [(x, y)]To store the values as the mouse moves.
Then, change the
imshowline in thewhile Trueloop to:if not cropping: cv2.imshow('image', image) elif cropping and sel_rect_endpoint: rect_cpy = image.copy() cv2.rectangle(rect_cpy, roi[0], sel_rect_endpoint[0], (0, 255, 0), 1) cv2.imshow('image', rect_cp)And you will see the rectangle as the mouse moves!
Awesome, thanks so much for the workaround Steve, that’s a great trick!
Hi Steve and Adrian,
Thanks for sharing such a great example to real-time boxing in OpenCV.
I’ve successfully run the combination of your 2 codes.
So, now I am trying to implement it as a function saved in a separate .py file and then import it/call it to a main program. However, when I do this the “real-time” drawing of the box did not perform as before. The box still get drawn after I release my mouse-click, but it is not as cool as a real-time drawing of the box.
My question is if such a limitation is to be expected when using a function saved in a different file, or is there a way around it?
Thank you very much.
Stephen
Hey Stephen, if you want to implement the functionality in a separate Python file (whether that’s a class or a set of functions), take a look at the
ClickCropperclass in the imutils repo. I haven’t had a chance to merge it into the repository yet, but it looks like what you’re trying to accomplish.how does this work it keeps on giving me an error
It raise error (NameError: name ‘roi’ is not defined)
Here are two changes that I made to Steve’s useful add-on to get it working:
1) Replace ‘roi[0]’ with ‘refPt[0]’ in ‘cv2.rectangle(rect_cpy, refPt[0], sel_rect_endpoint[0], (0, 255,0), 1)’.
2) Also note that ‘rect_cp’ should be ‘rect_cpy’ in ‘cv2.imshow(‘image’, rect_cp)’.
Thanks for the great post!
Just used this for my grab-cut algorithm. Thanks a lot!
Thanks a lot!
Awesome, glad to hear that it was helpful for you! š
The grab cut does not work properly when you set the flag cv2.WINDOW_OPENGL (in cv2.namedWindow)
Thanks for the tip Darwin!
Your blog is one of the rare really useful ones I found on internet.
Tricky problems with simple,straightforward but efficient solutions.
Without mentioning your precious article sent for free: Tthe ultimate barcode detection guide
Wish you all the best, Man.
Regards from BEGUERADJ
Thanks so much Begueradj, I really appreciate your kind compliment š
How would you format this so you could import the cropper tool as a module to be used in a seperate script? I’m confused as to how you “load” the picture you want to crop into the click_and_crop function? Basically, if you want to do this:
import click_and_crop as cc
image = cv2.imread(….)
and then somehow call
cc.click_and_crop(image)
how would you go about making this kind of thing work?
Thanks!
Hey Noah, I would consider reading up on constructing Python functions and classes. This isn’t so much a computer vision problem as a code organization problem.
Adrian as everyone here says, your blog is great and very usefull. I am using bits and pieces of your code for a seven segment display reader. I can get the mouse events to work great on a static image, but I am using the raspberry Pi 2 with the picam. When ever the mouse is over a window the code seems to freeze and get stuck in the mouse event loop. Once you remove the mouse from the window everything starts running nicely again.
Is there a way to keep the main code moving while the mouse is hovering over a window?
Hey Rick, that certainly is odd behavior. I have not tested this code out on a Pi 2 (only on my laptop), so I don’t have an answer for that off hand. The code should definitely not freeze like that, it makes me wonder if there is a problem with GTK and how the Pi is handling the window events. Can you try the code on your laptop and see if you get the same behavior? If not, then we’ll know the problem is related to the Pi.
I am having the same problem. As soon as I start the script, the mouse is unresponsive. The keyboard/camera/script still functions, I just can’t use the mouse to select a ROI. If I press ‘q’ to stop the script, the mouse becomes responsive again.
I have tested on a macOS and it works fine, so I assume it’s a problem with the Pi. Any help would be appreciated!
A quick update…if I VNC into the Pi, I can successfully use the mouse and select a ROI. So I assume the problem lies with how the Pi is handling the USB ports whilst the connected USB webcam is being used.
Still strange why only the mouse becomes unresponsive, and the problem persists even when using a powered USB hub for the kb+mouse.
I am very beginner of openCV and python, from Japan. I would appreciate your advice.
When I run this click_and_crop.py code, I have error message :
>>>
usage: click_and_crop.py [-h] -i IMAGE
click_and_crop.py: error: argument -i/–image is required
>>>
I have installed correct version:Python 2.7 and OpenCV 2.4.11
jurassic picture placed in the same directory.
Please execute the Python script via command line, not IDLE:
$ python click_and_crop.py --image jurassic_park_kitchen.jpgThere is no need to first launch the
pythoninterpreter and then execute the code. Just use the command above and the program will work.Thank you for your advice. I have succeeded running this code on raspberry pi.(^^āŖ
Congrats! š
Hi Adrian,
I found this tutorial very helpful. But for my project ,i should be implementing the click and crop technique for a real time video and return the cropped frame image after the mouse event.
Please help ..
Unfortunately you can’t really do click and crop for a real-time video. Basically, you would have the video playing, press a key to pause the stream, and then do your clicking and cropping. I actually detail how to do that in this post.
Adrian, Can you add a line to a live video feed with cv2.line() and re-engineer the code above?
Hey John — while I’m happy to provide code and explanations via blog posts for free, I cannot write custom code for you on a case-by-case basis. I hope you understand. If you need help getting started with computer vision I would suggest going through Practical Python and OpenCV.
Hi Adrian,
You made my day with this tutorial. Do you know how can i show the rectangle during the clic and not only after the drop?
Thank you very much!
Hi Adrian!
This post is really helpful. I am new to opencv and trying out few things with it. I have two seperate images and i want to display both on the same window and when i click on one image it should be superimposed on the second image. Can you help me?
Thank you in advance!
Normally I wouldn’t recommend using OpenCV for complex GUIs. The GUI toolkit provided with OpenCV is mainly for debugging and creating minimal applications. For more complex apps, you should look into an actual GUI framework, such as Tkinter or Qt. As for superimposing two images, you should read up on transparent overlays.
Hi Adrian,
I need to mark cars in the image. There are multiple cars in the image, which I need to mark as rectangles. How shall I update the code?
If you’re trying to mark and annotate your image, I think using dlib’s imglab tool is a better option.
Hi Adrian,
Thank you for the tutorial. I have a question: when I run this code although it works, I see the following error appearing continuously in the console :
cv2.rectangle(image, refPt[0], refPt[1], (0, 255, 0), 2)
IndexError: list index out of range
I was wondering if it’s normal or there is something wrong. Also, when I press “r”, the window does not restore, even when I press the “r” key several times until I choose a point in the image, and only after that pressing “r”will restore image, which is kind of annoying.
Another question: I try to do cropping several times in a loop, to get different regions of interest coordinates. So, basically each time a message appears on the console, asking the user to choose certain area of interest. After the area has been chosen, the refPt content is first stored in a variable, then it becomes empty again for the next round. But, for some reason it gets stuck there, the window doesn’t change and I cannot choose any new rectangle in the image. I even try to destroy the window after each round, and create a new one for the next round, still no success. I would appreciate if you could give me a hint on what I might be probably doing wrong here.
Update:
I got my code running in the loop, but I still have the issue of receiving the same error continuously, plus having the same problem with the “r” key.
That’s quite strange regarding the “r” key. Are you using an US-based keyboard layout? If not, the “r” key may be mapped to a different ordinal number.
Thank you for your reply Adrian. I got to fix the errors and make my code run smoothly through the loop. It was just some indentation thing, which is weird as I didn’t get any indentation error.
Congrats on resolving the error!
Hi Sir Adrian How could add code the solution of “MJ” and Steve to cut the region in reverse? I’ve been testing and I can not find the solution. Thanks in advance.
Can you elaborate on what you mean by “cut the region in reverse”? I’m not quite sure I understand what you mean.
Hello Adrian. Thank you very much for answering so quickly.
Your code works for me perfectly, again thanks for these tutorials fantanticos, it has started perform them less than a month ago and the truth I have learned well, fast and I’m gone really well.
Ā Reviews companions “MJ” and “Steve” add some more to this code so useful. Could you indicate how they have done it if it is so friendly and will not involve much hassle? Because I have tried to add and I could not … Again many thanks.
thank you, very useful.
hello Adrian Rosebrock
how extract the coordinates of selected rectangle area and add a label to them.
What type of label are you referring to?
Hi Adrian…this was really helpful…but i just wanted to know if this could be done without the argument parser..i.e just by calling the click_and_crop function for a loaded image…selecting the coordinates and returning them
Also if i want to do multiple cropping on the same image and create a new image with the extracted cropped parts at the same coordinates location off my original image with every other part that was not selected as white.
Thanks for this tutorial…it was extremely helpful
You don’t need the argument parser. If you wanted to skip this step, just hardcode the path to
cv2.imread.Your posts are extremely helpful!
typo on line 58
Hi!
I’m trying to do something similar with zoom, and I’m using EVENT_MOUSEWHEEL.
(scrolling mouse)
it seems that this function doesn’t work on Linux (Ubuntu).
Did you try to use this event?
I can’t find any information on the net.
Hi Daniella — I’m honestly not sure about that event as I’ve never used it before. Sorry I couldn’t be of more help here!
how to save ROI in folder
(How to SAVE ROI image in folder)
You would just need to use the
cv2.imwritefunction:cv2.imwrite("path/to/output.jpg", roi)Hi Adrian,
Really helpful tutorial. But when I try to crop the same image again, the program stops and quits the image screen. do you know how I can fix this? I need to be able to display several portions of the cropped image.
You would need to modify the functionality of the code. Right now this script assumes you’re interested in just one crop. You’ll want to include the
if len(refPt) == 2:statement in yourwhileloop and give each ROI window a unique name (otherwise the window names will overwrite each other).Quick note: the global variable `cropping` isn’t actually used in this script. I imagine it was used in an earlier trial where it was used for a `while` loop or `if` statement condition, but in the current state, it is simply set and unused.
I didnt work this program. please help me. how do I working this program in python Idle?
Sample:
import click_and_crop as cc
image = cv2.imread(ā¦.)
cc.click_and_crop(image)
This script cannot be easily executed using IDLE. Please save the program as a script and execute it via the command line.
Thank you Adrain for the post. How can we modify the code such that.. Every picture in the folder gets looped and cropping multiple parts of an single image and saving it into different file .
Example: Lets say I have two folders named “Players” and “CroppedPlayers”. There are 10 images of football being played inside “Players” folder and “CroppedPlayers” folder is empty. From each image I want to crop multiple players and save in side CroppedPlayers folder as separate image.
Suppose each image has 11 players in it and I have 10 such images I want to have 110 images as result saved in different folder.
(Sincerely looking forward)
There are multiple ways to accomplish this. You can use the tool as-is to mark ROIs, loop over each ROI, extract it, and write it to disk via
cv2.imwrite. I cover the fundamentals of OpenCV and image processing inside Practical Python and OpenCV.Otherwise, it might be easier to use a dedicated annotation tool such as dlib’s “imglab”, then process the resulting XML file and extract the ROIs.
Hi, Adrian.
A very nice post, Enjoyed it. However, is it possible to retain the contents inside a box, but remove the rectangle around it?
Suppose, I draw a 100’s of Green rectangle in the image, but now I wish to remove few(say 10) of these & retain the rest (other 90) of them. How do I do that?
I would suggest maintaining a Python list of rectangles and drawing the rectangles on a copy of the input image. You can then add/remove the rectangles list as you please.
Hi Adrian.
This post is simply amazing. However, is it possible to crop directly from the os screen, not from OpenCV window? Something like rectangular-snip in screen capturer program.
OpenCV’s GUI functions are limited. They are primarily meant for debugging or for extremely simple applications. You’ll want to look into the GUI libraries for your respective operating system to obtain the functionality you are describing.
Hi I found this code very helpful. I want the functionality of uploading a file via directory and then using the click and crop function on the image rather than typing name of file from the command line. can anybody help me with that ? I’m having a hard time figuring that out.
click_and_crop.py: error: the following arguments are required: -i/–image. Why am i getting this error?
If you are new to command line arguments, that’s okay, but make sure you read up on them first.
Hi Adrian.
when i use big size of image ,i can’t see whole of image for click some of point,
when i open image with open with …. that is ok i can see whole of image but when i use that image i python file , i can’t see whole of image, seems zoomig of image.
OpenCV provides limited sits of GUI functions. It’s not meant for full-blown GUI development. You might want to look into a dedicated GUI library such as QT or Tkinter.
Hello Adrian thanks for this post.
For some reason/task I want to use the right click on the video streaming window , but when I right click on that video streaming window a menu shows up , can u guide me to disable the menu option which originates on right clicking, so that I can use that right click hassle free.
What OS are you using? That unfortunately sounds like it may be limitation of your OS or window manager.
i am using Ubuntu 17.10
Yeah, that unfortunately sounds like your OS is opening an option menu on right click. OpenCV’s simple GUI features cannot update that. You should look into a dedicated GUI library like Qt, TKinter, or Kivy.
Turns out that there is a fix for this – avoid using the Qt functionality which comes by default in cv2.namedWindow().
To do this, change the call to cv2.namedWindow() as follows:
cv2.namedWindow(“image”, cv2.WINDOW_GUI_NORMAL|cv2.WINDOW_AUTOSIZE)
See also: https://stackoverflow.com/questions/28834980/nameerror-global-name-cv-gui-normal-is-not-defined
Hi, I was wondering how can I make something similar, but instead of making a new window and inserting the ROI in that window, make it so that in the same image I can choose another part where I will paste that ROI? Sort of like copying a ROI and pasting it in the same image
OpenCV’s GUI functionality is meant to be very basic and really only for debugging. If you want more advanced GUI functionality I would suggest using a dedicated GUI library like QT, Tkinter, etc.
Hi Adrian,
Fantastic tutorial. Could you make any suggestions on how to modify the code for cv2.circle in place of cv2.rectangle?
Hi, thanks for this post! Extremely useful.
I’m running it in Python 3.6 with OpenCV 3.4.2 on a MacBook Pro and the response time is very slow. It takes 2 or 3 seconds after I release the button for the rectangle to appear.
Anyone else experiences this? Any workarounds?
Hello Adrian, how to get the coordinates of the cropped image??
Hey Adrian, great tutorial, introduced me to a lot of new elements that can be used. Though I have one doubt which is- Where is the Boolean stored in ‘cropped’ used? I can’t seem to find anywhere in the code where it is. Sorry if its something obvious but this has been puzzling me for quite a while.
thanks anyways