Faster Real-Time Video Processing using Multi-Threading in Python
Overview
This blog post presents 2 code examples — with and without multi-threading in python — for reading video frames from a webcam connected to a laptop or desktop. With only few extra lines of code and without requiring to understand multi-threading in detail, real time video processing can be made faster with a higher FPS — more frames processed per second.
Complete code can also be accessed here — LINK
Reason multi-threading helps with faster processing
There are 2 main sub-parts in the video processing code — reading the next available frame from a webcam and performing some video processing on the frame like running a deep learning model for detection of faces, etc.
In a program without multi-threading, both reading the next frame and processing happen sequentially. Program waits for the next frame to be available and once it is available, performs the required processing on it. Time taken for reading the frame is mainly due to the time it takes for requesting, waiting and transferring the next video frame from camera to memory. Time taken for video processing is mainly due to the time it takes to perform computations on the video frame, either on the CPU or GPU.
In a program with multi-threading, reading the next frame and processing it are not required to be sequential. While one thread performs the task of reading the next frame, the main thread can use the CPU or GPU to process the last read frame . In this way by overlapping the two tasks, total time for reading and processing the frames can be reduced.
Code Walkthrough — Without Multi-Threading
Below is a walkthrough of the code without multi-threading. Complete code can also be accessed here LINK
- Importing required libraries
# importing required libraries
import cv2 # OpenCV library
import time # time library 2. Opening video capture stream using OpenCV library
# opening video capture stream
vcap = cv2.VideoCapture(0)
if vcap.isOpened() is False :
print("[Exiting]: Error accessing webcam stream.")
exit(0)
fps_input_stream = int(vcap.get(5)) # get fps of the hardware
print("FPS of input stream{}".format(fps_input_stream))grabbed, frame = vcap.read() # reading single frame for initialization/ hardware warm-up
3. Processing frames read from input stream
1. vcap.read() function call is used for reading the next available frame
2. delay variable is used in the code for simulating time taken for processing the frame. Different amounts of delay can be used to evaluate performance.
3. cv2.imshow(‘frame’, frame) is used to display the frame. With a higher value assigned to the delay variable , for example delay=1 for 1 second delay, we can notice that the displayed video is not smooth due to missing frames
# processing frames in input stream
num_frames_processed = 0
start = time.time()
while True :
grabbed, frame = vcap.read()
if grabbed is False :
print('[Exiting] No more frames to read')
break # adding a delay for simulating video processing time
delay = 0.03 # delay value in seconds
time.sleep(delay)
num_frames_processed += 1 # displaying frame
cv2.imshow('frame' , frame)
key = cv2.waitKey(1)
if key == ord('q'):
break
end = time.time()
4. Printing elapsed time and fps . Also, releasing input stream and closing all windows
# printing time elapsed and fps
elapsed = end-start
fps = num_frames_processed/elapsed
print("FPS: {} , Elapsed Time: {} ".format(fps, elapsed))# releasing input stream , closing all windows
vcap.release()
cv2.destroyAllWindows()
Code Walkthrough — With Multi-Threading
Below is a walkthrough of the code with multi-threading. Complete code can also be accessed here LINK
- Importing required libraries
# importing required libraries
import cv2 # OpenCV library
import time # time library
from threading import Thread # library for multi-threading2. Defining a helper class for implementing multi-threading
1. __init__() — initialization method for starting video capture stream and creating a thread object
2. start() — method to start the thread for reading next available frame
3. update() — method passed to thread which it runs in the background to read the next available frame
4. read() — method to return the latest available frame
5. stop() — method to stop reading more frames
# defining a helper class for implementing multi-threading
class WebcamStream :
# initialization method
def __init__(self, stream_id=0):
self.stream_id = stream_id # default is 0 for main camera
# opening video capture stream
self.vcap = cv2.VideoCapture(self.stream_id)
if self.vcap.isOpened() is False :
print("[Exiting]: Error accessing webcam stream.")
exit(0)
fps_input_stream = int(self.vcap.get(5)) # hardware fps
print("FPS of input stream: {}".format(fps_input_stream))
# reading a single frame from vcap stream for initializing
self.grabbed , self.frame = self.vcap.read()
if self.grabbed is False :
print('[Exiting] No more frames to read')
exit(0) # self.stopped is initialized to False
self.stopped = True # thread instantiation
self.t = Thread(target=self.update, args=())
self.t.daemon = True # daemon threads run in background
# method to start thread
def start(self):
self.stopped = False
self.t.start() # method passed to thread to read next available frame
def update(self):
while True :
if self.stopped is True :
break
self.grabbed , self.frame = self.vcap.read()
if self.grabbed is False :
print('[Exiting] No more frames to read')
self.stopped = True
break
self.vcap.release() # method to return latest read frame
def read(self):
return self.frame # method to stop reading frames
def stop(self):
self.stopped = True
3. Starting webcam stream and processing video frames . Code is similar to the non-threaded version but uses the newly defined WebcamStream class for reading frames from webcam using a multi-threaded approach.
# initializing and starting multi-threaded webcam input stream
webcam_stream = WebcamStream(stream_id=0) # 0 id for main camera
webcam_stream.start()# processing frames in input stream
num_frames_processed = 0
start = time.time()
while True :
if webcam_stream.stopped is True :
break
else :
frame = webcam_stream.read() # adding a delay for simulating video processing time
delay = 0.03 # delay value in seconds
time.sleep(delay)
num_frames_processed += 1 # displaying frame
cv2.imshow('frame' , frame)
key = cv2.waitKey(1)
if key == ord('q'):
break
end = time.time()
webcam_stream.stop() # stop the webcam stream
4. Printing elapsed time and fps . Also, closing all windows
# printing time elapsed and fps
elapsed = end-start
fps = num_frames_processed/elapsed
print("FPS: {} , Elapsed Time: {} ".format(fps, elapsed))# closing all windows
cv2.destroyAllWindows()
Thanks for reading !!!
