Description
In this problem you will perform block matching motion estimation between two consecutive video frames. Follow the instructions below to complete this problem.
- Download the two video frames from files frame_1 and frame_2 files. The frames/images are of hight 288 and width 352.
- Load the frame with the file name “frame1.jpg” into a 288 x 352 MATLAB array using function “imread”, and then convert the array type from 8-bit integer to real number using function “double” or “cast” (note that the range of intensity values after conversion is between 0 and 255.) Denote by 𝐼“the converted MATLAB array. Repeat this step for the frame with the file name “frame2.jpg” and denote the resulting MATLAB array by 𝐼#. In this problem, 𝐼# corresponds to the current frame, and 𝐼“ corresponds to the previous frame (i.e. the reference frame).
- Consider the 32 x 32 target block in 𝐼# that has its upper-left corner at (65,81) and lower-right corner at (96,112). Note this is MATLAB coordinate convention, i.e., the first number between the parenthesis is the row index extending from 1 to 288 and the second number is the column index extending from 1 to 352. The target block is therefore a 32 x 32 sub-array of 𝐼#.
- Denote the target block by 𝐵%&’()%. Motion estimation via block matching searches for the 32 x 32 sub-array of 𝐼“ that is “most similar” to 𝐵%&’()%. Recall that in the lectures we have introduced various forms of matching criteria e.g. correlation coefficient, mean-squared error (MSE), mean-absolute-error (MAE), etc.
In this problem, we use MAE as the matching criterion. Given two blocks 𝐵“ and 𝐵# both of size M x N, the MAE is defined as
= :
1
𝑀𝐴𝐸(𝐵“, 𝐵#) = 𝑀 × 𝑁 5 5|𝐵“(𝑖, 𝑗) − 𝐵#(𝑖, 𝑗)|
><” ;<”
To find the block in 𝐼“ that is most similar to 𝐵%&’()% in the MAE sense, you will need tos can through all the 32 x 32 blocks in 𝐼“, compute the MAE between each of these blocks and 𝐵%&’()%, and find the one that yields the smallest of MAE.
Note that in practice motion search is only performed over a region of the reference frame, but for the sake of simplicity, we perform motion search over the entire reference frame 𝐼“, in this part. Determine the coordinates of the upper-left corner of the matched block in MATLAB convention. What is the backward motion vector for the center pixel of 𝐵%&’()%?
- What is the corresponding MAE value (up to two decimal points)?
- Find and show the frame difference (DFD = 𝐼# − 𝐼“). Estimate the backward motion vector of each block in frame 𝐼# by using 16 x 16 blocks and searching for the best matching block in 𝐼“. Use a search region of size 64×64. Plot the motion vectors on frame 𝐼# using the MATLAB function quiver().
Predict frame 2, i.e., 𝐼?# by motion compensating the blocks in frame 𝐼“. Find and show the dispaced frame difference (DFD = 𝐼# − 𝐼?#) (similar to the images in slide 57 of motion estimation lecture notes). Compute the PSNR value between the original image 𝐼# (converted to type ‘double’) and the predicted image 𝐼?#.
- Repeat step (6) using logarithmic search and compare your results. What are the advantages/disadvantages of using logarithmic search?
- Discuss the advantages and disadvantages of the block-matching based motion estimation method.
Problem 2
In this problem you will perform median filtering to enhance the quality of a noise corrupted image. Recall that median filtering is effective for removing “salt-and-pepper” noise from images. Follow the instructions below to complete this problem.
- Download the noisy image. Load the noisy image into a MATLAB array and convert the type of the array from 8-bit integer ‘uint8’ to real number ‘double’. Visualize the noisy image using the built-in MATLAB function “imshow”. The function “imshow” takes as its argument either [0,255] for an 8-bit integer array (i.e. of type ‘uint8’), or [0-1] for a normalized real-valued array (i.e. of type ‘double’). To provide imshow with the correct argument, you would need either “cast” your real-valued array into “uint8“, or normalize it by 255.
- Perform 3×3 median filtering using the built-in MATLAB function “medfilt2”. For this problem, the only argument you need to provide “medfilt2” with is the array you have created in step (1). Visualize the filtered image using “imshow”. Remember to either cast the result to ‘uint8’ or normalize it before feeding it to “imshow”.
- Perform a second-pass median filtering on the filtered image that you have obtained from step (2). Visualize the two-pass filtered image. Compare it with the noisy input image and the 1-pass filtered image.
- Download the noise-free image. Compute the PSNR value between the noise-free image and the noisy input image (up to two decimal points).
- Calculate the PSNR value between the noise-free image and the 1-pass filtering output (up to two decimal points).
- Calculate the PSNR value between the noise-free image and the 2-pass filtering output (up to two decimal points).
