Description

(a) Bilinear Demosaicing 

To capture color images, digital camera sensors are usually arranged in form of a color filter array (CFA), called the Bayer array, as shown in Figure 1. Since each sensor at a pixel location only captures one of the three primary colors (R, G, B), the other two colors have to be re-constructed based on their neighbor pixel values to obtain the full color. Demosaicing is the process of translating this Bayer array of primary colors into a color image that contains the R, G, B values at each pixel.

Figure 1: (a) Single CCD sensor covered by a CFA and (b) Bayer pattern [1].

Implement the simplest demosaicing method based on bilinear interpolation. Exemplary demosaicing results are given in Figure 2. With this method, the missing color value at each pixel is approximated by bilinear interpolation using the average of its two or four adjacent pixels of the same color. To give an example, the missing blue and green values at pixel 𝑅_3,4 are estimated as:

As for pixel 𝐺_3,3, the blue and red values are calculated as:

Figure 2: (a) The original image and (b) the demosaiced image by bilinear interpolation.

Figure 3: The dog image with the CFA sensor input.

• Apply the bilinear demosaicing to the Dog image in Figure 3 and show your results.
• Compare your demosaiced image with the ground truth color image Dog_ori. Do you observe any artifacts? If yes, explain the cause of the artifacts and provide your ideas to improve the demosaicing performance.

(b) Malvar-He-Cutler (MHC) Demosaicing (20%)

Malvar et al. [2] proposed an improved linear interpolation demosaicing algorithm. It yields a higher quality demosaicing result by adding a 2nd-order cross-channel correction term to the basic bilinear demosaicing result. Both the bilinear and the MHC demosaicing results of the Fruit_Shop image are shown in Figure 4.

Figure 4: Demosacing results of Fruit_Shop image: the CFA input (left), the bilinear demosaicing result (middle) and the MHC demosaicing result (right).

The MHC algorithm is stated below.

To estimate a green component at a red pixel location, we have

where 𝐺̂^𝑏𝑙 is the bilinear interpolation result and the 2^nd term is a correction term. For the 2^nd term, 𝛼 is a weight factor, and Δ_𝑅 is the discrete 5-point Laplacian of the red channel:

To estimate a red component at a green pixel location, we have

where 𝛥_𝐺 is a discrete 9-point Laplacian of the green channel.

To estimate a red component at a blue pixel location,

where 𝛥_𝐵 is a discrete 9-point Laplacian of the blue channel. The weights control how much correction is applied, and their default values are:

 

The above formulas can be generalized to missing color components at each sensor location. Consequently, the MHC demosaicing can be implemented by convolution with a set of linear filters. There are eight different filters for interpolating the different color components at different locations as illustrated in Figure 5 [2].

 

Figure 5: Filter coefficients.

• Implement the MHC linear demosaicing algorithm and apply it to the Dog image in Figure 3. Show your results.
• Compare the MHC and the bilinear demosaicing results and explain the performance differences between these two algorithms in your own words.

 

 (c) Histogram Manipulation (20%)

Implement two histogram equalization techniques:

• Method A: the transfer-function-based histogram equalization method,
• Method B: the cumulative-probability-based histogram equalization method to enhance the contrast of the Toy image in Figure 6 below.
• Plot the histograms of the red, green and blue channels of the original image. The figure should have the intensity value as the x-axis and the number of pixels as the y-axis.
• Apply Method A to the original image and show the enhanced image. Plot the transfer function for each channel.
• Apply Method B to the original image and show the enhanced image. Plot the cumulative histogram for each channel.
• Discuss your observations on these two enhancement results. Do you have any idea to improve the current result?

Note that MATLAB users CANNOT use functions from the Image Processing Toolbox except displaying function like imshow().

 

Figure 6: Toy image

 

Problem 2: Image Denoising

In this problem, you will implement a set of denoising algorithms to improve image quality. You can use the PSNR (peak-signal-to-noise-ratio) quality metric to assess the performance of your denoising algorithm. The PSNR value for R, G, B channels can be, respectively, calculated as follows:

 

Remove noise in the image in Figure 7(b), compare it with the original image in Figure 7(a), and answer the following questions:

(a) Basic denoising methods

• What is the type of embedded noise in Figure 7(b)?
• Apply a linear filter of size N by N to the noisy image. Compare the performance of two choices of the filter parameters – the uniform weight function and the Gaussian weight function.

 

Figure 7: The original and noisy corn images.

 

 

(b) Bilateral Filtering

In most low-pass linear filters, we often see degradation of edges. However, using some nonlinear filters, we can preserve the edges. Bilateral filters are one such kind of filters. A discrete bilateral filter is given by:

where (𝑘, 𝑙) is the neighboring pixel location within the window centered around (𝑖, 𝑗), 𝐼 is the image with noise, 𝑌 is the filtered image. 𝜎_𝑐 and 𝜎_𝑠 are the spread parameters.

• Implement the bilateral denoising filter and apply it to the noisy image.
• Explain the roles of 𝜎_𝑐 and 𝜎_𝑠. Discuss the change in filter’s performance with respect to the values of 𝜎_𝑐 and 𝜎_𝑠.
• Does this filter perform better than linear filters you implemented in Problem 3(a)? Justify your answer in words.

 

(c) Non-Local Means (NLM) Filtering

The non-local mean filter utilizes the pixel value from a larger region rather the mean of a local window centered around the target pixel. A discrete non-local mean filter with Gaussian weighting function is as follows:

where 𝐼, 𝑌 are the noisy and filtered images respectively, 𝑁_𝑥,𝑦 is the window centered around location (𝑥, 𝑦), and ℎ is the filtering parameter, 𝑁^′ ≤ 𝑁 and 𝑀^′ < 𝑀 denote the window size of your choice.

The Gaussian weighted Euclidian distance between window 𝐼(𝑁_𝑖,𝑗)and 𝐼(𝑁_𝑘,𝑙) is defined as:

2                                                                                                                                                                                                   2

‖𝐼(𝑁_𝑖,𝑗) − 𝐼(𝑁_𝑘,𝑙)‖_2,𝑎        =     ∑     𝐺_𝑎(𝑛₁, 𝑛₂)(𝐼(𝑖 − 𝑛₁, 𝑗 − 𝑛₂) − 𝐼(𝑘 − 𝑛₁, 𝑙 − 𝑛₂))

𝑛1,𝑛2∈ℵ

 

where ℵ denotes the local neighborhood centered at the origin, 𝑛₁, 𝑛₂ ∈ ℵ denotes the relative position in the neighborhood window. 𝑎 > 0 is the standard deviation of the Gaussian kernel.

• Implement the NLM filter and apply it to the noisy image. Try several filter parameters and discuss their effect on filtering process. Clearly state your final choice of parameters in your report.
• Compare the performance of NLM with filters used in Problem 3(a) and Problem 3(b).

 

 

(d) Block matching and 3-D (BM3D) transform filter (10%)

In this part, you will get familiar with another state-of-the-art denoising algorithm proposed in [3].

• Please explain the BM3D algorithm in your own words.
• Implement the BM3D filter (Write your own code or use any available online source code but include the source in your reference) to denoise the noisy image House in Figure 7. Show the final noise-removed image, and compare the PSNR value and visual quality with those in (a).

Note: It is recommended that you use the code provided by the authors on their website [4]. Their code is written in MATLAB.

 

(e) Mixed noises in color image

Figure 8 (b) is a noisy color image corrupted with mixed types of noises. Please identify noise types in the image and answer the following questions:

• What types of noises are there?
• Should you perform filtering on individual channels separately for both noise types?
• What filters would you like use to remove mixed noise? Can you cascade these filters in any order? Justify your answer.

 

 

(a)                                                           (b)

Figure 8: (a) the original pepper image (b) the pepper image with mixed noises.

 

             

Appendix:

Problem 1: Image Demosaicing and Histogram Manipulation

Dog.raw                              600×532	8-bit	gray
Dog_ori.raw                        600×532	24-bit	color(RGB)
Toy.raw                               560×400 Problem 2: Image Denoising	24-bit	color(RGB)
Corn_gray.raw                    320×320	8-bit	gray
Corn_noisy.raw                  320×320	8-bit	gray

 

Reference Images

All images in this homework are from Google images [5] or the USC-SIPI image database [6].