[SOLVED] CSCI5561 Homework 4-Neutral Network

Figure 1: You will implement (1) a multi-layer perceptron (neural network) and (2) convolutiona neural network to recognize hand-written digit using the MNIST dataset.

The goal of this assignment is to implement neural network to recognize hand-written digits in the MNIST data.

MNIST Data You will use the MNIST hand written digit dataset to perform the first task (neural network). We reduce the image size (28 × 28 → 14 × 14) and subsample the data. You can download the training and testing data from here: http://www.cs.umn.edu/~hspark/csci5561/ReducedMNIST.zip

Description: The zip file includes two MAT files (mnist_train.mat and mnist_test.mat). Each file includes im_* and label_* variables:

• im_* is a matrix (196 × n) storing vectorized image data (196 = 14 × 14)
• label_* is n × 1 vector storing the label for each image data.

n is the number of images. You can visualize the i^th image, e.g., imshow(uint8(reshape(im_train(:,i), [14,14]))).

2             Single-layer Linear Perceptron

Figure 2: You will implement a single linear perceptron that produces accuracy near 30%. Random chance is 10% on testing data.

You will implement a single-layer linear perceptron (Figure 2(a)) with stochastic gradient descent method. We provide main_slp_linear where you will implement GetMiniBatch and TrainSLP_linear.

function [mini_batch_x, mini_batch_y] = GetMiniBatch(im_train, label_train, batch_size)

Input: im_train and label_train are a set of images and labels, and batch_size is the size of the mini-batch for stochastic gradient descent.

Output: mini_batch_x and mini_batch_y are cells that contain a set of batches (images and labels, respectively). Each batch of images is a matrix with size 194×batch_size, and each batch of labels is a matrix with size 10×batch_size (one-hot encoding). Note that the number of images in the last batch may be smaller than batch_size. Description: You may randomly permute the the order of images when building the batch, and whole sets of mini_batch_* must span all training data.

function y = FC(x, w, b)

Input: x∈R^m is the input to the fully connected layer, and w∈R^n×m and b∈Rⁿ are the weights and bias.

Output: y∈Rⁿ is the output of the linear transform (fully connected layer). Description: FC is a linear transform of x, i.e., y = wx + b.

function [dLdx dLdw dLdb] = FC_backward(dLdy, x, w, b, y)

Input: dLdy∈R^1×n is the loss derivative with respect to the output y.

Output: dLdx∈R^1×m is the loss derivative with respect the input x, dLdw∈R^1×(n×m) is the loss derivative with respect to the weights, and dLdb∈R^1×n is the loss derivative with respec to the bias.

Description: The partial derivatives w.r.t. input, weights, and bias will be computed. dLdx will be back-propagated, and dLdw and dLdb will be used to update the weights and bias.

function [L, dLdy] = Loss_euclidean(y_tilde, y)

Input: y_tilde∈R^m is the prediction, and y∈ 0,1^m is the ground truth label.

Output: L∈R is the loss, and dLdy is the loss derivative with respect to the prediction. Description: Loss_euclidean measure Euclidean distance L = ky − yk². e

function [w, b] = TrainSLP_linear(mini_batch_x, mini_batch_y)

Input: mini_batch_x and mini_batch_y are cells where each cell is a batch of images and labels.

Output: w∈R^10×196 and b∈R^10×1 are the trained weights and bias of a single-layer perceptron.

Description: You will use FC, FC_backward, and Loss_euclidean to train a singlelayer perceptron using a stochastic gradient descent method where a pseudo-code can be found below. Through training, you are expected to see reduction of loss as shown in Figure 2(b). As a result of training, the network should produce more than 25% of accuracy on the testing data (Figure 2(c)).

Algorithm 1 Stochastic Gradient Descent based Training

1: Set the learning rate γ

2: Set the decay rate λ ∈ (0,1]

3: Initialize the weights with a Gaussian noise w ∈N(0,1)

4: k = 1

5: for iIter = 1 : nIters do

6:               At every 1000^th iteration, γ ← λγ

7:              0 and

8:                for Each image x_i in k^th mini-batch do

9:                     Label prediction of x_i

10:                   Loss computation l

11:                  Gradient back-propagation of  using back-propagation.

12:                  w                             w                       and

13:           end for

14:              k++ (Set k = 1 if k is greater than the number of mini-batches.)

15:

3             Single-layer Perceptron

Figure 3: You will implement a single perceptron that produces accuracy near 90% on testing data.

You will implement a single-layer perceptron with soft-max cross-entropy using stochastic gradient descent method. We provide main_slp where you will implement TrainSLP. Unlike the single-layer linear perceptron, it has a soft-max layer that approximates a max function by clamping the output to [0,1] range as shown in Figure 3(a).

function [L, dLdy] = Loss_cross_entropy_softmax(x, y)

Input: x∈R^m is the input to the soft-max, and y∈ 0,1^m is the ground truth label.

Output: L∈R is the loss, and dLdy is the loss derivative with respect to x.

Description: Loss_cross_entropy_softmax measure cross-entropy between two distributionswhere ye_i is the soft-max output that approximates the max operation by clamping x to [0,1] range:

,

where x_i is the i^th element of x.

function [w, b] = TrainSLP(mini_batch_x, mini_batch_y)

Output: w∈R^10×196 and b∈R^10×1 are the trained weights and bias of a single-layer perceptron.

Description: You will use the following functions to train a single-layer perceptron using a stochastic gradient descent method: FC, FC_backward, Loss_cross_entropy_softmax

Through training, you are expected to see reduction of loss as shown in Figure 3(b). As a result of training, the network should produce more than 85% of accuracy on the testing data (Figure 3(c)).

4             Multi-layer Perceptron

Figure 4: You will implement a multi-layer perceptron that produces accuracy more than 90% on testing data.

You will implement a multi-layer perceptron with a single hidden layer using a stochastic gradient descent method. We provide main_mlp. The hidden layer is composed of 30 units as shown in Figure 4(a).

function [y] = ReLu(x)

Input: x is a general tensor, matrix, and vector.

Output: y is the output of the Rectified Linear Unit (ReLu) with the same input size. Description: ReLu is an activation unit (y_i = max(0,x_i)). In some case, it is possible to use a Leaky ReLu (y) where

function [dLdx] = ReLu_backward(dLdy, x, y)

Input: dLdy∈R^1×z is the loss derivative with respect to the output y ∈R^z where z is the size of input (it can be tensor, matrix, and vector).

Output: dLdx∈R^1×z is the loss derivative with respect to the input x.

function [w1, b1, w2, b2] = TrainMLP(mini_batch_x, mini_batch_y)

Output: w1 ∈R^30×196, b1 ∈R^30×1, w2 ∈R^10×30, b2 ∈R^10×1 are the trained weights and biases of a multi-layer perceptron.

Description: You will use the following functions to train a multi-layer perceptron using a stochastic gradient descent method: FC, FC_backward, ReLu, ReLu_backward, Loss_cross_entropy_softmax. As a result of training, the network should produce more than 90% of accuracy on the testing data (Figure 4(b)).

5             Convolutional Neural Network

Input        Conv (3)        ReLu              Pool (2×2)      Flatten   FC   Soft-max                              Accuracy: 0.947251

(a) CNN                                                                              (b) Confusion

Figure 5: You will implement a convolutional neural network that produces accuracy more than 92% on testing data.

You will implement a convolutional neural network (CNN) using a stochastic gradient descent method. We provide main_cnn. As shown in Figure 4(a), the network is composed of: a single channel input (14×14×1) → Conv layer (3×3 convolution with 3 channel output and stride 1) → ReLu layer → Max-pooling layer (2 × 2 with stride 2) → Flattening layer (147 units) → FC layer (10 units) → Soft-max. function [y] = Conv(x, w_conv, b_conv)

Input: x∈R^H×W×C1 is an input to the convolutional operation, w_conv∈R^H×W×C1×C2 and b_conv∈R^C2 are weights and bias of the convolutional operation.

Output: y∈R^H×W×C2 is the output of the convolutional operation. Note that to get the same size with the input, you may pad zero at the boundary of the input image. Description: This convolutional operation can be simplified using MATLAB built-in function im2col.

function [dLdw, dLdb] = Conv_backward(dLdy, x, w_conv, b_conv, y) Input: dLdy is the loss derivative with respec to y.

Output: dLdw and dLdb are the loss derivatives with respect to convolutional weights and bias w and b, respectively.

Description: This convolutional operation can be simplified using MATLAB built-in function im2col. Note that for the single convolutional layer, is not needed.

function [y] = Pool2x2(x)

Input: x∈R^H×W×C is a general tensor and matrix.

Output: y∈R^H₂ ^×W₂ ^×C is the output of the 2×2 max-pooling operation with stride 2. function [dLdx] = Pool2x2_backward(dLdy, x, y) Input: dLdy is the loss derivative with respect to the output y. Output: dLdx is the loss derivative with respect to the input x.

function [y] = Flattening(x) Input: x∈R^H×W×C is a tensor.

Output: y∈R^HWC is the vectorized tensor (column major).

function [dLdx] = Flattening_backward(dLdy, x, y) Input: dLdy is the loss derivative with respect to the output y. Output: dLdx is the loss derivative with respect to the input x.

function [w_conv, b_conv, w_fc, b_fc] = TrainCNN(mini_batch_x, mini_batch_y) Output: w_conv ∈ R^3×3×1×3, b_conv ∈ R³, w_fc ∈ R^10×147, b_fc ∈ R¹⁴⁷ are the trained weights and biases of the CNN.

Description: You will use the following functions to train a convolutional neural network using a stochastic gradient descent method: Conv, Conv_backward, Pool2x2,

Pool2x2_backward, Flattening, Flattening_backward, FC, FC_backward, ReLu, ReLu_backward, Loss_cross_entropy_softmax. As a result of training, the network should produce more than 92% of accuracy on the testing data (Figure 5(b)).