NYCU Homework 5- Gaussian Process & SVM Solved

Description

Homework 5

Gaussian Process & SVM

1. Gaussian Process

In this section, you are going to implement the Gaussian Process and visualize the result.

• Training data o input.data is a 34×2 matrix. Every row corresponds to a 2D data point

(X_i,Y_i).

o Y_i = f(X_i) + 𝜖𝜖_i  is a noisy observation, where 𝜖𝜖_i ~ N(∙|0, β^-1). You can use β = 5 in this implementation.

• What you are going to do o Part1: Apply Gaussian Process Regression to predict the distribution of f and visualize the result. Please use a rational quadratic kernel to compute similarities between different points.

Details of the visualization:

• Show all training data points.
• Draw a line to represent the mean of f in range [-60,60].
• Mark the 95% confidence interval of f.

(You can use matplotlib.pyplot to visualize the result, e.g. use matplotlib.pyplot.fill_between to mark the 95% confidence interval, or you can use any other package you like.)

o Part2: Optimize the kernel parameters by minimizing negative marginal log-likelihood, and visualize the result again. (You can use

scipy.optimize.minimize to optimize the parameters.)

 

1. SVM on MNIST dataset

Use SVM models to tackle classification on images of hand-written digits (digit class only ranges from 0 to 4, as the figure shown below).

 

• Training data
  • csv is a 5000×784 matrix. Every row corresponds to a 28×28 grayscale image.
  • csv is a 5000×1 matrix, which records the class of the training samples.
• Testing data o csv is a 2500×784 matrix. Every row corresponds to a 28×28 grayscale image. o Y_test.csv is a 2500×1 matrix, which records the class of the test samples.
• What you are going to do o Part1: Use different kernel functions (linear, polynomial, and RBF kernels) and have comparison between their performance. o Part2: Please use C-SVC (you can choose by setting parameters in the function input, C-SVC is soft-margin SVM). Since there are some parameters you need to tune for, please do the grid search for finding parameters of the best performing model. For instance, in C-SVC you have a parameter C, and if you use RBF kernel you have another parameter 𝛾𝛾, you can search for a set of (C, 𝛾𝛾) which gives you best performance in cross-validation. (There are lots of sources on the internet, just google for it.)

o Part3: Use linear kernel + RBF kernel together (therefore a new kernel function) and compare its performance with respect to others. You would need to find out how to use a user-defined kernel in libsvm.

 

• Report
  • Submit a report in pdf format. The report should be written in English.
  • Please strictly follow the report format. We will deduct some points according to the situation if you don’t follow it.
  • Please don’t explain the code line by line. You need to explain it clearly and well structured. For example, explain which part you done in the function.
  • Since this homework is mainly graded by report, please spend more time on it. (e.g. well organized) We won’t give you any point it you just finish the code.
  • Report format:

n I. Gaussian Process

■ a. code with detailed explanations (20%)

• For example, show the formula of rational quadratic kernel and the process you optimize the kernel parameters
• Note that if you don’t explain your code, you cannot get any points in section 2 and 3 either.

○ Part1 (10%)

○ Part2 (10%)

■ b. experiments settings and results (20%)

• Show the figures and the hyperparameters we asked you to show
• Note that if you don’t explain your code in the above section, you cannot get any points in this section either.

○ Part1 (10%)

○ Part2 (10%)

■ c. observations and discussion (10%)

• Anything you want to discuss, such as comparing the performance when using different hyperparameters.

n II. SVM

■ a. code with detailed explanations (20%)

• Paste the screenshot of your functions with comments and explain your code. For example, show the formula of different kernel functions and the process you search for the kernel parameters, etc.
• Note that if you don’t explain your code, you cannot get any points in section b and c either.

○ Part1 (5%)

○ Part2 (10%)

○ Part3 (5%)

■ b. experiments settings and results (20%)

• Show everything we asked you to show

○ Part1 (5%)

○ Part2 (10%)

○ Part3 (5%)

■ c. observations and discussion (10%)

• Anything you want to discuss, such as trying different userdefined kernel functions and comparing the performance.