[SOLVED] MachineLearning Homework 5-Gaussian Process & SVM

I. Gaussian Process
In this section, you are going to implement the Gaussian Process and visualize the result.

● Training data
o input.dataisa34x2matrix.Everyrowcorrespondstoa2Ddatapoint

(Xi,Yi).
o Yi = f(Xi) + 𝜖i is a noisy observation, where 𝜖i ~ N(∙|0, β-1). You can use β =

5 in this implementation. ● What you are going to do

o Part1:ApplyGaussianProcessRegressiontopredictthedistributionoff and visualize the result. Please use a rational quadratic kernel to compute similarities between different points.
Details of the visualization:

– Show all training data points.
– Draw a line to represent the mean of f in range [-60,60]. – Mark the 95% confidence interval of f.

(You can use matplotlib.pyplot to visualize the result, e.g. use matplotlib.pyplot.fill_between to mark the 95% confidence interval, or you can use any other package you like.)

o Part2:Optimizethekernelparametersbyminimizingnegativemarginal log-likelihood, and visualize the result again. (You can use scipy.optimize.minimize to optimize the parameters.)

II. SVM on MNIST dataset

Use SVM models to tackle classification on images of hand-written digits (digit class only ranges from 0 to 4, as the figure shown below).

● Training data

III. Report

o X_train.csvisa5000x784matrix.Everyrowcorrespondstoa28x28gray- scale image.

o Y_train.csvisa5000x1matrix,whichrecordstheclassofthetraining samples.

● Testing data
o X_test.csvisa2500x784matrix.Everyrowcorrespondstoa28x28gray-

scale image.
o Y_test.csvisa2500x1matrix,whichrecordstheclassofthetestsamples.

● What you are going to do
o Part1:Usedifferentkernelfunctions(linear,polynomial,andRBFkernels)

and have comparison between their performance.
o Part2:PleaseuseC-SVC(youcanchoosebysettingparametersinthe

function input, C-SVC is soft-margin SVM). Since there are some parameters you need to tune for, please do the grid search for finding parameters of the best performing model. For instance, in C-SVC you have a parameter C, and if you use RBF kernel you have another parameter 𝛾, you can search for a set of (C, 𝛾) which gives you best performance in cross-validation. (There are lots of sources on the internet, just google for it.)

o Part3:Uselinearkernel+RBFkerneltogether(thereforeanewkernel function) and compare its performance with respect to others. You would need to find out how to use a user-defined kernel in libsvm.

• ●  Submit a report in pdf format. The report should be written in English.
• ●  Report format:

  ◼ I. Gaussian Process

  • a. code with detailed explanations (20%)
    • ●  For example, show the formula of rational quadratic kernel and the process you optimize the kernel parameters
    • ●  Note that if you don’t explain your code, you cannot get any points in section 2 and 3 either.
      • ○  Part1 (10%)
      • ○  Part2 (10%)
  • b. experiments settings and results (20%)
    • ●  Show the figures and the hyperparameters we asked you to show
    • ●  Note that if you don’t explain your code in the above section, you

      cannot get any points in this section either.

      • ○  Part1 (10%)
      • ○  Part2 (10%)

IV. Turn in

■ c. observations and discussion (10%)
● Anything you want to discuss, such as comparing the performance

when using different hyperparameters.

◼ II. SVM
■ a. code with detailed explanations (20%)

• ●  Paste the screenshot of your functions with comments and explain your code. For example, show the formula of different kernel functions and the process you search for the kernel parameters, etc.
• ●  Note that if you don’t explain your code, you cannot get any points in section b and c either.
  • ○  Part1 (5%)
  • ○  Part2 (10%)
  • ○  Part3 (5%)

■ b. experiments settings and results (20%)
● Show everything we asked you to show

• ○  Part1 (5%)
• ○  Part2 (10%)
• ○  Part3 (5%)

■ c. observations and discussion (10%)
● Anything you want to discuss, such as trying different user-

defined kernel functions and comparing the performance.

1. Report (.pdf)
2. Source code (e.g. gaussian_process.py, svm.py …)

You should zip source code and report in one file and name it like ML_HW5_yourstudentID_name.zip, e.g. ML_HW5_0856XXX_王小明.zip.

P.S. If the zip file name has format error or the report is not in pdf format, there will be a penalty (-10). Please submit your homework before the deadline, late submission is not allowed.

Packages allowed in this assignment:

You are only allowed to use numpy, scipy.optimize, scipy.spatial.distance, and package for visualizing results. Official introductions can be found online.
Important: scikit-learn is not allowed.