[SOLVED] Machine Learning Homework 3

0. I. Pen-and-paper [12v]
   Consider the problem of learning a regression model from 5 univariate observations

   ((0.8), (1), (1.2), (1.4), (1.6)) with targets (24,20,10,13,12).

   1. 1)  [5v] Consider the basis function, 𝜙𝑗(𝑥) = 𝑥𝑗, for performing a 3-order polynomial regression,

      3

      𝑧̂(𝑥,𝐰)=∑𝑤𝜙(𝑥)=𝑤 +𝑤𝑥+𝑤𝑥2+𝑤𝑥3. 𝑗𝑗0123

      𝑗=0
      Learn the Ridge regression (𝑙2 regularization) on the transformed data space using the closed

      form solution with 𝜆 = 2.

      Hint: use numpy matrix operations (e.g., linalg.pinv for inverse) to validate your calculus.

   2. 2)  [1v] Compute the training RMSE for the learnt regression model.
   3. 3)  [6v] Consider a multi-layer perceptron characterized by one hidden layer with 2 nodes. Using the activation function 𝑓(𝑥) = 𝑒0.1𝑥 on all units, all weights initialized as 1 (including biases), and the half squared error loss, perform one batch gradient descent update (with learning rate 𝜂 = 0.1) for the first three observations (0.8), (1) and (1.2).

      II. Programming and critical analysis [8v]

   Consider the following three regressors applied on kin8nm.arff data (available at the webpage):

   • −  linear regression with Ridge regularization term of 0.1
   • −  twoMLPs–𝑀𝐿𝑃 and𝑀𝐿𝑃 –eachwithtwohiddenlayersofsize10,hyperbolictangent 12

     function as the activation function of all nodes, a maximum of 500 iterations, and a fixed seed(random_state=0).𝑀𝐿𝑃 shouldbeparameterizedwithearlystoppingwhile𝑀𝐿𝑃

12

should not consider early stopping. Remaining parameters (e.g., loss function, batch size, regularization term, solver) should be set as default.

Using a 70-30 training-test split with a fixed seed (random_state=0):
4) [4v] Compute the MAE of the three regressors: linear regression, 𝑀𝐿𝑃 and 𝑀𝐿𝑃 .

5. 5)  [1.5v] Plot the residues (in absolute value) using two visualizations: boxplots and histograms. Hint: consider using boxplot and hist functions from matplotlib.pyplot to this end
6. 6)  [1v] How many iterations were required for 𝑀𝐿𝑃 and 𝑀𝐿𝑃 to converge? 12
7. 7)  [1.5v] What can be motivating the unexpected differences on the number of iterations? Hypothesize one reason underlying the observed performance differences between the MLPs.

END