Part 1: Find a least-squares binary classifier for handwritten mnist digit set, i.e. determine if an image 𝑥 is a digit 𝑘 or not digit 𝑘. Use label 𝑦$ = 1 if 𝑥$ is digit 𝑘 and 𝑦$ = −1 otherwise. Find 𝛽, and 𝑣 , such that 𝑦$ ≈ 𝑦+$ = 𝐬𝐢𝐠𝐧 (𝛽2𝑥$ + 𝑣) = 𝐬𝐢𝐠𝐧(𝑦5$)
Note: The real-valued offset 𝑣 in the linear expression 𝛽2𝑥$ + 𝑣 can be “folded” into vector “𝛽”.
In order to do that, think of 𝛽 ,
where 𝑥$’th image is written as a 1-D column vector of length 𝑀 = 289: 𝑥$ = [𝑥$8 𝑥$9 … 𝑥$; 1]2.
Count the number of correctly identified hand-written digits using your classifier. You can use either the same “training” set, or “test” set.
Part 2: Modify the LS setting in Part 1 to include the regularization parameter:
B
A(𝑦5$ − 𝑦$)9 + 𝜆‖𝛽‖9 $C8
Display the image corresponding to 𝛽 for different values of 𝜆. Do you see any change? Count the number of correctly identified hand-written digits using your classifier. Is there any change of that count as you vary 𝜆? How does that compare with result in Part 1?
Part 3: Compute 10 largest eigenvalues and corresponding eigenvectors of covariance matrix for the selected digit 𝑘 of the “train” data set. Display the eigenvectors as 28×28 images.
Part 4: Extend the linear formulation of Least Squares in Part 1 to a non-linear LS:
B
A(𝑦5$ − 𝑦$)9, $C8
where 𝑦5$ = 𝛽2 𝜑(𝑥$) + 𝑣 (instead of 𝑦5$ = 𝛽2 𝑥$ + 𝑣 as in Part 1) and 𝜑(𝑥) being a Logistic function:
𝜑(𝑥) = ,
Where parameter 𝑘 represents the rate of steepness of the logistic curve and 𝑥M corresponds to the midpoint of the sigmoid 𝜑(𝑥) (c.f. figure above with 𝑥M = 0 and 𝑘 = 1).
The logistic function 𝜑(𝑥) replaces the 𝐬𝐢𝐠𝐧(x) used in linear formulation and since the range of 𝜑(𝑥) is [0,1] (vs. [-1,1] for 𝐬𝐢𝐠𝐧), the labels should be: 𝑦$ = 1 if 𝑥$ is digit 𝑘 and 𝑦$ = 0 otherwise (vs. 1 and -1 used in linear case).
You can use ‘least_squares’ function available in scipy.optimize (with ‘lm’ option for
Levenberg-Marquardt algorithm) to find parameters 𝛽. The rate 𝑘 of the logistic function needs to be hand-picked though. Experiment with different values for 𝑘.
Compare with results in Part 1.
Also, experiment with adding a normalization, as in Part 2, penalizing large norms of 𝛽:
∑B$C8(𝑦5$ − 𝑦$)9 + 𝜆‖𝛽‖9, and compare with results in Part 2.
Material for Part 3:
https://www.academia.edu/16125627/Neural_and_statistical_classifierstaxonomy_and_two_case_studies
https://towardsdatascience.com/principal-component-analysis-your-tutorial-and-code9719d3d3f376
