Description

1. We define a new kind of discriminant analysis for a classification problem with a binary response. The classes have prior probabilities π₀ and π₁. Given the class, k, the conditional probability of the inputs X₁,…,X_p is multivariate normal with a class-dependent mean µ_k and covariance matrix σ_kΣ. The matrix Σ is common to both classes and σ_k is a classdependent constant. All parameters, π_k, µ_k, σ_k, for each class, as well as Σ, are set to their Maximum Likelihood estimates.
   •  Provide an equation describing the classifier’s decision boundary or discriminant. What would the boundary look like?
   •  Why might this classifier be preferable to Linear Discriminant Analysis?
   •  Why might this classifier be preferable to Quadratic Discriminant Analysis?
2.  Compare leave-one-out cross validation to 10-fold cross validation, with reference to the bias-variance tradeoff.
3.  A total of n samples were simulated from the following distribution

X₁,X₂,X₃,X₄ ∼ N(0,1) i.i.d.

where f is non-linear. Consider the following regression methods for Y : linear regression with predictors X₁, X₂, X₃, and X₄, and 3-nearest neighbors regression. On the same plot, sketch a plausible learning curve for each method. A learning curve for regression shows the average test MSE as a function of n. Explain your reasoning.

4.  The clusterings below were produced by single-linkage hierarchical clustering and k-means clustering. Determine which one is which and explain your reasoning.
5. We apply the Bootstrap to a dataset with n distinct observations x₁,…,x_n.
   •  What is the probability that the j^th observation is included in a specific bootstrap sample?
   • What is the expected value for the fraction of distinct observations in a specific bootstrap sample (i.e. the number of distinct observations from x₁,…,x_n divided by n). Does this expectation converge as n grows large? Hints: (i) use the probability from part (a), (ii) lim_n→∞(1 − 1/n)ⁿ = e⁻¹.
6.  A group of 33 people were asked to report their happiness on a scale from 0 to 20. We apply a linear model with an intercept to regress happiness onto 2 predictors, the yearly income and the amount of money paid in taxes last year.

The t-statistics for income and taxes have corresponding p-values of 0.14 and 0.52, respectively. The RSS of the model is 30 and the sample variance of the happiness is 1.5.

What would you conclude about the relationship between happiness, income, and tax contributions?

7. (a) Identify which classifier among k-nearest neighbors with k = 15 and logistic regression would be more appropriate for each dataset below. Explain how one might adjust the True Positive rate of each method.

Note: Red circles are negative and blue triangles are positive.

(b)  Each of the ROC curves below corresponds to one of the datasets in part (a). In each case, we applied the optimal classifier among k-nearest neighbors and logistic regression. Match each ROC curve to its corresponding dataset and explain your reasoning.

 

 

Cheat sheet

The sample variance of x₁,…,x_n is:

,

The residual sum of squares for a regression model is:

t-test:

The t-statistic for hypothesis H₀ : β_i = 0 is

βˆi

t =

SE(β^ˆ_i) F-test:

The F-statistic for hypothesis H₀ : β_p−_q+1 = β_p−_q+2 = ··· = β_p = 0 is

,

where RSS₀ is the residual sum of squares for the null model H₀, and RSS is the residual sum of squares for the full model with all predictors. Asymptotically, the F-statistic has the F-distribution with degrees of freedom d₁ = q and d₂ = n − p − 1.

Minimum F-statistic to reject H₀ at a significance level α = 0.01

			d1
		1	2                     3	4
d2	1 10 20	4052.181 10.044 8.096	4999.500   5403.352 7.559               6.552 5.849               4.938	5624.583 5.994 4.431
	30	7.562	5.390               4.510	4.018
	120	6.851	4.787               3.949	3.480

Logistic regression:

Logistic regression assigns to positive if the estimated conditional probability

LDA:

The log-posterior of class k given an input x is:

where C is a constant which does not depend on k.

QDA:

The log-posterior of class k given an input x in QDA is:

where C is a constant which does not depend on k.