[SOLVED] MATH5473 Homework 3- MLE and James-Stein Estimator

1. Maximum Likelihood Method: consider n random samples from a multivariate normal distribution, X_i ∈ R^p ∼ N(µ,Σ) with i = 1,…,n. (a) Show the log-likelihood function

trace(Σ

where, and some constant C does not depend on µ and

Σ;

• Show that f(X) = trace(AX⁻¹) with 0 has a first-order approximation,

f(X + ∆) ≈ f(X) − trace(X⁻¹A⁰X⁻¹∆)

hence formally df(X)/dX = −X⁻¹AX⁻¹ (note (I + X)⁻¹ ≈ I − X);

• Show that g(X) = logdet(X) with 0 has a first-order approximation,

g(X + ∆) ≈ g(X) + trace(X⁻¹∆)

hence dg(X)/dX = X⁻¹ (note: consider eigenvalues of X^−1/2∆X^−1/2);

• Use these formal derivatives with respect to positive semi-definite matrix variables toshow that the maximum likelihood estimator of Σ is

.

A reference for (b) and (c) can be found in Convex Optimization, by Boyd and Vandenbergh, examples in Appendix A.4.1 and A.4.3:

https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf 2. Shrinkage: Suppose y ∼ N(µ,I_p).

1

Homework 3. MLE and James-Stein Estimator                                                                                                                    2

• Consider the Ridge regression

.

Show that the solution is given by

.

Compute the risk (mean square error) of this estimator. The risk of MLE is given when

C = I.

• Consider the LASSO problem,

.

Show that the solution is given by Soft-Thresholding

µˆ^soft_i            = µ_soft(y_i;λ) := sign(y_i)(|y_i| − λ)₊.

√

For the choice λ =         2logp, show that the risk is bounded by

p

Ekµˆ^soft(y) − µk² ≤ 1 + (2logp + 1)^Xmin(µ²_i,1).

i=1

Under what conditions on µ, such a risk is smaller than that of MLE? Note: see Gaussian Estimation by Iain Johnstone, Lemma 2.9 and the reasoning before it.

• Consider the l₀ regularization

,

where         = 0). Show that the solution is given by Hard-Thresholding

µˆhardi          = µhard(yi;λ) := yiI(|yi| > λ).

Rewriting ˆµ^hard(y) = (1 − g(y))y, is g(y) weakly differentiable? Why?

• Consider the James-Stein Estimator

Show that the risk is

Ekµˆ^JS(y) − µk² = EU_α(y)

where U_α(y) = p−(2α(p−2)−α²)/kyk². Find the optimal α^∗ = argmin_α U_α(y). Show that for p > 2, the risk of James-Stein Estimator is smaller than that of MLE for all µ ∈ R^p.

Homework 3. MLE and James-Stein Estimator                                                                                                                    3

• In general, an odd monotone unbounded function Θ : R → R defined by Θ_λ(t) with parameter λ ≥ 0 is called shrinkage rule, if it satisfies

[shrinkage] 0 ≤ Θ_λ(|t|) ≤ |t|;

[odd] Θ_λ(−t) = −Θ_λ(t);

[monotone] Θ_λ(t) ≤ Θ_λ(t⁰) for t ≤ t⁰;

[unbounded] lim_t→∞ Θ_λ(t) = ∞.

Which rules above are shrinkage rules?

3. Necessary Condition for Admissibility of Linear Estimators. Consider linear estimator for y ∼ N(µ,σ²I_p) µˆ_C(y) =

Show that ˆµ_C is admissible only if

• C is symmetric;
• 0 ≤ ρ_i(C) ≤ 1 (where ρ_i(C) are eigenvalues of C); (c) ρ_i(C) = 1 for at most two i.

These conditions are satisfied for MLE estimator when p = 1 and p = 2.

Reference: Theorem 2.3 in Gaussian Estimation by Iain Johnstone, http://statweb.stanford.edu/~imj/Book100611.pdf

4. *James Stein Estimator for p = 1,2 and upper bound: If we use SURE to calculate the risk of James Stein Estimator,

it seems that for p = 1 James Stein Estimator should still have lower risk than MLE for any µ. Can you find what will happen for p = 1 and p = 2 cases?

Moreover, can you derive the upper bound for the risk of James-Stein Estimator?

.