Question 1
In maximum likelihood estimation, the estimator is obtained by maximizing the log likelihood function.
However, most of the log likelihood has to be optimized by Newton-Raphson algorithm. In this question, we will
learn to program Newton-Raphson algorithm for a univariate function. Consider the function f(θ) = 3(θ)2−1, 1+(θ)2
θ > 0. Use the Newton-Raphson method to find the maximizer of the function. Implement the algorithms in R and run your code to obtain the maximizer. (Attach your screenshots of the output in R). You can use the online symbolic differentiation calculator to obtain f′(θ) and f′′(θ) (https://www.symbolab.com/solver/second- derivative-calculator).
Question 2
In the following marketing set, we have 9 years with the sales in 10 million euro and the advertising expenditure in million euro.
- a) Based on the 9 observation and perform a ridge regression. Program it with R. Output the ridge regression results at a few different values of λ.
- b) In your ridge regression, when λ increases, what do you observe from the values of the estimated coefficients. Does any of the estimated coefficients shrink to zero like the L1 LASSO regression? Describe the difference between the output of a ridge regression and the output of a lasso regression.
Question 3
a) In this question, we will investigate the problem of a multiple testing. Consider the hypothesis testing of H0 : μ = 0, vs. Ha : μ ̸= 0. Under the null hypothesis, the Z test statistic is a standard normal random variable. We reject the null hypothesis when |Z| is greater than 1.96 at the significance level of 0.05. Write a R program to simulate 1000 Z test statistic from standard normal N(0,1). (you can use the sample codes in our lecture notes).
b) If we perform hypothesis testing using the significance level of 0.05. Among the 1000 test statistics you generated, how many of them are rejected?
c) Apply the Bonferroni method to control the overall type I error rate to be 0.05. Based on your program, how many rejections do you obtain from your simulation?
d) As the Zs are generated from the null hypothesis, we consider these rejections are all false positive discoveries. Please use a short paragraph to summarize the problem we are faing when we perform multiple testings.
Question 4
Consider a data set with the response vector Y = (y1, …, yn)T and the data matrix 1 x1,1 x1,2
. . .
1 xn,2 xn,2
We model the relationship between X and Y using the linear regression model: yi = θ0 +θ1xi1 +θ2xi2 +εi,i = 1,…,n,whereε∼N(0,σ2). Lettheparametervectorbedenotedasθ=(θ0,θ1,θ2)T. Wewishtominimize the sum of weighted squared residuals: SSE = ni=1 wi(yi − (θ0 + θ1xi1 + θ2xi2))2. Derive the formula for the solution of θ which minimizies the weighted sum of squared errors.
Question 5
Analyze the German data set from the site: https://archive.ics.uci.edu/ml/datasets/statlog+(german+credit+ data).
a) Perform the logistic regression on the data set. Build a predictive model using some of the predictors in the model. Please use 900 observations as the training set and use your model to predict the default status of the remaining 100 loans. Choose one of the regression coefficients and interpret the regression coefficient. What is the cutoff value of the probability do you use for your analysis? How many default ones are predicted to be non-default ones (number of false negative)? How many non-default ones are predicted to be default ones (number of false positive). Then you need to improve your model by adding more predictors or adding some higher order terms or interaction terms. Please demonstrate that your new model has lesser errors than your first model in the 100 testing cases.
b) Please investigate how the sensitivity and specificity change with respect to the different cutoff value of probability.
Question 6
In logistic regression, we assume Y = (Y1 , …, Yn )T are a collection of n binary observations. For each Yi , we observe Xi = (Xi1 , Xi2 , Xi3 )T predictors. We assume
log( pi )=XiTθ 1−pi
where θ = (θ1, …θ3)T is the vector of regression coefficients.
a) Formulate the overall log likelihood of the data set l(Y )
b) Derive the first derivative ∂l(Y ) . ∂θ2
c) Derive the second derivative ∂2l(Y ) . ∂θ2∂θ3
d) Suppose θˆ = (0.1, 0.2, 0.3)T , and we have a new observation Xn+1 = (3, 2, 4)T . Predict the probability p of success for this new observation.
