DATA303 Assignment 4-Inference and Statistical Learning for Binary Response Models Solved

When data are collected through surveys involving human subjects, those who design the survey need to think carefully about factors that might influence whether respondents refuse to provide specific information or are unable to do so, a problem known as “non-response”. Respondents are less likely to provide responses to questions on sensitive subjects. One such sensitive subject is income with Juster et al. (2006) estimating that roughly one-third of questions related to income result in non-response, although this is highly variable from survey to survey. In the United States, Lillard, Smith, and Welch (1986) reported non-response for total income of 2.5% in the 1940 Current Population Survey (CPS) and then a steady increase to 26.6% for the 1982 CPS. This high non-response rate has seemingly stabilized or decreased around the turn of the century with Moore, Stinson, and Welniak (2000) reporting a non-response rate for income questions of roughly 25% for the 1996 CPS and Dixon (2005) showing non-response for income questions to have dropped to 14.2% by the 2002-2003 CPS.

In the context of sub-Saharan African countries, Argent (2009) reported non-response according to a variety of income categories in the National Income Dynamics Study in South Africa with non-response rates ranging from 2.3% to 52.4%. In Mozambique, Fonseca (2014) reported a non-response rate of 39.5% for income questions for household surveys administered to 1,710 households across 68 communities as part of the WASHCost programme. Although estimates were not provided, Fonseca reported non-response rates to be even higher for income questions for similar surveys administered to households in Ghana as part of WASHCost.

Although the reasons for non-response may be varied, non-response to income-related questions is generally believed to be related to income of the respondent and so not missing at random. Lillard, Smith, and Welch (1986) found non-response to income-related questions to increase with income of the respondent. Biewen (2001), on the other hand, found non-response to income-related questions to be highest for those in the tails of the income distribution. In his examination of the German Socio-Economic Panel (GSOEP) study, Schräpler (2004) looked at refusals and responses of “don’t know” to income-related questions and found that refusals were significantly higher with those reporting vocational positions classified as “high” (e.g., executives, civil servants) while responses of “don’t know” were significantly higher with those reporting vocational positions classified as “low” (e.g., unskilled workers). As there is likely to be a fairly strong relationship between vocation classification and income, this result would appear to be in line with the findings of Biewen (2001). And Argent (2009) noted that there “is a general consensus that refusals to income questions are unlikely to be random with respect to income, with those of very high and very low incomes being less likely

to respond,” also in agreement with Biewen (2001).

In this assignment, we consider income data collected as part of a study carried out in three towns in northern Mozambique. This study sought to understand how much people would be willing to pay for water piped to their premises and factors that may influence how much they would be willing to pay. At the end of the survey, participants were asked a number of income-related questions, and our focus will be on factors that are associated with whether respondents provide a numeric value for their total income. A subset of variables collected as part of this study is contained in the dataset WTP.csv, and a list of the variables is presented in the table on the next page. Like many social surveys, there are a variety of special codes used in this dataset to reflect different types of missing data, and these are as follows:

Code Description

-1 Respondent refused to answer the question.

9998. 9998  The question is not applicable to the respondent. This is most commonly due to a

      response to a previous question.

9999. 9999  The respondent specifies that they do not know.

NA A response was not recorded.

2

Variable

HH
DAY
TOWN
YEARS
SEX
AGE
STATUS

EDUC

DISABLED

Description

Household identifier for a given enumeration area (1-15) Day of interview
Town (0 = “Nampula”, 1 = “Liupo”, 2 = “Ribaue”) Years household has lived in <TOWN>

Sex of the respondent (0 = “Male”, 1 = “Female”)
Age of the respondent (in years)
Marital status of the respondent (0 = “Single”, 1 = “Married”, 2 = “Marital union”, 3 = “Divorced”, 4 = “Separated”, 5 = “Widowed”)
Education level of the respondent (0 = “None”, 1 = “Primary of the 1st degree”, 2 =

“Primary of the 2nd degree”, 3 = “Secondary of the 1st degree”, 4 = “Secondary of the 2nd degree”, 5 = “Higher level”)
Disability status of the respondent (0 = “None”, 1 = “Physical”, 2 = “Sight/visual”, 3 = “Other sensory”, 4 = “Mental”)

HEAD
SEX_HH
AGE_HH
STATUS_HH
EDUC_HH
DISABLED_HH
AGE_HH_SPOUSE
EDUC_HH_SPOUSE
DISABLED_HH_SPOUSEDisability status of the spouse of the head of the household

HH_SIZE
N_ADULTS
PRIMARY_WS

SUFFICIENT_WATER
TOTAL_TIME

PAY_WATER
TOTAL_COST

ELECTRIC
TOTAL_INCOME
TIME_LENGTH

Number of persons regularly living in the household
Number of adults regularly living in the household
Primary water source used by household (1 = “Tap in the house”, 2 = “Tap in the yard”, 3 = “Public tap”, 4 = “Tap of a neighbour”, 5 = “Public borehole”, 6 = “Well”, 7 = “Protected spring”, 8 = “Unprotected spring”, 9 = “River, lake, or stream”). These are considered to be hierarchical with 1 considered to be the best type of water source and 9 the worst
Does the household have sufficient access to water for its daily needs? (0 = “No”, 1 =

“Yes”)
Total time required to travel to water source, queue for water and return home when collecting water.
Does the household pay for water? (0 = “No”, 1 = “Yes”)
Average amount (in Mozambican meticals) the household spends each month on water-related costs (including the cost of water, water treatment, transportation of water to the home, etc.)
Is the household connected to the electrical grid? (0 = “No”, 1 = “Yes”)
Average total monthly income of the household (in Mozambican meticals)
How long did the survey take to complete (in minutes)?

Is the respondent the head of the household? (0 = “No”, 1 = “Yes”) Sex of the head of the household (0 = “Male”, 1 = “Female”)
Age of the head of the household
Marital status of the head of the household

Education level of the head of the household
Disability status of the head of the household
Age of the spouse of the head of the household
Education level of the spouse of the head of the household

The data are available in the file WTP.csv, which can be read into R using the code below but with the path changed to point to the location of the file on your computer.

# Read in the Mozambican willingness to pay dataset.

wtp <- read.csv("WTP.csv")

3

Assignment Questions

1. Data pre-processing: (8 marks)

1. (2 marks) The variable TOTAL_INCOME records the numeric value for total income for respondents who could or were willing to provide this information Use this variable to add a new variable INCOME_NONRESPONSE to the data frame wtp. The new variable INCOME_NONRESPONSE should be a binary variable indicating whether the person did not provide a numeric total income data (0 =

   “Provided numeric income data”, 1 = “Did not provide numeric income data”). Show your code to produce this new variable as well as a table of the frequency of outcomes of 0 and 1.

2. (2 marks) We will restrict our focus to a subset of demographic variables and variables that are generally associated with income (and so can be considered as proxies for income). In particular, we will consider a reduced dataset consisting only of the following nine variables:

TOWN SEX AGE EDUC HEAD PAY_WATER ELECTRIC TIME_LENGTH INCOME_NONRESPONSE

Create a new data frame wtp.reduced that consists of only these variables. Show your code to produce this new data frame.

c. (2 marks) Now create a new data frame called wtp.complete, which only keeps respon- dents/observations from wtp.reduced that have no missing data. Show your code to produce this new data frame. (Note: Pay close attention to special codes for missing values.) In total, what proportion (to 3dp) of respondents/observations have been removed from the original dataset to produce this final data frame?

d. (2 marks) Which variables contained in wtp.complete are factors? List these variables, and show code to overwrite these variables in the data frame wtp.complete so that they are recognised by R as factors. (Note: DO NOT convert the variable INCOME_NONRESPONSE to a factor and overwrite the original variable.)

2. Inferential analysis: (18 marks)

Now we will focus on how non-response to a question asking for a numeric value for the total average monthly income of the household is related to demographic factors of the respondent and proxies for income.

1. (3 marks) Fit a logistic regression model of income non-response (INCOME_NONRESPONSE) on sex of the respondent (SEX), their age (AGE), highest level of education completed (EDUC), whether the household pays for water (PAY_WATER), and whether the household is connected to the electrical grid (ELECTRIC). For this logistic regression model, calculate the variance inflation factors for predictors (to 3dp) to determine whether or not there is evidence of significant multicollinearity among the predictors in the model. If so, comment on which predictor(s) should be removed, and use this model for subsequent parts of this question.
2. (3 marks) Provide summary output for the logistic regression model specified in part (a). Explain what you can conclude based on Wald tests of coefficients. Provide evidence to support your conclusion.
3. (3 marks) For any significant Wald tests in part (b), provide a precise interpetation of what the estimated coefficient suggests about the “effect” of the predictor on the response, and calculate a corresponding 95% confidence interval (to 3dp) for the estimated “effect”.
4. (3 marks) Fit the model considered in part (a) but additionally include interactions between • sex of the respondent and whether the household pays for water and

4

• sex of the respondent and whether the household is connected to the electrical grid.

Provide summary output for this model. For this model, explain what it means for sex of the respondent to interact with i) whether the household pays for water and ii) whether the household is connected to the electrical grid.

5. (3 marks) Perform an appropriate test to determine if the logistic regression model fit in part (d) provides a significantly better fit than the model that was fit in part (a). Be sure to write out the full form of the logistic regression models fit in parts (a) and (d), clearly explaining what variables represent, and state

   i) the hypotheses of the test,
   ii) the value of the test statistic,

   iii) the distribution of the test statistic, iv) the p-value of the test, and

   v) your conclusion.

6. (3 marks) Finally, for the best model of the two you fit (in parts (a) and (d)), perform a Hosmer-Lemeshow test for g = 5, 10, and 15 groups, and comment on what these suggest about the goodness-of-fit of this model to the income non-response data.

3. Statistical learning: (12 marks)

Now we perform an exploratory analysis to try to identify the best set of predictors in predicting whether a respondent will not report a numeric value for income. Consider as predictors all variables in wtp.complete (other than the outcome of interest, INCOME_NONRESPONSE).

1. (4 marks) Find the optimal models identified by forward and backward selection algorithms. Report the predictors included in these optimal models. If these models are different, highlight how they differ, and explain why forward and backward selection algorithms may not arrive at the same optimal model.
2. (4 marks) Find the optimal models identified by best subset selection using AIC and BIC as selection criteria. Report the predictors included in these optimal models. If these models are different, highlight how they differ, and explain why the criteria of AIC and BIC may lead to different “best” models.
3. (4 marks) Consider all possible combinations of the eight predictor variables for a cross-validation routine to select the optimal model(s) based on maximising area under the receiver operating characteristic curve (AUC). Use 50 repetitions of 10-fold cross-validation. If this model (or these models) differ from those identified as “best” in parts (a) and (b), explain why this may be the case.

Assignment total: 38 marks

References

Argent, J. 2009. “Household Income: Report on NIDS Wave 1.” 3. National Income Dynamics Study, University of Cape Town, Cape Town.

Biewen, M. 2001. “Item Non-Response and Inequality Measurement: Evidence from the German Earnings Distribution.” Allgemeines Statistisches Archiv 85: 409–25.

Dixon, J. 2005. Comparison of Item and Unit Nonresponse in Household Surveys. Bureau of Labor Statistics, Washington, D.C.

Fonseca, C. 2014. “The Death of the Communal Handpump? Rural Water and Sanitation Household Costs in Lower-Income Countries.” PhD thesis, Applied Sciences, Water Sciences, Cranfield University.

Juster, F. T., H. Cao, M. Perry, and M. Couper. 2006. “The Effect of Unfolding Brackets on the Quality of Wealth Data in HRS.” WP 2006-113. Michigan Retirement Research Center, University of Michigan, Ann Arbor.

Lillard, L., J. P. Smith, and F. Welch. 1986. “What Do We Really Know about Wages? The Importance of Nonreporting and Census Imputation.” Journal of Political Economy 94 (3): 489–506.

5

Moore, J. C., L. L. Stinson, and E. J. Welniak. 2000. “Income Measurement Error in Surveys: A Review.” Journal of Official Statistics 16 (4): 331–62.

Schräpler, J. P. 2004. “Respondent Behavior in Panel Studies: A Case Study for Income Nonresponse by Means of the German Socio-Economic Panel (SOEP).” Sociological Methods & Research 33: 118–56.

6