COMS4771 HW2 Solved

Description

• [Constrained optimization] Show that the distance from the hyperplane g(x) = wx+w₀ = 0 to a point x_a is |g(x_a)|/kwk by minimizing the squared distance kx − x_ak² subject to the constraint g(x) = 0.
• [A better output Perceptron algorithm guarantee] In class, we saw that when the training sample S is linearly separable with a maximum margin γ > 0, then the Perceptron algorithm run cyclically over S is guaranteed to converge after updates, where R is the radius of the sphere containing the sample points. This does not guarantee however that the hyperplane solution returned by Perceptron, i.e. w_T achieves a margin close to γ.
  • Show an example training dataset S in R² that has margin γ, and an order of updates made by the Perceptron algorithm where the hyperplane solution returned has arbitrarily bad margin on S.
  • Consider the following modification to the perceptron algorithm:

Modified Perceptron Algorithm

Input: training dataset S = (x_i,y_i)_i=1,…,n

Output: learned vector w

• Initialize w₀ := 0,t := 0
• while there exists an example (x,y) ∈ S, such that 2y(w_t x) ≤ γkw_tk
• set w_t+1 := w_t + yx
• set t := t + 1
• return w_t.
• If the Modified Perceptron Algorithm (MPA) terminates after T rounds, what margin guarantee is achieved by the hyperplane w_T returned by MPA? Justify your answer.
• We will now prove step-by-step the mistake bound for the Modified Perceptron Algorithm (MPA) algorithm.
  2. Show that after T rounds Tγ ≤ kw_Tk, and observe that if kw_Tk < 4R²/γ, then T < 4R²/γ².

In what follows, we will assume that kw_Tk ≥ 4R²/γ.

1. Show that for any iteration t when mistake was made, the following holds:

kw_tk² ≤ (kw_t−1k + γ/2)² + R².

• Infer from that that for any iteration t, we have

.

1. Using the previous question, show that for any iteration t such that either kw_t−1k ≥

, we have

1. Show that kw₀k ≤ R ≤ 4R²/γ. Since by assumption we have, conclude that there must exist some largest iteration t₀ such that kw_t0−1k ≤ and. vi. Show that, and finally deduce the mistake bound.

• [Making data linearly separable by feature space mapping] Consider the infinite dimensional feature space mapping

.

(It may be helpful to sketch the function for understanding the mapping and answering the questions below)

• Show that for any n distinct points x₁,…,x_n, there exists a σ > 0 such that the mapping Φ_σ can linearly separate any binary labeling of the n
• Show that one can efficiently compute the dot products in this feature space, by giving an analytical formula for Φ_σ(x) · Φ_σ(x⁰) for arbitrary points x and x⁰.

• [Designing socially aware classifiers] Traditional Machine Learning research focuses on simply improving the accuracy. However, the model with the highest accuracy may be discriminatory and thus may have undesirable social impact that unintentionally hurts minority groups^[1]. To overcome such undesirable impacts, researchers have put lots of effort in the field called Computational Fairness in recent years.

Two central problems of Computational Fairness are: (1) what is an appropriate definition of fairness that works under different settings of interest? (2) How can we achieve the proposed definitions without sacrificing on prediction accuracy?

In this problem, we will focus on some of the ways we can address the first problem. There are two categories of fairness definitions: individual fairness^[2] and group fairness^[3]. Most works in the literature focus on the group fairness. Here we will study some of the most popular group fairness definitions and explore them empirically on a real-world dataset.

Generally, group fairness concerns with ensuring that group-level statistics are same across all groups. A group is usually formed with respect to a feature called the sensitive attribute. Most common sensitive features include: gender, race, age, religion, income-level, etc. Thus, group fairness ensures that statistics across the sensitive attribute (such as across, say, different age groups) remain the same.

For simplicity, we only consider the setting of binary classification with a single sensitive attribute. Unless stated otherwise, we also consider the sensitive attribute to be binary. (Note that the binary assumption is only for convenience and results can be extended to non-binary cases as well.) Notations:

Denote X ∈ R^d, A ∈ {0,1} and Y ∈ {0,1} to be three random variables: non-sensitive features of an instance, the instance’s sensitive feature and the target label of the instance respectively, such that (X,A,Y ) ∼ D. Denote a classifier f : R^d → {0,1} and denote Yˆ := f(X).

For simplicity, we also use the following abbreviations:

P := P(X,A,Y )∼D                   and            Pa := P(X,a,Y )∼D

We will explore the following are three fairness definitions.

• Demographic Parity (DP)

P0[Yˆ = yˆ | Y = y] = P1[Yˆ = yˆ | Y = y]                  ∀y, yˆ   ∈ {0,1}

(equal true positive- and true negative-rates across the sensitive attribute)

• Predictive Parity (PP)

P0[Y = y | Yˆ = yˆ] = P1[Y = y | Yˆ = yˆ]              ∀y, yˆ   ∈ {0,1}

(equal positive predictive- and negative predictive-value across the sensitive attribute)

Part 0: The basics.

• Why is it not enough to just remove the sensitive attribute A from the dataset to achieve fairness as per the definitions above? Explain with a concrete example.

Part 1: Sometimes, people write the same fairness definition in different ways.

• Show that the following two definitions for Demographic Parity is equivalent under our setting:

P0[Yˆ = 1] = P1[Yˆ = 1] ⇐⇒ P[Yˆ = 1] = Pa[Yˆ = 1]              ∀a ∈ {0,1}

Part 2: In this part, we will explore the COMPAS dataset (available in hw2data.zip). The task is to predict two year recidivism. Download the COMPAS dataset from the class’s website. In this dataset, the target label Y is twoyearrecid and the sensitive feature A is race.

• Develop the following classifiers: (1) MLE based classifier, (2) nearest neighbor classifier, and (3) decision tree classifier, for the given dataset.

For MLE classifier, you can model the class conditional densities by a Multivariate Gaussian distribution. For nearest neighbor classifier, you should consider different values of k and the distance metric (e.g. L₁,L₂,L_∞).

(you may use builtin functions to develop your classifier, and you do not need to write the classifier form scratch.)

You do not need to submit any code on Courseworks.

• Which classifier (discussed in previous part) is better for this prediction task? You must justify your answer with appropriate performance graphs demonstrating the superiority of one classifier over the other. Example things to consider: how does the training sample size affects the classification performance.
• To what degree the fairness definitions are satisfied for each of the classifiers you developed? Show your results with appropriate performance graphs.

For each fairness measure, which classifier is the most fair? How would you summarize the difference of these algorithms?

• Choose any one of the three fairness definitions. Describe a real-world scenario where this definition is most reasonable and applicable. What are the potential disadvantage(s) of this fairness definition?

(You are free to reference online and published materials to understand the strengths and weaknesses of each of the fairness definitions. Make sure cite all your resources.)

[1] see e.g. Machine Bias by Angwin et al. for bias in recidivism predication, and Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification by Buolamwini and Gebru for bias in face recognition

[2] see e.g. Fairness Through Awareness by Dwork et al.

[3] see e.g. Equality of Opportunity in Supervised Learning by Hardt et al.