1 Simpson’s Paradox
1.1
We define ”you” as the 1st player, ”friend” as the 2nd player. Therefore,
Wi := the ith player wins
Uj := the player uses the jth machine With maximum likelihood estimation, we have
where ). Hence, ”You” is more likely to win on both machine 1 and 2 than ”friend”.
1.2
where ). Hence, overall ”friend” is more likely to win than ”you”.
1.3
where p(W1|U1) > p(W2|U1), p(W1|U2) > p(W2|U2), but p(W1) < p(W2).
This is because p(W1|U1) > p(W2|U1) > p(W1|U2) > p(W2|U2), but the 1st player(”you”) plays the 2nd machine much more than the 1st one, i.e., p(U1) << p(U2). As a result, your probability of winning is close to p(W1|U2) and becomes smaller than p(W2).
2 Matrix as Operations
2.1
We know that
Hence
W11 = 0.9
W12 = −1.7
WW2122 = 0= 0..79
Hence
2.2
Because tan(α) = 3, suppose α < 180◦, then cos( . Hence
!
Next, we know
for
Hence,
Next, because U rotates counter-clockwise by β degrees,
Because tan( . Hence
!
Therefore,
2.3
Hence det(WTW −λI) = (1.3−λ)(3.7−λ)−0.92 = λ2 −5λ+4 = (λ−1)(λ−4). Therefore, the eigenvalues and eigenvectors of WTW are
From 2.2 we know that WT = UΣV , where U,V are orthogonal and Σ is symmetric. Hence
W = (UΣV )T = V TΣTUT = V −1ΣU−1
Therefore, if we transform the unit circle by W, it is equivalent to rotate the unit cricle by β degrees clockwise (the result is still a unit circle), scaling the y-axis by 2 and keeping x-axis unchanged (result is an ellipse), and rotating the ellipse by α degrees anti-clockwise.
In the end, we will get an inclined ellipse. Moreover, suppose that the semi-major axis of this ellipse is of length a, semi-minor axis is of length b, we have a2 = 4 = λ1,b2 = 1 = λ2; the major axis is in the same direction as u1, whil the minor axis is in the same direction as u2.
This is bacause
Hence, if we transform the unit circle with WTW, it is equivalent to scaling the y-axis by 2 twice, therefore a2 = 22 = 4 = λ1, b1 = 12 = 1 = λ1. The matrix , which is equivalent to mapping the standard coordinate system to the space spanned by u2 and u1, which explains why the long-axis and short-axis are in the direction of u2 and u1.
2.4
The area of the ellipse is 2 · 1 · π = 2π = det(W) · area of a unit circle. Therefore,
the area of the shape transformed by W
= det(W)
the area of the original shape
This is because W can be diagonalized as follows:
where λ1 …λn are eigenvalues of W, u1 …un are its orthonormal eigenvectors. Therefore, transformations of a shape with U and V do not influence the area of it, where as Σ multiplies its area by
|det(W)|. Therefore, transforming a shape with AB will multiply its area by |det(AB)|. On the other hand, transforming a shape with AB is equivalent to transforming it with B, then transforming it with A, which multiplies its area by |det(A)||det(B)|. These two understandings are equivalent, therefore det(AB) = det(A)det(B).
3 Some Practices
3.1
Because
Therefore
3.2
Because X is a discrete random variable, we know that
By Cauchy-Schwarz inequality, we know
!
3.3
Because A, B are PSD matrices, we know
∀z ∈ Rn,zTAz ≥ 0,zTBz ≥ 0
Then for ∀λ ∈ [0,1],
zT(λA + (1 − λ)B)z = X(λA + (1 − λ)B)i,jzizj
i,j
= X(λAi,j + (1 − λ)Bi,j)zizj
i,j
= λXAi,jzizj + (1 − λ)XBi,jzizj
i,j i,j
where λ,1 − λ,Pi,j Ai,jzizj,Pi,j Bi,jzizj ≥ 0, hence LHS≥ 0, λA + (1 − λ)B is also a PSD matrix.
4 Density Estimation of Multivariate Gaussian
4.1
Figure 1: the 1000 points in X sampled from multivariate Gaussian distribution
Below is the image of the 1000 points sample from Gaussian distribution. With numpy, we obtained the
mean and covariance of X, which are 0. and , respec-
tively.
4.2
Figure 2: the histograms of x-coordinates (left) and y-coordinates (right) of X
4.3
From 4.2, we can confidently conjecture that both x and y-coordinates of X follow Gaussian distribution. By calculation with numpy, we got
1. 083303371449156, and σy = 2.2546182318630255.
4.4
After generating 1000 samples ) and 1000 samples ), we plotted a graph of coordinates (x,y) as follows:
Figure 3: the distribution of (x,y) with x ∼ N(µx,σx2), y ∼ N(µy,σy2)
Compared to the graph in 4.1, where the distribution of (x,y) are basically within an inclined ellipse, the distribution in this graph is mostly within an horizontally placed ellipse within (µx,µy) as its center, σx and σy as its semi-major axis and semi-minor axis. This is because x and y are independently sampled in this problem, whereas the x and y-coordinates of X are not indenpendent.
4.5
Figure 4: 30 samples from X, y = 2x + 2, and the projection of the samples on y = 2x + 2
4.6
Figure 5: the histogram of the x-coordinates of the projected samples
From the graph above, we can tell the x-coordinates of the projected points also follow Gaussian distribution. After calculating with numpy, we obtained
5 KNN for Iris flowers classification
5.1
After counting the number of samples in each class, it turns out there are 50 samples in setosa, versicolor, and virginica class.
5.2
After applying KNeighborsClassifier with k = 1 to the data, the accuracy turns out be 100%. However, this result makes no sense because the top-1 closest point to a data point is just itself, which is meaningless for our classification task.
5.3
Figure 6: train and test accuracy with different k’s
According to graph above, it turns out the optimal k is 9, because when k = 9 both train and test accuracy are high, and the gap between them is small.
5.4
After putting the given data (sepal width = 5.0, petal width = 4.1, sepal length = 3.8, petal length = 1.2) into my prediction model (KNN with k = 9), the predicted class turns out to be class 0, which is Setosa.
6 K Means clustering
6.1
From the graph below, we can tell the elbow point appears when k=4. At this point, the samples within the same class have high similarity, while k is not too large, which guarantees the model’s robustness. Therefore k=4 clusters should be used for this data.
Figure 7: the sum of within-class inertia for different k’s
6.2
After rerunning k-means with k = 4, we found out there are 25 samples in each class. The value of inertia is 4844.925817623823.
6.3
Figure 8: the plot of the data mapped to its first two dimensions. The color of each data indicates the class it belongs to.
