[SOLVED] CS329 Homework#1

Question 1
Consider the polynomial function
M
y(x, w) = w0 + w1x + w2x + … + wMxM = ∑ wjxj
j=0
Calculate the coefficients w = {wi} that minimize its sum-of-squares error function. Here a suffix i or j denotes the index of a component, whereas (x)i denotes x raised to the power of i.
Question 2
Suppose that we have three colored boxes r(red), b(blue), and g(green).Box r contains 3 apples, 4 oranges, and 3 limes, box bcontains 1 apple, 1 orange, and 0 limes, and box g contains 3 apples, 3 oranges, and 4 limes. If a box is chosen at random with probabilities p(r) = 0.2, p(b) = 0.2, p(g) = 0.6, and a piece of fruit is removed from the box (with equal probability of selecting any of the items in the box), then what is the probability of selecting an apple? If we observe that the selected fruit is in fact an orange, what is the probability that it came from the green box?
Question 3
Given two statistically independent variables x and z, show that the mean and variance of their sum satisfies
E[x + z] = E[x] + E[z] var[x + z] = var[x] + var[z]
1
Question 4
In probability theory and statistics, the Poisson distribution, is a discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time or space if these events occur with a known constant rate and independently of the time since the last event. If X is Poisson distributed, i.e. X ∼ Possion(λ), its probability mass function takes the following form:
Xe−λ P(X|λ) = λ
X!
It can be shown that if E(X) = λ. Assume now we have n data points from Possion(λ) : D = {X1, X2,…, Xn}. Show that the sample mean = ∑ Xi is the maximum likelihood estimate(MLE) of λ. If X is exponential distribution and its distribution density function is ffor x > 0 and f(x) = 0 for x ≤ 0. Show that the sample meanXi is the maximum likelihood estimate(MLE) of λ.

Hints. For a single target variable t, the loss is given by
Z Z
E[L] = {y(x) − t}2p(x, t)dxdt
The result is as follows
y x x t x]
(x)
Question 6
(a) We defined the entropy based on a discrete random variable X as
H[X] = −∑ p(xi)lnp(xi)
i
Now consider the case that X is a continuous random variable with the probability density function p(x). The entropy is defined as
Z
H[X] = − p(x)lnp(x)dx
Assume that X follows Gaussian distribution with the mean µ and variance σ, i.e.
p
Please derive its entropy H[X].
(b) Write down the mutual information I(y|x). Then show the following equation
I[x, y] = H[x] − H[x|y] = H[y] − H[y|x]

Hints. You should install matplotlib.pyplot, and read classes PolynomialFeature, LinearRegression, and BayesianRegression in the file.