[SOLVED] COMP9418 Assignment 2

1             [50 Marks] Expectation Maximisation

Consider a model with continuous observed variables x ∈ R^D and hidden variables t ∈ {0,1}^K and z ∈ R^Q. The hidden variable t is a K-dimensional binary random variable with a 1-of-K representation, where t_k ∈ {0,1} and ^P_k t_k = 1, i.e. exactly one component of t_k is equal to 1 while all others are equal to 0. The prior distribution over t is given by

p(t_k = 1|θ) = π_k,                                                              (1)

where mixing weights  satisfy 0 ≤ π_k ≤ 1 and                 = 1. This can also be

k=1

COMP9418, UNSW Sydney                       Advanced Topics in Statistical Machine Learning, 18s2

Hidden variable z is a Q-dimensional continuous random variable with prior distribution

p(z|θ) = p(z) = N(0,I).                                                         (3)

The conditional likelihood of x given z and t_k = 1 is a Gaussian defined as

p(x|z,t_k = 1,θ) = N(x|W_kz + b_k,Ψ),                                               (4)

where W_k ∈ R^D×Q, b_k ∈ R^D and Ψ ∈ R^D×D is a diagonal covariance matrix. Another way to express this is

K

p(x|z,t,θ) = ^YN(x|W_kz + b_k,Ψ)^tk.                                                 (5)

k=1

Let us collectively denote the set of all observed variables by X  and hidden variables by Z  and T . The joint distribution is denoted by p(Z,T,X|θ), and is governed by the set of model parameters .

In the questions below, unless otherwise stated explicitly, you must show all your working. Omission of details or derivations may yield a reduction in the corresponding marks.

1. [5 marks] Draw the graphical representation for this probabilistic model, making sure to include the parameters θ in the graph. (Non-random variables can be included similarly to random variables, except that circles are not drawn around them).
2. [5 marks] In terms of K,D,Q, give an expression for the number of parameters we are required to estimate under this model.
3. [10 marks] In the E-step of the expectation maximization (em) algorithm, we are required to compute the expected sufficient statistics of the posterior over hidden variables. The posterior responsibility of mixture component k for a data-point n is expressed as

def                                                     old

rnk = p(tnk = 1|xn,θ ) = Ep(tnk|x,θold)[tnk].                                         (6)

The conditional posterior over local hidden factor z_n is a Gaussian with mean m_nk and covariance C_nk,

p(z_n|t_nk = 1,x_n,θ^old) = N(z_n|m_nk,C_nk).                                         (7)

The covariance is given by

C,                                                    (8)

where

def

mnk = Ep_(zn|_tnk=1_,xn,θold₎[zn], and S. (9) i) [5 marks] Give analytical expressions for the responsibilities r_nk and the expected sufficient statistics m_nk and S_nk in terms of the old model parameters θ^old.

1. [1 marks] To de-clutter notation and simplify subsequent analysis, it is helpful to introduce augmented factor loading matrix and hidden factor vector,

W^˜ ,         and        z˜ .                          (10)

Accordingly, give expressions for the sufficient statistics of the conditional posterior on augmented hidden factor vectors,

def E n nk n old ˜ m˜ nk = p(z˜ |t =1,x ,θ )[z˜n], and S

Note you need only express this in terms of m_nk and S_nk.

COMP9418, UNSW Sydney                       Advanced Topics in Statistical Machine Learning, 18s2

• [4 marks] Show that the sufficient statistics of the joint posterior factorise as follows,

.

1. [10 marks] Write down the full expression for the expected complete-data log likelihood (also known as auxiliary function) for this model,

Q(θ,θ^old) ^def= Ep_(Z,T|X,θold₎[logp(Z,T,X|θ)]. (12) e) [20 marks] Optimize the auxiliary function Q w.r.t. model parameters θ to obtain M-step updates. Show all your working and highlight each individual update equation.

2             [50 Marks] Practical Part

See Jupyter notebook comp9418 ass2.ipynb.