Description
M2 Mathématiques,
TP 2: Expectation-Maximisation algorithm – Importance sampling
Exercise 1: Discrete distributions Let n E N* and X = {x1,.. . , xn} a set of n distinct real numbers. Let (pi)iQ1,nj a sequence of real numbers such that :
Vi E I1, n�, pi > 0 and
�n i=1
pi = 1.
1. Explain how to generate a random variable X having the discrete distribution on X given by (pi)iE�1,n� :
ViEI[1,n, P(X = xi) = pi. 2. Write (in Python, Julia, Matlab, Octave…) the corresponding algorithm. 3. Generate a sequence (Xi)iQ1,N1 of i.i.d. random variables having the same distribution as X for large values of N. Compare the empirical distribution to the theoretical distribution of X. (In Python, you can use the function numpy.histogram).
Exercise 2: Gaussian mixture model and the EM algorithm A Gaussian mixture model (GMM) is useful for modelling data that comes from one of several groups: the groups might be different from each other, but data points within the same group can be well-modelled by a Gaussian distribution. The main issue is to estimate the parameters of the mixture, i.e to find the most likely ones. Moreover, we aim to determine if our sample follow a Gaussian mixture distribution or not.
Let consider a n-sample. For each individual, we observe a random variable Xi and assume there is an unobserved variable Zi for each person which encode the class of Xi. More formally, we consider a mixture of m Gaussian: let (c1,.. . , cm) E Rm +such that �m i=1 �i = 1 and the following hierarchical model:
ViE1,n, VjE1,m, P9(Zi = j) = cj
and
ViE1,n, VjE1,m Xi | 9, {Zi = j} � N (�j� �j) � Unless otherwise stated, we suppose that m is fixed.
1. Identify the parameters, denoted 8, of the model and write down the likelihood of 9 given the outcomes (xi)iQ1,nj of the i.i.d n-sample (Xi)iE�1�n�, i.e the p.d.f
L(x1� � � � � xn; �) =
i=1
f�(xi).
