[SOLVED] Machine Learning 2 Week 10-DL2

Exercise Sheet 10

Exercise 1: Mixture Density Networks (20 + 10 P)

In this exercise, we prove some of the results from the paper Mixture Density Networks by Bishop (1994). The mixture density network is given by

with the mixture elements

m
p(t|x) = 􏰄 αi(x)φi(t|x)

i=1

1 exp 􏰁 − ∥t − μi(x)∥2 􏰂. (2π)c/2σi(x)c 2σi(x)2

φi(t|x) =
The contribution to the error function of one data point q is given by

We also define the posterior distribution

i=1

αi φi πi(x,t) = 􏰃mj=1 αjφj

m
Eq =−log􏰗􏰄αi(xq)φi(tq|xq)􏰘

which is obtained using the Bayes theorem.
(a) Compute the gradient of the error Eq w.r.t. the mixture parameters, i.e. show that

∂Eq πi (i) ∂α =−α

ii

∂Eq 􏰁μik −tk􏰂 (ii)∂μ =πi σ2
ik i

(b) We now assume that the neural network produces the mixture coefficients as: exp(ziα )

where zα denotes the outputs of the neural network (after the last linear layer) associated to these mixture coefficients. Compute using the chain rule for derivatives (i.e. by reusing some of the results in the first part of this exercise) the derivative ∂Eq/∂ziα.

Exercise 2: Conditional RBM (20 + 10 P)

The conditional restricted Boltzmann machine is a system of binary variable comprising inputs x ∈ {0, 1}d, outputs y ∈ {0,1}c, and hidden units h ∈ {0,1}K. It associates to each configuration of these binary variables the energy:

E(x,y,h) = −x⊤Wh − y⊤Uh and the probability associated to each configuration is then given as:

p(x, y, h) = Z1 exp(−E(x, y, h)) where Z is a normalization constant that makes probabilities sum to one.

αi = 􏰃Mj=1 exp(zjα)

(a) Let sigm(t) = exp(t)/(1 + exp(t)) be the sigmoid function. Show that (i) p(hk = 1 | x, y) = sigm􏰍x⊤W:,k + y⊤U:,k􏰎

(ii) p(yj = 1|h,x) = sigm􏰍U⊤ h􏰎 j,:

(b) Show that

where

p(x, y) = Z1 exp(−F (x, y))

K F(x,y)=−􏰄log􏰍1+exp􏰍x⊤W:,k +y⊤U:,k􏰎􏰎

k=1
is the free energy and where Z is again a normalization constant.

Exercise 3: Programming (40 P)

Download the programming files on ISIS and follow the instructions.

In [5]: def pca(z,x,d):

# ——————————- # TODO: replace by your code
# ——————————- import solution

                  y = solution.pca(z,x,d)

                  # -------------------------------

return y

The PCA-based inpainting technique is tested below on 10 test points for which a patch is missing. We observe that the patch-like perturbation is less severe when d is low, but the reconstructed part of the digit appears blurry. Conversely, if setting d high, more details become available, but the missing pattern appears more prominent.

In [6]: Xn,m = removepatch(Xt[:10])

              utils.vis10(pca(Xn,Xr,10)*m+Xn*(1-m))
              utils.vis10(pca(Xn,Xr,60)*m+Xn*(1-m))
              utils.vis10(pca(Xn,Xr,360)*m+Xn*(1-m))

Energy-Based Learning (20 P)

We now consider the energy-based learning framework where we learn an energy function to discriminate between correct and incorrect reconstructions.

In [7]: torch.manual_seed(0) enet = nn.Sequential(

                  nn.Linear(784,256),nn.Hardtanh(),
                  nn.Linear(256,256),nn.Hardtanh(),
                  nn.Linear(256,1),

)

To be able to generate good contrastive examples (i.e. incorrect reconstructions that are still plausible enough to confuse the energy- based model and for which meaningful gradient signal can be extracted), we consider a generator network that takes as input the incomplete images.

In [8]: gnet = nn.Sequential( nn.Linear(784,256),nn.Hardtanh(),

                  nn.Linear(256,256),nn.Hardtanh(),

                  nn.Linear(256,784),nn.Hardtanh()
              )

The whole architecture is depicted in the diagram below:

In [9]: import torch.optim as optim N = 10000

mb = 100
optimizer = optim.SGD(list(enet.parameters())+list(gnet.parameters()), lr=0.05) for epoch in numpy.arange(100):

for i in range(N//mb):

                      optimizer.zero_grad()

                      # Take a minibatch and train it

                      x   = Xr[mb*i:mb*(i+1)].data*1.0
                      z,m = removepatch(x)

                      # Build the forward pass from the input until the loss function

# ——————————- # TODO: replace by your code
# ——————————- import solution

                      err = solution.err(x,m,z,gnet,enet)

                      # -------------------------------

                      # Compute the gradient and perform one step of gradient descent

                      err.backward()
                      optimizer.step()

if epoch%10==0: print(epoch,err)

              /home/gregoire/.local/lib/python3.8/site-packages/torch/autograd/__init__.py:130: Use
              rWarning: CUDA initialization: The NVIDIA driver on your system is too old (found ver
              sion 10010). Please update your GPU driver by downloading and installing a new versio
              n from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: http
              s://pytorch.org to install a PyTorch version that has been compiled with your version
              of the CUDA driver. (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:10
              0.)

                Variable._execution_engine.run_backward(

              0 tensor(0.6744, grad_fn=<MeanBackward0>)
              10 tensor(0.1035, grad_fn=<MeanBackward0>)
              20 tensor(0.2146, grad_fn=<MeanBackward0>)
              30 tensor(0.3614, grad_fn=<MeanBackward0>)
              40 tensor(0.3134, grad_fn=<MeanBackward0>)
              50 tensor(0.3515, grad_fn=<MeanBackward0>)
              60 tensor(0.4389, grad_fn=<MeanBackward0>)
              70 tensor(0.3787, grad_fn=<MeanBackward0>)
              80 tensor(0.4541, grad_fn=<MeanBackward0>)
              90 tensor(0.4365, grad_fn=<MeanBackward0>)

After optimizing for a sufficient number of epochs, the solution has ideally come close to some nash equilibrium where both the generator and energy-based model perform well. In particular, the generator should generate examples that look similar to the true examples. The code below plots the incomplete digits and the reconstruction obtained by the generator network.

In [10]: x = Xt[:10]
z,m = removepatch(x)

              utils.vis10(z)
              utils.vis10(gnet(z)*m+z*(1-m))