[SOLVED] CS5446 Assignment 3

Problem 1: Q-Learning with Continuous State

Consider a system with a single continuous state variable x and actions a1 and a2. An agent can observe the value of the state variable as well as the reward in the observed state. Assume a discount factor γ = 0.9.

(a) AssumethatfunctionapproximationisusedwithQ(x,a1)=w0,1+w1,1x+w2,1×2 and Q(x, a2) = w0,2 + w1,2x + w2,2×2. Give the Q-learning update equations.

Q(x, a) ← Q(x, a) + α[R(x) + 0.9 max Q(x′, a′) − Q(x, a)], a′

where a′ ∈ a1, a2, α indicates learning rate and R is the reward.

(b) Assume that wi,j = 1 for all i, j. The following transition is observed: x = 0.5, observed reward r = 10, action a1, next state x = 1. What are the updated values of the parameters assuming a learning rate of 0.5?

wi,1 ←wi,1 +α[r+0.9 max Q(x′,a′)−Q(x,a1)]∂Q(x,a1),wherei=0,1,2; a′ ∈a1 ,a2 ∂ wi,1

wi,1 ← 1+0.5[10+0.9(1+1+1)−(1+0.5+0.25)]∂w0,1+0.5w1,1+0.25w2,1 , where i = 0, 1, 2; ∂ wi,1

w0,1 ← 1+5.475∗1 = 6.475;
w1,1 ← 1 + 5.475 ∗ 0.5 = 3.7375; w2,1 ← 1 + 5.475 ∗ 0.25 = 2.36875;

Problem 2: Policy Gradient with Continuous State

Assume that Q-function with function approximation is used together with the softmax function to form a policy πθ(s,a) = eQθ(s,a)/􏰀a′ eQθ(s,a′) . Assume that there are two actions with Q(x, a1) = w0,1 + w1,1x + w2,1×2 and Q(x, a2) = w0,2 + w1,2x + w2,2×2 for a real valued variable x.

1. (a)  Give the update equations for the REINFORCE algorithm. Assume that the the return at the current step is G and the action taken is a1.wi,j ← wi,j + α ∗ G∇wi,j ln πwi,j (x, a1),
   where i = 0, 1, 2, j = 1, 2, and α indicates learning rate.
2. (b)  Assume that wi,j = 1 for all i, j and return G = 5 is received. What are the updated values of the parameters assuming x = 0.5 and a learning rate of 0.5?wi,j ←wi,j +α∗G∇wi,j lnπwi,j(x,a1),wherei=0,1,2andj=1,2;

w w

0,1 1,1

← 1 + 0.5 ∗ 5∇ ln w0,1

ew0,1+0.5+0.25
ew0,1 +0.5+0.25 +e1+0.5+0.25

= 1 + 2.5 ∗

e = 1 + 1.25 = 2.25 e+e

← 1+0.5∗5∇

w1,1

ln

e1+0.5w1,1+0.25 e1+0.5w1,1 +0.25 +e1+0.5+0.25

= 1+2.5∗

0.5e0.5 = 1+0.625 = 1.625 e0.5 +e0.5

1

CS 5446:

AI Planning and Decision Making

w

w

w

2,1 0,2

= 1 + 2.5 ∗
=1+2.5∗ −e =1−1.25=0.25

← 1 + 0.5 ∗ 5∇ ln w2,1

←1+0.5∗5∇ ln w0,2

e1+0.5+0.25w2,1 e1+0.5+0.25w2,1 +e1+0.5+0.25

e1+0.5+0.25 e1+0.5+0.25 +ew0,2 +0.5+0.25

0.25e0.25 = 1.3125 e0.25 +e0.25

1,2 2,2

← 1+0.5∗5∇ w1,2

ln

e1+0.5+0.25 e1+0.5+0.25 +e1+0.5w1,2 +0.25

e+e
= 1+2.5∗ −0.5e0.5 = 1−0.625 = 0.325

w ← 1 + 0.5 ∗ 5∇

w2,2

ln e1+0.5+0.25 e1+0.5+0.25 +e1+0.5+0.25w2,2

= 1 + 2.5 ∗ −0.25e0.25 = 0.6875 e0.25 +e0.25

e0.5 +e0.5

2