1.策略网络简介

所谓的策略网络，即建立一个神经网络模型，它可以通过观察环境状态，直接预测出目前最应该执行的策略（policy），执行这个策略可以获得最大的期望收益（包括现在的和未来的reward）。和之前的任务不同，在强化学习中可能没有绝对正确的学习目标，样本的feature和label也不在一一对应。我们的学习目标是期望价值，即当前获得的reward和未来潜在的可获取的reward。所以在策略网络中不只是使用当前的reward作为label，而是使用Discounted Future Reward，即把所有未来奖励一次乘以衰减系数γ。这里的衰减系数是一个略小于但接近1的数，防止没有损耗地积累导致Reward目标发散，同时也代表了对未来奖励的不确定性的估计。

2.Gym

Gym是OpenAI推出的开源的强化学习的环境生成工具。在Gym中有两个核心的概念，一个是Environment，指我们的任务或者问题，另一个就是Agent，即我们编写的策略或者算法。Agent会将执行的Action传给Environment，Environment接受某个Action后，再将结果Observation(即环境状态)和Reward返回给Agent。

安装Gym

sudo pip install gym

这里说几个问题：如果装的是python3版本的话使用pip3，否则装的是python2版本的。
还有一个经常出现的一个问题就是Error: could not create ‘some path’: Permisssion denied。出现这个问题主要的原因就是没有加sudo，其实安装软件会经常碰到Permission denied，就是权限不够，在一般情况下安装软件的话还是习惯性加个sudo，实在需要改权限再用chmod就行了。

3.CartPole的代码实现

#coding:utf-8
import numpy as np
import cPickle as pickle
import tensorflow as tf
#%matplotlib inline
import matplotlib.pyplot as plt
import mathimport gym
env = gym.make('CartPole-v0')env.reset()
random_episodes = 0
reward_sum = 0
while random_episodes < 10:env.render()observation, reward, done, _ = env.step(np.random.randint(0,2))reward_sum += rewardif done:random_episodes += 1print "Reward for this episode was:",reward_sumreward_sum = 0env.reset()# 超参数
H = 50 # number of hidden layer neurons
batch_size = 25 # every how many episodes to do a param update?
learning_rate = 1e-1 # feel free to play with this to train faster or more stably.
gamma = 0.99 # discount factor for rewardD = 4 # input dimensionalitytf.reset_default_graph()# 神经网络的输入环境的状态，并且输出左/右的概率observations = tf.placeholder(tf.float32, [None,D] , name="input_x")
W1 = tf.get_variable("W1", shape=[D, H],initializer=tf.contrib.layers.xavier_initializer())
layer1 = tf.nn.relu(tf.matmul(observations,W1))
W2 = tf.get_variable("W2", shape=[H, 1],initializer=tf.contrib.layers.xavier_initializer())
score = tf.matmul(layer1,W2)
probability = tf.nn.sigmoid(score)# 定义其他部分tvars = tf.trainable_variables()
input_y = tf.placeholder(tf.float32,[None,1], name="input_y")
advantages = tf.placeholder(tf.float32,name="reward_signal")# 定义损失函数
loglik = tf.log(input_y*(input_y - probability) + (1 - input_y)*(input_y + probability))
loss = -tf.reduce_mean(loglik * advantages) 
newGrads = tf.gradients(loss,tvars)# 为了减少奖励函数中的噪声，我们累积一系列的梯度之后才会更新神经网络的参数adam = tf.train.AdamOptimizer(learning_rate=learning_rate) # Our optimizer
W1Grad = tf.placeholder(tf.float32,name="batch_grad1") # Placeholders to send the final gradients through when we update.
W2Grad = tf.placeholder(tf.float32,name="batch_grad2")
batchGrad = [W1Grad,W2Grad]
updateGrads = adam.apply_gradients(zip(batchGrad,tvars))def discount_rewards(r):""" take 1D float array of rewards and compute discounted reward """discounted_r = np.zeros_like(r)running_add = 0for t in reversed(xrange(0, r.size)):running_add = running_add * gamma + r[t]discounted_r[t] = running_addreturn discounted_rxs,hs,dlogps,drs,ys,tfps = [],[],[],[],[],[]
running_reward = None
reward_sum = 0
episode_number = 1
total_episodes = 10000
init = tf.initialize_all_variables()# Launch the graph
with tf.Session() as sess:rendering = Falsesess.run(init)observation = env.reset() # Obtain an initial observation of the environment# Reset the gradient placeholder. We will collect gradients in # gradBuffer until we are ready to update our policy network. gradBuffer = sess.run(tvars)for ix,grad in enumerate(gradBuffer):gradBuffer[ix] = grad * 0while episode_number <= total_episodes:# Rendering the environment slows things down, # so let's only look at it once our agent is doing a good job.if reward_sum/batch_size > 100 or rendering == True : env.render()rendering = True# Make sure the observation is in a shape the network can handle.x = np.reshape(observation,[1,D])# Run the policy network and get an action to take. tfprob = sess.run(probability,feed_dict={observations: x})action = 1 if np.random.uniform() < tfprob else 0xs.append(x) # observationy = 1 if action == 0 else 0 # a "fake label"ys.append(y)# step the environment and get new measurementsobservation, reward, done, info = env.step(action)reward_sum += rewarddrs.append(reward) # record reward (has to be done after we call step() to get reward for previous action)# 批量更新if done: episode_number += 1# stack together all inputs, hidden states, action gradients, and rewards for this episodeepx = np.vstack(xs)epy = np.vstack(ys)epr = np.vstack(drs)tfp = tfpsxs,hs,dlogps,drs,ys,tfps = [],[],[],[],[],[] # reset array memory# compute the discounted reward backwards through timediscounted_epr = discount_rewards(epr)# size the rewards to be unit normal (helps control the gradient estimator variance)discounted_epr -= np.mean(discounted_epr)discounted_epr /= np.std(discounted_epr)# Get the gradient for this episode, and save it in the gradBuffertGrad = sess.run(newGrads,feed_dict={observations: epx, input_y: epy, advantages: discounted_epr})for ix,grad in enumerate(tGrad):gradBuffer[ix] += grad# If we have completed enough episodes, then update the policy network with our gradients.if episode_number % batch_size == 0: sess.run(updateGrads,feed_dict={W1Grad: gradBuffer[0],W2Grad:gradBuffer[1]})for ix,grad in enumerate(gradBuffer):gradBuffer[ix] = grad * 0# Give a summary of how well our network is doing for each batch of episodes.running_reward = reward_sum if running_reward is None else running_reward * 0.99 + reward_sum * 0.01print 'Average reward for episode %f.  Total average reward %f.' % (reward_sum/batch_size, running_reward/batch_size)if reward_sum/batch_size >= 200: print "Task solved in",episode_number,'episodes!'breakreward_sum = 0observation = env.reset()print episode_number,'Episodes completed.'

这里我注释掉了一行代码%matplotlib inline，这行代码是魔法命令。如果直接在终端里运行会报错UsageError: Invalid GUI request ‘inline’, valid ones are [‘qt4’, ‘glut’, ……(主要都是一些编译器)]。报错是因为没有合适的GUI，魔法命令是要在ipython中使用的。上面语句的作用是在Ipython中显示图片内嵌在notebook中，这样可以直接在程序运行结束后显示图片。O(∩_∩)O