目录

问题描述：

实现步骤：

1.环境的部署与实现

2.贪心策略(The epsilon-greedy algorithm)

3.玻尔兹曼勘探(The softmax exploration algorithm)

4.置信上限算法(The upper confidence bound algorithm)

5.汤普森采样算法(The Thompson sampling algorithm)

参考：

---

问题描述：

        多臂老虎机问题(Multi-Armed Bandit Problem)是强化学习的经典问题。MAB实际上是一个台机器，在赌场玩的一种赌博游戏，你拉动手臂(杠杆)并得到一个支付(奖励)基于随机生成的概率分布。

        我们的目标是，随着时间序列，找出哪台机器可以得出最大的累计奖励，即最大化累计奖励

实现步骤：

1.环境的部署与实现

 pip3 install gym_bandits

import  gym
import gym_bandits
import numpy as np
env = gym.make("BanditTenArmedGaussian-v0")print(env.action_space.n)

2.贪心策略(The epsilon-greedy algorithm)

        在贪心策略中，我们要么选择表现最好的臂，要么是随机选择臂

'''initialize all variables'''
#number of rounds
num_rounds = 20000
#count of number of times an arm was pulled
count =np.zeros(10)
#sum of rewards of each arm
sum_rewards = np.zeros(10)#q value is the average reward
Q = np.zeros(10)#define epsilon_greedy function
def epsilon_greedy(epsilon):rand = np.random.random()if rand < epsilon:action = env.action_space.sample()else:action = np.argmax(Q)return action#start pulling arm
for i in range(num_rounds):#select the arm using epsilon greedyarm = epsilon_greedy(0.5)#get the rewardobservation,reward,done,info = env.step(arm)#update the count of that armcount[arm] += 1#sum the rewardsum_rewards[arm] += reward#calculate Q value which is the average rewards of the armQ[arm] = sum_rewards[arm]/count[arm]
print('the optimal arm is {}'.format(np.argmax(Q)))

3.玻尔兹曼勘探(The softmax exploration algorithm)

        在softmax探索中，我们根据玻尔兹曼概率选择臂

import math
import random''' in softmax exploration, we select an arm based on a probability from
the Boltzmann distribution'''
#define the softmax function
def softmax(tau):total = sum(math.exp(val/tau) for val in Q)probs = [math.exp(val/tau) /total for val in Q]threshold = random.random()cumulative_prob = 0.0for i in range(len(probs)):cumulative_prob += probs[i]if (cumulative_prob > threshold):return ireturn np.argmax(probs)#begining
for i in range(num_rounds):#selct the using armarm = softmax(0.5)#get the rewardobservation,reward,done,info = env.step(arm)#update the count of armcount[arm] += 1#sum the rewardssum_rewards[arm] += reward#calculate Q value Q[arm] = sum_rewards[arm]/count[arm]print("the optimal arm is {}".format(np.argmax(Q)))

4.置信上限算法(The upper confidence bound algorithm)

        在此算法中，我们注重于在初期表现很差，但是在后面的回合中，表现不错的臂。置信上限算法也称为乐观面对不确定性

'''
1. Select the action (arm) that has a high sum of average reward and upper
confidence bound
2. Pull the arm and receive a reward
3. Update the arm's reward and confidence bound'''
#define the upper confidence bound function
def UCB(iters):ucb = np.zeros(10)#explore all the armif iters <10:return ielse:for arm in range(10):#calculate upper boundupper_bound = math.sqrt((2*math.log(sum(count))) / count[arm])#add upper bound to the Q valueucb[arm] = Q[arm] + upper_bound#return the arm which has maximum valuereturn (np.argmax(ucb))#begining 
for i in range(num_rounds):#select the arm using UCBarm = UCB(i)#get the rewardobservation,reward,done,info = env.step(arm)#update the countcount[arm] += 1#sum the rewardssum_rewards[arm] += reward#calculate Q valueQ[arm] = sum_rewards[arm] /count[arm]print("the optimal arm is {}".format(np.argmax(Q)))

5.汤普森采样算法(The Thompson sampling algorithm)

        是一种基于先验的概率算法分布。

'''
1. Sample a value from each of the k distributions and use this value as a prior
mean.
2. Select the arm that has the highest prior mean and observes the reward.
3. Use the observed reward to modify the prior distribution.'''
#initialize alpha and beta value
alpha = np.ones(10)
beta  = np.ones(10)#define the thompson_sampling function
def thompson_sampling(alpha,beta):samples = [np.random.beta(alpha[i] +1,beta[i] +1) for i in range(10)]return np.argmax(samples)#begining
for i in range(num_rounds):arm = thompson_sampling(alpha,beta)observation,reward,done,info = env.step(arm)count[arm] += 1sum_rewards[arm] += rewardQ[arm] = sum_rewards[arm] /count[arm]if reward>0:alpha[arm] += 1else:beta[arm] += 1
print('the optimal arm is {}'.format(np.argmax(Q)))

参考：

《Hands-on Reinforcement Learning with Python. Master Reinforcement and Deep Reinforcement Learning using OpenAI Gym and TensorFlow》