[Reinforcement Learning] 02——Exploration and Utilization

1. Explore and exploit

Exploration and exploitation is an important issue in sequential decision-making tasks, mainly the trade-off between choosing the known optimal decision and trying other decisions.

• Use Exploitation to select the known optimal decision
• Exploration: Try other decisions, which may be the optimal decision in the future. Use exploration to make the current strategy continue to move toward the optimal strategy $Pi\to {Pi}^{\ast }$。用公式说明一下：$${\mathcal{E}}_t=\left\{p_t^i\mid i=1,\dots ,n\right\}\stackrel{\text{explore}}{\to }{\mathcal{E}}_{t+1}=\left\{p_t^i|i=1,\dots ,n\right\}\cup \left\{p_e^j|j=1,\dots ,m\right\}$$ $E_t$is the current strategy pool, $E_{t+1}$is the strategy pool after exploration. Through exploration, new strategies ${Pi}_e^j$, then in the new strategy pool, it is necessary to find a strategy with greater value than the original strategy without exploration: $$\exists V^{\star }(\cdot |{\pi }_t^i\sim E_t)\le V^{\star }(\cdot |{\pi }_{t+1}^i\sim E_{t+1}){Pi}_{t+1}^i\sim \{p_e^i\mid i=1,\dots ,m\}$$

2. Explore strategies

• Naive method (Naive Exploration): Add policy noise $ϵ-greedy$；
• Optimistic Initialization: Give a higher initialization value to facilitate exploration;
• Uncertainty Measurement: Explore strategies with higher uncertainty;
• Probability Matching: sampling strategy, selecting the best strategy;
• State Searching: Exploring unexplored strategies (the environment is known).

3. Multi-armed bandit machine

In the multi-armed bandit (MAB) problem, there is a player with $A\; slot\; machine\; with\; K$ levers. Pulling each lever corresponds to a probability distribution$R.$ _ Every time we pull one of the levers, we can get a reward rrfrom the reward probability distribution corresponding to the lever.$r$ . The problem is solved within a limited time (Operation$Within\; T$ times), obtain the greatest rewards through continuous attempts and exploration. Sincethe probability distribution of rewards is unknown, we need to make a trade-off between "exploring the winning probability of the lever" and "selecting the lever with the most winnings based on experience". "What kind of operation strategy can be used to obtain the highest cumulative reward" is the multi-armed bandit problem.

3.1. Formal description

The multi-armed bandit problem can be expressed as a tuple $(A,R)$ , where:

• $A$ action set,$a_i\in A,i=1,2,...,K$
• $R$ is the reward probability distribution,$R(r|a_i)=\mathbb{P}(r|a_i)$
• Assuming that only one lever can be pulled at each time step, the goal of the multi-armed bandit is to maximize TT for a period of timeAccumulated reward within$T$$\max {\sum }_{t=1}^T{r}_t,r_t\sim \mathcal{R}\left(\cdot |a_t\right)$

We know earlier that the probability distribution of the reward is unknown, so $R$ is actually an estimate of the reward probability distribution$\hat{R}(r|a_i)$.

3.2. Estimating expected rewards

There is the following relationship between the expected reward and the number of samples:
$$Q_n(a_i)=\frac{r_1+r_2+\cdots +r_{n-1}}{n-1}$$
But the space complexity of this method is $O\left(n\right)$ , the incremental approach can reduce the complexity to$O(1)$:
$$\begin{array}{rl}{Q}_k& =\frac{1}{k}\sum _{i=1}^k{r}_i\\ & =\frac{1}{k}\left(r_k+\sum _{i=1}^{k-1}{r}_i\right)\\ & =\frac{1}{k}(r_k+(k-1)Q_{k-1})\\ & =\frac{1}{k}(r_k+k{Q}_{k-1}-Q_{k-1})\\ & =Q_{k-1}+\frac{1}{k}[r_k-Q_{k-1}]\end{array}$$

The algorithm flow is as follows.

• 对于$\forall a\in A$ , initialize counter$N(a)=0$ and expected reward estimate$\hat{Q}(a)=0$
• for $t=i\to T$ do
• Based on strategy $\pi$ , perform an action$a_t$
• Get the return $r_t=Bandit(a_t)$
• Update counter $N(a_t)=N\left(a_t\right)+1$
• Update the expected reward estimate: $\hat{Q}(a_t)=\hat{Q}\left(a_t\right)+\frac{1}{N\left(a_t\right)}[r_t-\hat{Q}(a_t)]$
• end for

3.3. Regret function

For each action, define its expected return $Q(a_i)={\mathbb{E}}_{r\sim \mathcal{R}(\cdot |a_i)}\left[r|a_i\right]$。

Therefore, there is at least one pull rod whose expected reward is not less than pulling any other pull rod. We express the optimal expected reward as $Q^{\ast }={\max }_{a_i\in \mathcal{A}}Q(a_i)$

In order to more intuitively and conveniently observe the gap between the expected reward of pulling a lever and the expected reward of the optimal lever, we introduce the concept of regret . Regret is defined as the action of pulling the current lever aaThe expected reward difference between$a$$R(a_i)=Q^{\ast }-Q\left(a_i\right)$

Cumulative regret is operation TTThe total amount of regret accumulated after pulling the lever$T$$p_R={\sum }_{t=1}^{T}R\left(a_t\right)$

The goal of the MAB problem is to maximize the cumulative reward, which is equivalent to minimizing the cumulative regret. $\min p_R=\max {\mathbb{E}}_{a\sim \pi }[{\sum }_{t=1}^{T}Q(a_t^i)]$

• If you keep exploring new strategies: $p_R\propto T\cdot R$ , the accumulated regret will increase linearly and cannot converge.
• If new strategies are not explored: $p_R\propto T\cdot R$ , the accumulated regret will increase linearly

Therefore, it is necessary to consider whether there is a sublinear way to ensure convergence. Here is a method proposed by Lai&Robbinsti:
using $R(a)=Q^{\ast }-Q\left(a\right)$ and feedback function distribution similarity:$D_{KL}(\mathcal{R}(r\mid a)\parallel {\mathcal{R}}^{\star }(r|a))$ to describe:

$$\underset{T\to \infty }{\lim }{p}_R\ge \log T\sum _{a|R(a)>0}\frac{R(a)}{D_{KL}(\mathcal{R}(r\mid a)\parallel {\mathcal{R}}^{\star }(r|a))}$$
The theoretical asymptotic optimal convergence is $O(\log T)$

Feedback function distribution similarity
assumes that there are two feedback functions $f\left(x\right)$ and$g\left(x\right)$ , they are in the interval$[a,$The density functions on$b$$]$$p_f\left(x\right)$ and$p_g\left(x\right)$ . Then Kullback-Leibler divergence can be used to describe the distribution similarity between them, the formula is as follows:

$D_{KL}(p_f||p_g)={\int }_a^b{p}_f(x)\log \frac{p_f(x)}{p_g\left(x\right)}dx$

where,$D_{KL}(p_f||p_g)$ means$f\left(x\right)$ and$The\; distribution\; difference\; of\; g\left(x\right)$ , the smaller the value, the more similar the two distributions are.

Now let’s consider what strategy to adopt $\pi$ , to maximize returns.

4. Greedy strategy and $ϵ-greedy$ strategy

For the greedy strategy, the optimal decision will be chosen every time. Obviously this is a process of exploiting . As can be seen from the previous content, cumulative regret increases linearly. And $ϵ-greedy$ strategy introduces noise$ϵ$ , with sampling probability$1-ϵ$ Exploitation(select the lever with the largest expected reward value in past experience), with sampling probability ϵ$ϵ$ Exploration(randomly select a lever) . The formula is as follows:
$$a_t=\{\begin{array}{llc}\arg {\max }_{a\in \mathcal{A}}\hat{Q}(a),& \text{Sampling probability : 1-}ϵ& \\ \text{Choose randomly}\text{from}A& \text{Sampling probability : }& \end{array}$$
Cumulative regret still increases linearly, but the growth rate is smaller.

Decaying Greedy Strategy
As the number of explorations continues to increase, we estimate the rewards of each action more and more accurately. At this time, we do not need to continue to expend great efforts in exploration. So in $In\; the\; specific\; implementation\; of\; ϵ$ -greedy algorithm, we canlet$ϵ$ decays with time, that is, the probability of exploration will continue to decrease. But note that$ϵ$ will not decay to 0 within a limited number of steps, because the completely greedy algorithm based on limited step observations is still a greedy algorithm with local information, and there is always a fixed gap from the optimal solution.

A possible attenuation strategy: (It is generally difficult to find a suitable attenuation strategy)
$$c\ge 0,d=\underset{a|{\Delta }_a>0}{\min }{D}_a,{ϵ}_t=\min \left\{1,\frac{c|\mathcal{A}|}{d^{2}t}\right\}$$

Different $The\; impact\; of\; ϵ$ strategy on average return and optimal action selection

As can be seen from the above figure, when $ϵ=At\; 0$ , there is no exploration part, only the exploitation part, and the average income basically remains unchanged with the time step; when$ϵ=When\; 0.1\; ,$the exploration part is increased, the income increases greatly at the beginning, and then remains at a higher level, and more optimal strategies can be selected; when$ϵ=When\; 0.01$ , it is between the two.

5. Aggressive initialization

给$Q(a_i)$ a higher initial value, also using a growth update method.

$ϵ-greedy$ and the positive initialization strategy have an impact on the optimal action. It can be seen that the positive initialization method is better than$ϵ-greedy$ can obtain a larger proportion of optimal strategies

• It is a biased estimate, and the influence of bias will decrease as the number of samples increases.
• May get stuck in local minima. (Adjust $)$ _

6. Explicitly consider the value distribution of actions

Based on the distribution of the above three actions, how to choose?

• encourage uncertainty
• The display location is selected based on distribution sampling

7. Confidence bound algorithm on UCB

The greater the uncertainty $Q(a_i)$ , the more valuable it is to explore. . We introduce the uncertainty measure U ( a ) {U}(a)here$U\left(a\right)$ , which decreases as the number of times an action is tried increases. We can use an uncertainty-based strategy to comprehensively consider existing expected reward valuations and uncertainties, with the core question being how to estimate uncertainty.

The upper confidence bound (UCB) algorithm is a classic uncertainty-based strategy algorithm. Its idea uses a very famous mathematical principle: Hoeffding 's inequality.

Hoeffding's inequality is an important inequality in probability theory, which describes the convergence speed of the law of large numbers. Roughly speaking, Hoeffding's inequality is used to estimate the probability distribution of the sum of independent and identically distributed random variables whose bounds are known. It is expressed as follows:

Let $X_1,X_2,...,X_n$Yesnn $n$ independent and identically distributed random variables,$0\le X_i\le 1$ , its empirical expectation is$\begin{array}{r}{\bar{x}}_n=\frac{1}{n}\sum _{j=1}^n{X}_j\end{array}$,可得 P { E [ X ] ≥ x ˉ n + u } ≤ e − 2 n u 2 \mathbb{P}\left\{\mathbb{E}\left[X\right]\geq\bar{x}_n+u\right\}\leq e^{-2nu^2} P{ E[X]≥xˉn​+u}≤e− 2 n u2

Now we apply Hoeffding's inequality to the multi-armed bandit problem. Q ^ t ${\hat{Q}}_t\left(a\right)$ Substitute${\bar{x}}_t$, the parameter u in the inequality $u={\hat{U}}_t\left(a\right)$ represents the uncertainty measure. Given a probability$p=e^{-2N_t(a)U_t(a{)}^{2.}}$ According to the above inequality,$Q_t\left(a\right)<{\hat{Q}}_t\left(a\right)+{\hat{U}}_t\left(a\right)$ At least$1-The\; probability\; of\; p$ exists, if$p$很小，则$Q_t\left(a\right)<{\hat{Q}}_t\left(a\right)+{\hat{U}}_{t}The\; probability\; of\; (a)$ existence will be very high,${\hat{Q}}_t\left(a\right)+{\hat{U}}_t\left(a\right)$ is the upper bound of the expected reward.

At this time, the upper bound algorithm selects the action with the largest expected reward upper bound, that is, $a=\arg {\max }_{a\in \mathcal{A}}\hat{Q}(a)+\hat{U}(a)$。其中${\hat{U}}_t\left(a\right)=\sqrt{\frac{-\log p}{2N_t\left(a\right)}}$.

Therefore, set a probability $After\; p$ , the corresponding uncertainty measure can be calculated. More intuitively, the UCB algorithm first estimates the upper bound of the expected reward of each pull rod${\hat{U}}_t\left(a\right)$ , so that the expected reward for pulling each lever is only a small probability$p$ exceeds this upper bound, and then selects the lever with the largest upper bound of the expected reward, thereby selecting the lever that is most likely to obtain the maximum expected reward.

Or use this representation: $$A_t\doteq \arg \underset{a}{\max }\left[Q_t(a)+c\sqrt{\frac{\ln t}{N_t\left(a\right)}}\right]$$

$ϵ-greedy$和$Comparison\; of\; UCB$ average returns.
It can be seen that, except for the previous steps,$U$ average return ratio after$CB$$ϵ-greedy$ high

8. Thompson sampling algorithm

• Choose actions based on the probability that each action will be optimal

数学表达：$$p(a)=\int \mathbb{I}\left[{\mathbb{E}}_{p(Q(a))}\left[Q(a;i)\right]=\underset{a^{\prime }\in \mathcal{A}}{\max }{\mathbb{E}}_{p(Q(a^{\prime }))}(Q(a^{\prime };i))\right]d\theta$$

Thompson sampling uses sampling, that is, based on the current reward probability distribution of each action$p(Q(a_i))$ perform a round of sampling to obtain a set of reward samples Q ( ai ) Q(a_i)for each tie rod$Q(a_i)$ , and then select the action aawith the largest reward in the sample$a$ . It can be seen that Thompson sampling is a Monte Carlo sampling method that calculates the highest reward probability for all levers.

After understanding the basic idea of ​​​​the Thompson sampling algorithm, we need to solve another problem: how to get each current action $The\; reward\; probability\; distribution\; of\; a$ and updated in the process? In practical situations, we usually useBeta distributionto model the current reward probability distribution of each action. Specifically, if a tie rod is selected$k$ times, where$m_1$The reward is 1, $m_2$The first reward is 0, then the reward compliance parameter of the lever is $(m_1+1,m_2+1)$ Beta distribution.

Beta distribution is a probability distribution with values ​​ranging from 0 to 1. It can be used to describe the probability of random events and is widely used in statistics, machine learning, Bayesian inference and other fields.

The probability density function of the Beta distribution is as follows:

$f(x;a,b)=\frac{1}{B(a,b)}{x}^{\alpha -1}(1-x{)}^{\beta -1}$

where,$x$ represents the value of the random variable,$\alpha$和$\beta$ is the parameter of the distribution,$B(a,\beta )$ is the Beta ratio, the same:

$B(a,b)=\frac{C\left(a\right)C(b}{C(a+b)}$

Among them, $\Gamma$ is the gamma function.

The shape of the Beta distribution is determined by the parameter$\alpha$和$\beta$ determines. When$a=b=When\; 1$ , the Beta distribution degenerates into a uniform distribution; when$a>1$ and$b>When\; 1$ , the Beta distribution has a unimodal, bell-shaped distribution shape; when$a<1$或$b<When\; 1$ , the Beta distribution has the shape of a skewed distribution. The expectation of the Beta distribution is$\frac{}{a+b}$，方差的$\frac{a}{(a+b{)}^2(a+b+1)}$。

Summarize

• Exploration and utilization of Exploitation are an integral part of trial-and-error learning in reinforcement learning ;
• A major difference between the multi-armed bandit problem and reinforcement learning is that its interaction with the environment does not change the environment. That is, the result of each interaction of the multi-armed bandit machine has nothing to do with previous actions, so it can be regarded as stateless reinforcement learning . reinforcement learning)
• The multi-armed bandit is the best environment to study exploration and utilization theory (the asymptotic optimal convergence of the theory is $O(\log T)$）；
• Various exploration and exploitation methods are commonly used in RL, especially in multi-armed bandits.

Image source: https://staticcdn.boyuai.com/comment/upload/PzjhxfGWOkCb4KdXTZDik/502/2020/07/24/4yjXIv48Dtqdn84LEySmD.jpg

code

import numpy as np
import matplotlib.pyplot as plt

class BernoulliBandit:
    """伯努利多臂老虎机,输入K表示拉杆个数"""
    def __init__(self, K):
        # 随机生成K个0～1的数,作为拉动每根拉杆的获奖概率
        self.probs = np.random.uniform(size=K)
        # 获奖概率最大的拉杆
        self.best_idx = np.argmax(self.probs);
        # 最大的获奖概率
        self.best_prob = self.probs[self.best_idx]
        self.K = K

    def step(self, Kth):
        # 当玩家选择了k号拉杆后,根据拉动该老虎机的k号拉杆获得奖励的概率返回1（获奖）或0（未获奖）
        if np.random.rand() < self.probs[Kth]:
            return 1
        else:
            return 0

class ProblemSolver:
    """多臂老虎机算法基本框架 """
    def __init__(self, bandit):
        self.bandit = bandit
        # 每根拉杆的尝试次数
        self.counts = np.zeros(self.bandit.K)
        # 当前步的累积懊悔
        self.regret = 0.0
        # 维护一个列表,记录每一步的动作
        self.actions = []
        # 维护一个列表,记录每一步的累积懊悔
        self.regrets = []

    def UpdateRegret(self, Kth):
        # 计算累积懊悔并保存, Kth为本次动作选择的拉杆的编号
        self.regret += self.bandit.best_prob - self.bandit.probs[Kth]
        self.regrets.append(self.regret)

    def RunOnce(self):
        # 返回当前动作选择哪一根拉杆, 由每个具体的策略实现,需要继承后重写
        raise NotImplementedError

    def RunLoop(self, NumofSteps):
        # 运行一定次数, num_steps为总运行次数
        for _ in range(NumofSteps):
            Kth = self.RunOnce()
            self.counts[Kth] += 1
            self.UpdateRegret(Kth)
            self.actions.append(Kth)

class EpsilonGreedy(ProblemSolver):
    """ epsilon贪婪算法,继承ProblemSolver类"""
    def __init__(self, bandit, epsilon=0.01, init_prob=1.0):
        super(EpsilonGreedy, self).__init__(bandit)
        self.epsilon = epsilon
        # 初始化拉动所有拉杆的期望奖励估值
        self.EstimateReward = np.array([init_prob] * self.bandit.K)

    def RunOnce(self):
        if np.random.rand() < self.epsilon:
            # 随机选择一根拉杆
            Kth = np.random.randint(0, self.bandit.K)
        else:
            # 选择期望奖励估值最大的拉杆
            Kth = np.argmax(self.EstimateReward)
            # 得到本次动作的奖励
            Reward = self.bandit.step(Kth)
            # 更新期望奖励估值
            self.EstimateReward[Kth] += 1.0 / (self.counts[Kth] + 1) * (Reward - self.EstimateReward[Kth])
        return Kth

class DecayingEpsilonGreedy(ProblemSolver):
    """ epsilon值随时间衰减的epsilon-贪婪算法,继承Solver类 """
    def __init__(self, bandit, init_prob=1.0):
        super(DecayingEpsilonGreedy, self).__init__(bandit)
        # 初始化拉动所有拉杆的期望奖励估值
        self.EstimateReward = np.array([init_prob] * self.bandit.K)
        self.TimeCount = 0

    def RunOnce(self):
        self.TimeCount += 1
        if np.random.rand() < 1.0 / self.TimeCount:
            # 随机选择一根拉杆
            Kth = np.random.randint(0, self.bandit.K)
        else:
            # 选择期望奖励估值最大的拉杆
            Kth = np.argmax(self.EstimateReward)
            # 得到本次动作的奖励
            Reward = self.bandit.step(Kth)
            # 更新期望奖励估值
            self.EstimateReward[Kth] += 1.0 / (self.counts[Kth] + 1) * (Reward - self.EstimateReward[Kth])
        return Kth

class DecayingEpsilonGreedy2(ProblemSolver):
    """ epsilon值随时间衰减的epsilon-贪婪算法（另一种衰减策略）,继承Solver类 """
    def __init__(self, bandit, init_prob=1.0, coef=1.0):
        super(DecayingEpsilonGreedy2, self).__init__(bandit)
        # 初始化拉动所有拉杆的期望奖励估值
        self.EstimateReward = np.array([init_prob] * self.bandit.K)
        self.TimeCount = 0
        self.coef = coef

    def RunOnce(self):
        self.TimeCount += 1
        if self.regret > 0:
            d = self.regret
        else:
            d = 1
        coef_epsilon = min(1, (self.coef * self.bandit.K) / (d * d * self.TimeCount))
        if np.random.rand() < coef_epsilon:
            # 随机选择一根拉杆
            Kth = np.random.randint(0, self.bandit.K)
        else:
            # 选择期望奖励估值最大的拉杆
            Kth = np.argmax(self.EstimateReward)
            # 得到本次动作的奖励
            Reward = self.bandit.step(Kth)
            # 更新期望奖励估值
            self.EstimateReward[Kth] += 1.0 / (self.counts[Kth] + 1) * (Reward - self.EstimateReward[Kth])
        return Kth

class UCB(ProblemSolver):
    """ UCB算法,继承Solver类 """
    def __init__(self, bandit, init_prob=1.0, coef=1.0):
        super(UCB, self).__init__(bandit)
        # 初始化拉动所有拉杆的期望奖励估值
        self.EstimateReward = np.array([init_prob] * self.bandit.K)
        self.TimeCount = 0
        self.coef = coef

    def RunOnce(self):
        self.TimeCount += 1
        # 计算上置信界
        ucb = self.EstimateReward + self.coef * np.sqrt(
            np.log(self.TimeCount) / 2 / (self.bandit.K + 1)
        )
        # 选出上置信界最大的拉杆
        Kth = np.argmax(ucb)
        # 得到本次动作的奖励
        Reward = self.bandit.step(Kth)
        # 更新期望奖励估值
        self.EstimateReward[Kth] += 1.0 / (self.counts[Kth] + 1) * (Reward - self.EstimateReward[Kth])
        return Kth

class ThompsonSampling(ProblemSolver):
    """ 汤普森采样算法,继承Solver类 """
    def __init__(self, bandit):
        super(ThompsonSampling, self).__init__(bandit)
        # 列表,表示每根拉杆奖励为1的次数
        self.SuccessCounter = np.zeros(self.bandit.K)
        # 列表,表示每根拉杆奖励为0的次数
        self.FailureCounter = np.zeros(self.bandit.K)

    def RunOnce(self):
        # 按照Beta分布采样一组奖励样本
        Samples = np.random.beta(self.SuccessCounter + 1, self.FailureCounter + 1)
        # 选出采样奖励最大的拉杆
        Kth = np.argmax(Samples)
        # 得到本次动作的奖励
        Reward = self.bandit.step(Kth)
        if Reward == 1:
            self.SuccessCounter[Kth] += 1
        else:
            self.FailureCounter[Kth] += 1
        return Kth

def PlotResults(solvers, solver_names):
    """生成累积懊悔随时间变化的图像。输入solvers是一个列表,列表中的每个元素是一种特定的策略。
    而solver_names也是一个列表,存储每个策略的名称"""
    plt.style.use('seaborn-v0_8-paper')
    for idx, solver in enumerate(solvers):
        time_list = range(len(solver.regrets))
        plt.plot(time_list, solver.regrets, label=solver_names[idx])
    plt.xlabel('Time steps')
    plt.ylabel('Cumulative regrets')
    plt.title('%d-armed bandit' % solvers[0].bandit.K)
    plt.legend()
    plt.show()

# test01
def test01():
    # 设定随机种子,使实验具有可重复性
    np.random.seed(1)
    K = 10
    bandit_10_arm = BernoulliBandit(K)
    print("随机生成了一个%d臂伯努利老虎机" % K)
    print("获奖概率最大的拉杆为%d号,其获奖概率为%.4f" %
          (bandit_10_arm.best_idx, bandit_10_arm.best_prob))

# test02-epsilon
def test02():
    np.random.seed(0)
    K = 10
    bandit_10_arm = BernoulliBandit(K)
    EpsilonGreedySolver = EpsilonGreedy(bandit_10_arm, epsilon=0.01)
    EpsilonGreedySolver.RunLoop(5000)
    print('epsilon-贪婪算法的累积懊悔为：', EpsilonGreedySolver.regret)
    PlotResults([EpsilonGreedySolver], ["EpsilonGreedy"])

# test03-multi-epsilon
def test03():
    np.random.seed(0)
    K = 10
    bandit_10_arm = BernoulliBandit(K)
    epsilon_lists = [1e-4, 0.01, 0.1, 0.25, 0.5]
    EpsilonGreedySolvers = [EpsilonGreedy(bandit_10_arm, e) for e in epsilon_lists]
    EpsilonGreedySolversNames = ["epsilon={}".format(e) for e in epsilon_lists]
    for Solver in  EpsilonGreedySolvers:
        Solver.RunLoop(5000)
        print('epsilon-贪婪算法的累积懊悔为：', Solver.regret)
    PlotResults(EpsilonGreedySolvers, EpsilonGreedySolversNames)

# test04-decay-epsilon
def test04():
    np.random.seed(0)
    K = 10
    bandit_10_arm = BernoulliBandit(K)
    DecayingEpsilonGreedySolver = DecayingEpsilonGreedy(bandit_10_arm)
    DecayingEpsilonGreedySolver.RunLoop(5000)
    print('epsilon值衰减的贪婪算法的累积懊悔为：', DecayingEpsilonGreedySolver.regret)
    # print('epsilon值衰减的贪婪算法的每步懊悔为：', DecayingEpsilonGreedySolver.regrets)
    PlotResults([DecayingEpsilonGreedySolver], ["EpsilonGreedy"])

# test05-decay2-epsilon
def test05():
    np.random.seed(0)
    K = 10
    bandit_10_arm = BernoulliBandit(K)
    DecayingEpsilonGreedySolver2 = DecayingEpsilonGreedy2(bandit_10_arm, coef=1.0)
    DecayingEpsilonGreedySolver2.RunLoop(5000)
    print('epsilon值衰减的贪婪算法的累积懊悔为：', DecayingEpsilonGreedySolver2.regret)
    PlotResults([DecayingEpsilonGreedySolver2], ["EpsilonGreedy"])

# test06-UCB
def test06():
    np.random.seed(0)
    K = 10
    bandit_10_arm = BernoulliBandit(K)
    UCBSolver = UCB(bandit_10_arm, coef=2.0)
    UCBSolver.RunLoop(5000)
    print('epsilon值衰减的贪婪算法的累积懊悔为：', UCBSolver.regret)
    PlotResults([UCBSolver], ["UCB"])

# test07-ThompsonSampling
def test07():
    np.random.seed(0)
    K = 10
    bandit_10_arm = BernoulliBandit(K)
    ThompsonSamplingSolver = ThompsonSampling(bandit_10_arm)
    ThompsonSamplingSolver.RunLoop(5000)
    print('epsilon值衰减的贪婪算法的累积懊悔为：', ThompsonSamplingSolver.regret)
    PlotResults([ThompsonSamplingSolver], ["ThompsonSampling"])

if __name__ == '__main__':
    test07()

Some results :

reference

[1] Boyu AI
[2] https://www.deepmind.com/learning-resources/introduction-to-reinforcement-learning-with-david-silver
[3] Hands-on learning reinforcement learning
[4] Reinforcement Learning
[5 ] A Tutorial on Thompson Sampling https://web.stanford.edu/~bvr/pubs/TS_Tutorial.pdf
[6] An Empirical Evaluation of Thompson Sampling