Code combining cosine decay learning rate and linear warmup

The following code is taken from tensorflow official tpu repository

def cosine_learning_rate_with_linear_warmup(global_step,
                                            init_learning_rate,
                                            warmup_learning_rate,
                                            warmup_steps,
                                            total_steps):
    """Creates the cosine learning rate tensor with linear warmup."""
    global_step = tf.cast(global_step, dtype=tf.float32)
    linear_warmup = (warmup_learning_rate + global_step / warmup_steps *
                   (init_learning_rate - warmup_learning_rate))
    cosine_learning_rate = (
        init_learning_rate * (tf.cos(
            np.pi * (global_step - warmup_steps) / (total_steps - warmup_steps))
                              + 1.0) / 2.0)
    learning_rate = tf.where(global_step < warmup_steps,
                             linear_warmup, cosine_learning_rate)
    return learning_rate

For the meaning of the five parameters, just look at the picture. It’s easy to take a look at the code.

In the warmup stage, the learning rate changes from warmup_learning_rateto init_learning_rate. In this stage, the learning rate increases or decreases linearly.

In the cosine decay stage, the learning rate decays like this:

$$lr=\frac{cos\left(\frac{gl-}{t-w}p\right)+1}{2}\ast init\_learning\_rate$$

c o s cos Variables in$cos\; :$

• $gl$是$global\_step$
• $w$ 是 $warmup\_steps$
• $t$ 是 $total\_step$

The attenuation curve is shown in the blue box in the figure below:

the degree of decline gradually accelerates at first, then gradually slows down, and converges to a very small value.