Select the learning rate - Moving Average

In the neural network model, the MOVING_AVERAGE_DECAY to 0.99, parameter w1 is set to 0, provided moving average w1
is set to 0.
① The start, the number of global_step wheel is set to 0, the parameter is updated to 1 w1, w1 is the running average:
w1 running mean = min (0.99,1 / 10) * 0 + (1- min (0.99,1 / 10) * = 0.9. 1
② when the wheel 100 is set to the number global_step, w1 parameter update 10, the moving average becomes:
w1 running mean = min (0.99,101 / 110) * 0.9 + (1- min ( 0.99,101 / 110) * 10 = 0.826 + 0.818 = 1.644
③ run again, updating parameters w1 to 1.644, the sliding average becomes:
w1 running mean = min (0.99,101 / 110) * 1.644 + (1- min (0.99,101 / 110) * 10 = 2.328
④ run again, updating parameters w1 to 2.328, the moving average:
w1 = 2.956 sliding average
code is as follows:

# Coding: UTF. 8- 
Import tensorflow TF AS 

# 1. Variables defined class and moving average 
# defines a 32-bit floating-point variables, the initial value of 0.0 This code is updated parameters w1, w1 optimization parameters, w1 made a moving average shadow 
W1 tf.Variable = (0, DTYPE = tf.float32)
 # define NUM_UPDATES (NN number of iterations), the initial value is 0, not be optimized (trained), this parameter is not trained 
global_step = tf.Variable (0 , trainable = False)
 # instantiated class moving average, a 0.99 decay rate, the current number of rounds global_step 
MOVING_AVERAGE_DECAY = 0.99 
EMA = tf.train.ExponentialMovingAverage (MOVING_AVERAGE_DECAY, global_step) # Moving average 
# brackets after the update is in ema.apply list, each time you run sess.run (ema_op), moving average seek to update the list of elements. 
# Will be used in practical applications tf.trainable_variables () automatically all the parameters to be trained as a summary list 
#= ema.apply ema_op ([W1]) 
# Apply (FUNC [, args [, kwargs]]) function is used when the function parameters already exists in the dictionary or a tuple, the indirect calling function. 
= ema_op ema.apply (tf.trainable_variables ()) 

# 2. View different iterations change the value of the variable. 
tf.Session with () AS Sess:
     # initialization 
    init_op tf.global_variables_initializer = () # initialize 
    sess.run (init_op) # calculates initialization 
    # obtaining moving average value w1 (to run with multiple nodes ema.average (w1), as elements listed in the list, write in sess.run in) 
    # print the current parameters w1 and w1 moving average 
    Print  " current global_step: " , sess.run (global_step) # print global_step 
    Print  " current w1 ", Sess.run ([w1, ema.average (w1)]) # calculates the moving average 
    
    # parameter is assigned the value of w1. 1 
    # tf.assign (A, NEW_NUMBER): This function is mainly a function of the value A becomes NEW_NUMBER 
    sess.run (tf.assign (W1,. 1 )) 
    sess.run (ema_op) 
    Print  " Current global_step: " , sess.run (global_step)
     Print  " Current W1 " , sess.run ([W1, ema.average ( w1)]) 
    
    # updated global_step values of w1 and simulate the round number is 100, the parameters w1 becomes 10, the following code is kept global_step 100, each time the moving average operation, the shadow value is updated 
    sess.run (tf. ASSIGN (global_step, 100))   # set global_step to 100 
    sess.run (tf.assign (W1, 10)) # set W1 is 10 
    sess.run (ema_op) #Run ema_op 
    Print  " Current global_step: " , sess.run (global_step) # print 
    Print  " Current W1: " , sess.run ([W1, ema.average (W1)])   # print      
    
    # each sess.run will be updated w1 sliding average 
    sess.run (ema_op)
     Print  " Current global_step: " , sess.run (global_step)
     Print  " Current w1: " , sess.run ([w1, ema.average (w1)]) 

    sess.run ( ema_op) 
    Print  " Current global_step: " , sess.run (global_step)
     Print  " Current W1:", sess.run([w1, ema.average(w1)])

    sess.run(ema_op)
    print "current global_step:" , sess.run(global_step)
    print "current w1:", sess.run([w1, ema.average(w1)])

    sess.run(ema_op)
    print "current global_step:" , sess.run(global_step)
    print "current w1:", sess.run([w1, ema.average(w1)])

    sess.run(ema_op)
    print "current global_step:" , sess.run(global_step)
    print "current w1:", sess.run([w1, ema.average(w1)])

    sess.run(ema_op)
    print "current global_step:" , sess.run(global_step)
    print "current w1:", sess.run([w1, ema.average(w1)])

Seen from the results of operation, the initial parameters w1 and moving average are 0; w1 parameter is set to 1, the sliding average value becomes 0.9;
when the number of iterations is updated to 100, after updating the parameter w1 10, moving average value becomes 1.644. Each time then, the parameter
moving average are close to parameters w1 w1. Visible, following a change in the moving average process variations.