Resume Training: during neural network training process due to some factors can not be trained, you need to save the current results then the next training exercise
fully connected back-propagation neural network, the code of the training process is as follows:
# Coding: UTF. 8- # 1 Forward propagation Import tensorflow TF AS # Network input node 784 (the number of pixels on behalf of each of the input picture) INPUT_NODE = 784 # Output node 10 (as represented by the output out of the class 0-9) OUTPUT_NODE = 10 # Hidden layer node 500 LAYER1_NODE = 500 DEF get_weight (Shape, regularizer): # parameters satisfy truncated normal distribution, and use regularization, W = tf.Variable (tf.truncated_normal (Shape, STDDEV = 0.1 )) # W = tf.Variable (tf.random_normal (Shape , STDDEV = 0.1)) # total loss of n for each parameter is added to the loss of the IF regularizer = None:! tf.add_to_collection ( ' lOSSES ' , tf.contrib.layers.l2_regularizer (regularizer) (W)) return W DEF get_bias (Shape): # one-dimensional array initialization, initialization is full 0 B = tf.Variable (tf.zeros (Shape)) return B DEF Forward (X, regularizer): # is the input layer to the hidden layer is a shape parameter w1 [784,500] w1 = get_weight ([INPUT_NODE, LAYER1_NODE], regularizer) # from the input layer to the hidden bias b1 shape of a length 500 one-dimensional array, B1 = get_bias ([LAYER1_NODE]) # front propagating structure as input to a first parameter w1 x layer matrix multiplication plus offset b1, then after relu function, to obtain the hidden layer output y1. tf.nn.relu = Y1 (tf.matmul (X, W1) + B1) # from the hidden layer to the output layer is a shape parameter w2 as [500, 10] w2 = get_weight ([LAYER1_NODE, OUTPUT_NODE], regularizer) # of b2 shape of the hidden layer to the bias output 10 is one-dimensional array length b2 = get_bias ([OUTPUT_NODE]) # forward propagating structure of the second layer is hidden outputs y1 and adding an offset parameter w2 matrix multiplication matrix multiplication plus bias b2, to obtain output y. #Since the output. Since the output y after softmax oftmax to function, to conform to a probability distribution, so the output y without relu function y = tf.matmul (Y1, w2 of) + B2 return y
# Coding: UTF. 8- # 2 back-propagation procedure # introduced tensorflow, input_data, and the forward propagation mnist_forward os module Import tensorflow TF AS from tensorflow.examples.tutorials.mnist Import Input_Data Import mnist_forward Import os # Image neural network feeding each round number BATCH_SIZE = 200 # Initial learning rate LEARNING_RATE_BASE = 0.1 # Learning rate decay rate LEARNING_RATE_DECAY = 0.99 # Regularization coefficient REGULARIZER = 0.0001 # Training rounds STEPS = 50000 # Moving average decay rate MOVING_AVERAGE_DECAY = 0.99 # Model save path MODEL_SAVE_PATH = " ./model/ " # Save the model name MODEL_NAME = " mnist_model " DEF Backward (MNIST): # a placeholder to training data x and y_ tag placeholder x = tf.placeholder (tf.float32, [None, mnist_forward.INPUT_NODE]) Y_ = tf.placeholder (tf.float32, [None, mnist_forward.OUTPUT_NODE]) # Before calling the file mnist_forward propagation ForWord () function, and set the regularization, the training data set is calculated on the prediction result Y Y = mnist_forward .forward (X, REGULARIZER) # current assignment calculation round number counter is set to the type of training not global_step tf.Variable = (0, trainable = False) # Call contains all the parameters regularization loss function loss Loss CE = tf.nn.sparse_softmax_cross_entropy_with_logits (logits = Y, Labels = tf.argmax (Y_,. 1 )) cem = tf.reduce_mean(ce) Loss = CEM + tf.add_n (tf.get_collection ( ' LOSSES ' )) # set exponential decay learning rate learning_rate learning_rate = tf.train.exponential_decay ( LEARNING_RATE_BASE, global_step, mnist.train.num_examples / BATCH_SIZE, LEARNING_RATE_DECAY, staircase=True) # Gradient attenuation model optimization algorithm, reducing the loss function # train_step = tf.train.GradientDescentOptimizer (learning_rate) .minimize (Loss, global_step = global_step) train_step = tf.train.MomentumOptimizer (learning_rate, 0.9) .minimize (Loss, global_step = global_step) # train_step = tf.train.AdamOptimizer (learning_rate) .minimize (Loss, global_step = global_step) # moving average parameters defined EMA = tf.train.ExponentialMovingAverage (MOVING_AVERAGE_DECAY, global_step) ema_op = ema.apply (tf.trainable_variables ()) # instantiated reducible moving average Saver # introduced slipping when the average model training model allows more robust performance on the test data with tf.control_dependencies ([train_step, ema_op] ): train_op = tf.no_op(name='train') saver = tf.train.Saver() with tf.Session() as sess: # Initialise all parameters init_op = tf.global_variables_initializer () sess.run (init_op) # Breakpoint training added ckpt operation ckpt = tf.train.get_checkpoint_state (MODEL_SAVE_PATH) IF ckpt and ckpt.model_checkpoint_path: saver.restore(sess, ckpt.model_checkpoint_path) # Each feed batch_size group (i.e., group 200) the training data and corresponding tags, loop iteration steps wheel for I in Range (the STEPS): xs, ys = mnist.train.next_batch(BATCH_SIZE) _, loss_value, step = sess.run([train_op, loss, global_step], feed_dict={x: xs, y_: ys}) if i % 1000 == 0: print("After %d training step(s), loss on training batch is %g." % (step, loss_value)) #将当前会话加载到指定路径 saver.save(sess, os.path.join(MODEL_SAVE_PATH, MODEL_NAME), global_step=global_step) DEF main (): # reads MNIST MNIST = input_data.read_data_sets ( " ./data/ " , one_hot = True) # backpropagation backward (mnist) if __name__ == '__main__': main()
# Coding: UTF. 8- # accuracy and generalization of the network authentication Import Time Import tensorflow TF AS from tensorflow.examples.tutorials.mnist Import Input_Data Import mnist_forward Import mnist_backward # program cycle interval of time of 5 seconds TEST_INTERVAL_SECS = 5 DEF Test (MNIST): # using tf.Graph () is calculated as defined previously reproduced FIG with tf.Graph () as_default () AS G:. # using training data x and a placeholder tag placeholder y_ x = TF. placeholder (tf.float32, [None, mnist_forward.INPUT_NODE ]) Y_ = tf.placeholder (tf.float32, [None, mnist_forward.OUTPUT_NODE]) # Before calling the file mnist_forward propagation ForWord () function Y = mnist_forward.forward (X, None) # Example of a sliding average saver object when the session is loaded so that all parameters of the model is assigned a respective sliding average, enhance the stability of the model EMA = tf.train.ExponentialMovingAverage (mnist_backward.MOVING_AVERAGE_DECAY) ema_restore = ema.variables_to_restore() Saver = tf.train.Saver (ema_restore) # calculation accuracy of the model on the test set correct_prediction = tf.equal (tf.argmax (Y,. 1), tf.argmax (Y_,. 1 )) accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) while True: with tf.Session() as sess: # Loading ckpt specified path ckpt = tf.train.get_checkpoint_state (mnist_backward.MODEL_SAVE_PATH) # if present model, a model is loaded into the current conversation, verified for accuracy on a test data set, and print out the current number wheel accuracy IF CKPT and ckpt.model_checkpoint_path: saver.restore(sess, ckpt.model_checkpoint_path) global_step = ckpt.model_checkpoint_path.split('/')[-1].split('-')[-1] accuracy_score = sess.run (Accuracy, feed_dict = {X: mnist.test.images, Y_: mnist.test.labels}) Print ( " the After Training% S STEP (S),% G = Accuracy Test " % (global_step, accuracy_score)) # If the model does not exist, there is no model prompted printed out, so test () function to complete the else : Print ( ' no File found the checkpoint ' ) return time.sleep(TEST_INTERVAL_SECS) DEF main (): # loading the test data set in the specified path MNIST = input_data.read_data_sets ( " ./data/ " , one_hot = True) test(mnist) if __name__ == '__main__': main()
Key Processing: Add ckpt Operation:
ckpt = tf.train.get_checkpoint_state (MODEL_SAVE_PATH)
IF ckpt and ckpt.model_checkpoint_path:
saver.restore (Sess, ckpt.model_checkpoint_path)
. 1, notes:
. 1) tf.train.get_checkpoint_state (checkpoint_dir, latest_filename = none)
this function indicates if the folder contains a valid breakpoint breakpoint status file is returned.
Parameter Description: checkpoint_dir: represents the directory where the file is stored breakpoint
latest_filename = None: The optional name breakpoints file, the default is "checkpoint"
2) saver.restore (sess, ckpt.model_checkpoint_path)
This function indicates the current recovery session will ckpt assigned values w and b. a
Parameter Description: sess: indicates the current session, held before the results will be loaded into this conversation
ckpt.model_checkpoint_path: indicates the position of the storage model, does not require analog-
type name, it will go to see the checkpoint file, to see who is the latest What is called.
Note: This article is by watching Cao Jian teacher at Peking University Tensorflow video, summary notes come.