1. Description

Training deep learning models is a time-consuming process. You can save model progress during and after training. So you can pick up where you left off and overcome lengthy training challenges.

In this blog post, we'll walk through how to save a model and progressively load it with Keras. We'll also explore model checkpoint callbacks, which are commonly used for model training.

2. Load the dataset

To demonstrate how to save a model, let's use the MNIST dataset. This dataset consists of digital images.

MNIST dataset

Before loading the MNIST dataset, let's import TensorFlow and Keras.

import tensorflow as tf
from tensorflow import keras

Now, let's load the training and test datasets using methods in Keras.load_data

(train_images,train_labels),(test_images,test_labels)=tf.keras.datasets.mnist.load_data()

The training input and output datasets consist of 60,000 samples, and the testing input and output datasets consist of 10,000 samples.

3. Data preprocessing

One of the most important steps in data analysis is data preprocessing. In deep learning, some data preprocessing techniques such as normalization and regularization can improve the performance of the model.

First, let's get the first 1000 samples from these datasets to run the code faster. Let's do this for the output variable first.

train_labels = train_labels[:1000]
test_labels = test_labels[:1000]

We will do the same for the training data. Data samples consist of digital images. These images are two-dimensional. Let's use this method to convert these examples to dimensions before feeding them to the model. Also, let's normalize the data to improve the performance of the model and make training faster.reshape

train_images = train_images[:1000].reshape(-1, 28 * 28) / 255.0
test_images = test_images[:1000].reshape(-1, 28 * 28) / 255.0

Great, our dataset is ready for the model. Let's move on to the model building step.

4. How to build a model

The easiest way to build deep learning models is the sequential technique in Keras. In this technique, layers are stacked one after the other. All we have to do now is define a function that contains the model. By doing this, we can build models more easily.

def create_model():
    model = tf.keras.Sequential([
      keras.layers.Dense(512, activation='relu',input_shape=(784,)),
      keras.layers.Dropout(0.2),
      keras.layers.Dense(10)
    ])
 
    model.compile(
      optimizer='adam',
      loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
      metrics=[tf.keras.metrics.SparseCategoricalAccuracy()]
    )
    return model

Let's take a look at the code. First, we define a function that creates and compiles a sequential model using Keras.

We build a model with two dense layers, the first layer has 512 neurons and an activation function. We also set up a dropout layer that randomly drops 20% of the input units to help prevent overfitting. After that, we write a dense layer with 10 neurons without activation function, as it will be used for logits.relu

Next, we compile the model with the optimizer and loss function. As an indicator, we set .AdamSparseCategoricalCrossentropySparseCategoricalAccuracy

Finally, we use the statement to return the compiled model.return

Great, we've defined a simple sequential model. Now, let's get a sample object called model from this function.

model = create_model()

Let us now look at the architecture of the model using the summarization method.

model.summary()

As you can see, our model consists of an input layer, a dropout layer, and an output layer. Let's move on to exploring callbacks.ModelCheckpoint

5. Use model checkpoint callbacks to save model weights

Models can be saved to reuse a trained model, or to continue training from where it left off.

As you know, building a model actually means training the weights of the model, called parameters. Callbacks allow you to save the model's weights during model training. To illustrate this, let's instantiate an object from this callback.ModelCheckpoint

First, let's create the directory where the models will be saved with the os module.

import os
checkpoint_path = "training_1/my_checkpoints"
checkpoint_dir = os.path.dirname(checkpoint_path)

Great, we've created our directory. Now let's create a callback to hold the model's weights.

checkpoint_cb = tf.keras.callbacks.ModelCheckpoint(
     filepath=checkpoint_path,
     # Let's save only the weights of the model
     save_weights_only=True)

Great, we've created our callback. Now, let's call the method and the callback for this method.fitpass

model.fit(train_images,
   train_labels,
   epochs=10,
   validation_data=(test_images, test_labels),
   callbacks=[checkpoint_cb])

Therefore, we save the model weights in a directory. The callback we use updates the checkpoint file at the end of each epoch. Let's use the os module to view the files in a directory.

os.listdir(checkpoint_dir)

# Output
['my_checkpoints.index', 'my_checkpoints.data-00000-of-00001', 'checkpoint']

As you can see, the weights are saved after the last epoch. Let's move on and look at how to load the weights.

6. Loading weight

After saving the weights, they can be loaded into the model. Note that you can only use saved weights for models with the same architecture.

Let's instantiate an object to demonstrate this.

model = create_model()

Note that we have not trained the weights for this model. These weights are randomly generated. Now let's see the accuracy score of this untrained model on the test data.evaluate

loss, acc = model.evaluate(test_images, test_labels, verbose=2)
print(f"Untrained model, accuracy: {100 * acc:5.2f}%")

# Output:
Untrained model, accuracy: 10.70%

As you can see, the untrained model is about 10% accurate on the test data. That's a pretty bad score, right?

Now, let's load the weights we saved earlier using this method and see the accuracy score of this model on the test data.load_weights

model.load_weights(checkpoint_path)

Great, we loaded the weights. Now, let's check the performance of this model on the test set.

loss, acc = model.evaluate(test_images, test_labels, verbose=2)
print(f"Untrained model, accuracy: {100 * acc:5.2f}%")

#Output:
Untrained model, accuracy: 87.40%

As you can see, the model turns out to be about 90% accurate.

In this section, we have seen how to save model weights and how to load them. Now let's move on to exploring how to save the entire model.

7. Save the entire model

q After training the model, it may be necessary to deploy the model. This method can be used to save the model's architecture, weights, and training configuration in a single file.save

You can save your model in two different formats as well. Remember that in Keras this format is used by default. Let's save the final model. Let's create a directory for it.SaveModelHDF5SavedModel

mkdir saved_model

Now let's save the model in this file.

model.save('saved_model/my_model')

Great, we saved our model. Let's take a look at the files in this directory.

ls saved_model/my_model

# Output:
assets fingerprint.pb keras_metadata.pb saved_model.pb variables

Here you can see files and subdirectories. The model's source code is not required to put the model into production. enough for deployment. Let's take a closer look at these files.SavedModel

This file contains the architecture and computational graph of the model.saved_model.pd

This file contains extra information required by Keras.keras_metadata.pb

Subdirectories contain parameter values such as weights and biases.variables

Subdirectories contain additional files, such as names of attributes and classes.assets

Great, we saw how to save the entire model. Now let's see how to load the model.

8. Load the model

You can use this method to load a saved model. To do this, let's first create the model architecture, then load the model.load_model

new_model = create_model()
new_model = tf.keras.models.load_model('saved_model/my_model')

Great, we have loaded the model. Let's look at the architecture of this model.

new_model.summary()

Note that this model was compiled with the same parameters as the original model. Let's see how accurate this model is on the test data.

loss, acc = new_model.evaluate(test_images, test_labels, verbose=2)
print(f'Restored model, accuracy: {100 * acc:5.2f}%')

# Output:
Restored model, accuracy: 87.40%

As you can see, the accuracy score of our model saved on the test data is 87%.

You can also save the model in format. But most TensorFlow deployment tools require this format.h5SavedModel

Nine. Summary

When training a model, you can save the model to continue where you left off. By saving your model, you can also share your model and allow others to recreate your work.

In this blog post, we covered how to save and load deep learning models. First, we learned how to save model weights using callbacks. Next, we saw saving and loading the entire model to deploy the model.ModelCheckpoint

Thanks for reading. You can find a link to the notes here .

References:

How to save and load deep learning models with Keras? |Towards AI (towardsai.net)

[NLP] Use Keras to save and load deep learning models