table of Contents

Evaluation Method 1. Experience Network Configuration

In the design and configuration of your depth learning model, you have to make many decisions.

Most of these decisions can be resolved by copying other network structure and use of heuristics. However, the best way is actually small experimental design, and empirical evaluation of the actual data.

This includes high-level decisions, such as the number of layers, the number and type of network. It also includes lower-level decisions, such as choosing loss function, activation function, the optimization process and the number of cycles.

Depth study on the issue often used to have a very large data sets, such problems often have thousands of instances.

Therefore, you need to have a powerful testing tool that lets you in on the invisible data to estimate performance for a given configuration, and reliably compare the performance with other configurations.

2. Data Split

Large amounts of data and complex models require a long training time.

Thus, generally using a simple separating the data into training data and testing data set or training and validation data sets.

2.1 with automatic validation data set

Keras your training data may be divided into a portion of the validation data set, and then evaluate the performance of each period of the validation data set.

You can set Fit () function on validation_split parameter (set as a percentage of your training data set size) is achieved.

For example, a reasonable value may be 0.2 or 0.33, i.e. 20% or 33% of the set of training data is used for verification.

The following example demonstrates how to use the automatic validation data set on a small binary classification problem. All the examples used herein have the Pima Indian onset diabetes data sets . You can from the UCI Machine Learning Repository to download and save data files in your current working directory, a file named Pima-indians-diabetes.csv .

# MLP with automatic validation set
from keras.models import Sequential
from keras.layers import Dense
import numpy
# fix random seed for reproducibility
numpy.random.seed(7)
# load pima indians dataset
dataset = numpy.loadtxt("pima-indians-diabetes.csv", delimiter=",")
# split into input (X) and output (Y) variables
X = dataset[:,0:8]
Y = dataset[:,8]
# create model
model = Sequential()
model.add(Dense(12, input_dim=8, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
# Fit the model
model.fit(X, Y, validation_split=0.33, epochs=150, batch_size=10)

Run the sample, you can see the detailed output for each cycle, showing the loss and the accuracy of the training data set and validation data set.

...
Epoch 145/150
514/514 [==============================] - 0s - loss: 0.5252 - acc: 0.7335 - val_loss: 0.5489 - val_acc: 0.7244
Epoch 146/150
514/514 [==============================] - 0s - loss: 0.5198 - acc: 0.7296 - val_loss: 0.5918 - val_acc: 0.7244
Epoch 147/150
514/514 [==============================] - 0s - loss: 0.5175 - acc: 0.7335 - val_loss: 0.5365 - val_acc: 0.7441
Epoch 148/150
514/514 [==============================] - 0s - loss: 0.5219 - acc: 0.7354 - val_loss: 0.5414 - val_acc: 0.7520
Epoch 149/150
514/514 [==============================] - 0s - loss: 0.5089 - acc: 0.7432 - val_loss: 0.5417 - val_acc: 0.7520
Epoch 150/150
514/514 [==============================] - 0s - loss: 0.5148 - acc: 0.7490 - val_loss: 0.5549 - val_acc: 0.7520

2.2 manual validation data set

Keras also allows you to manually set up to be validated data set during training.

In this example, we use Python's scikit-learn machine learning library train_test_split () function to our data into training and test data sets. We use 67% of the training, and the remaining 33% of the data for validation.

Validation data set by validation_data assigned Keras parameters Fit () function. It requires a set of input and output data of the array:

# MLP with manual validation set
from keras.models import Sequential
from keras.layers import Dense
from sklearn.model_selection import train_test_split
import numpy
# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)
# load pima indians dataset
dataset = numpy.loadtxt("pima-indians-diabetes.csv", delimiter=",")
# split into input (X) and output (Y) variables
X = dataset[:,0:8]
Y = dataset[:,8]
# split into 67% for train and 33% for test
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, random_state=seed)
# create model
model = Sequential()
model.add(Dense(12, input_dim=8, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
# Fit the model
model.fit(X_train, y_train, validation_data=(X_test,y_test), epochs=150, batch_size=10)

As before, an example of detailed operation of the training output, including the loss of accuracy of the model and on each cycle of the training and validation data sets.

...
Epoch 145/150
514/514 [==============================] - 0s - loss: 0.4847 - acc: 0.7704 - val_loss: 0.5668 - val_acc: 0.7323
Epoch 146/150
514/514 [==============================] - 0s - loss: 0.4853 - acc: 0.7549 - val_loss: 0.5768 - val_acc: 0.7087
Epoch 147/150
514/514 [==============================] - 0s - loss: 0.4864 - acc: 0.7743 - val_loss: 0.5604 - val_acc: 0.7244
Epoch 148/150
514/514 [==============================] - 0s - loss: 0.4831 - acc: 0.7665 - val_loss: 0.5589 - val_acc: 0.7126
Epoch 149/150
514/514 [==============================] - 0s - loss: 0.4961 - acc: 0.7782 - val_loss: 0.5663 - val_acc: 0.7126
Epoch 150/150
514/514 [==============================] - 0s - loss: 0.4967 - acc: 0.7588 - val_loss: 0.5810 - val_acc: 0.6929

2.3 k- fold cross validation manual

Gold standard machine learning models is assessed k- fold cross-validation (k-fold cross validation).

It provides a reliable assessment of the performance data model is unknown. It is through the training data set into k subsets, a subset of the launch to do the test set, the remaining subset of turns and compare it to train the model. Repeat this process until all the data sets have become the validation data set. Finally, the average performance of all models of assessment.

Cross-validation is usually not used to assess the depth of learning models, because the calculation more costly. K- fold cross validation, for example, usually 5 or 10 fold. Therefore, we must construct and evaluate model 5 or 10, greatly increasing the time of evaluation models.

However, when the problem is small enough, or if you have enough computing resources, k- fold cross-validation can make you less biased estimate of model performance.

In the following example, we use the Python scikit-learn machine learning library StratifiedKFold class, the training data set is divided into 10 fold. Folding is hierarchical, which means that the algorithm tries to balance the number of instances of each class

This example uses 10 divisions to create and evaluate data model 10, and collect all the points. By "verbose = 0" is transmitted to the Model Fit () function and evaluate () function, close the details of the output of each cycle.

Printing performance of each model, and storage. Then print performance model mean and standard deviation at the end of operation, to provide a reliable model estimation accuracy.

# MLP for Pima Indians Dataset with 10-fold cross validation
from keras.models import Sequential
from keras.layers import Dense
from sklearn.model_selection import StratifiedKFold
import numpy
# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)
# load pima indians dataset
dataset = numpy.loadtxt("pima-indians-diabetes.csv", delimiter=",")
# split into input (X) and output (Y) variables
X = dataset[:,0:8]
Y = dataset[:,8]
# define 10-fold cross validation test harness
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)
cvscores = []
for train, test in kfold.split(X, Y):
  # create model
	model = Sequential()
	model.add(Dense(12, input_dim=8, activation='relu'))
	model.add(Dense(8, activation='relu'))
	model.add(Dense(1, activation='sigmoid'))
	# Compile model
	model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
	# Fit the model
	model.fit(X[train], Y[train], epochs=150, batch_size=10, verbose=0)
	# evaluate the model
	scores = model.evaluate(X[test], Y[test], verbose=0)
	print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
	cvscores.append(scores[1] * 100)
print("%.2f%% (+/- %.2f%%)" % (numpy.mean(cvscores), numpy.std(cvscores)))

Run example requires less than a minute, produces the following output:

acc: 77.92%
acc: 68.83%
acc: 72.73%
acc: 64.94%
acc: 77.92%
acc: 35.06%
acc: 74.03%
acc: 68.83%
acc: 34.21%
acc: 72.37%
64.68% (+/- 15.50%)

Reprinted: http: //machinelearningmastery.com/evaluate-performance-deep-learning-models-keras

Nick Blog

Published 38 original articles · 98 won praise · views 360 000 +

Private letter concerns

Performance Evaluation Keras depth learning model

Evaluation Method 1. Experience Network Configuration

2. Data Split

2.1 with automatic validation data set

2.2 manual validation data set

2.3 k- fold cross validation manual

Guess you like