Monte-Carlo Dropout
Monte-Carlo Dropout (Monte Carlo dropout), referred to as MC dropout.
Dropout appreciated that way departing from the Bayesian theory, Bayesian Dropout interpreted as approximate Gaussian process.
Foggy, proof theory looks pretty complicated, we are interested can refer to the paper: Dropout AS A Bayesian Approximation:. Representing Model Uncertainty in Deep Learning and the paper's Appendix .
But in fact, MC dropout with them on the simple, no need to modify existing neural network model, only the neural network model with dropout level, whether it is a standard dropout or its variants, such as drop-connect, are possible.
When training, MC dropout dropout forms and there is no difference, in accordance with the normal model training methods training can be.
During the test, the forward propagation, Dropout neural networks can not be disabled. This is the only difference and normally use.
MC MC dropout is reflected in the many times before that we need to spread the same input to the process, so that in blessing dropout can get the output "different network structure", the outputs of these statistics mean and variance, you can get the model predictions and uncertainty. Moreover, this process can work in parallel, so the time can be equal to the propagation time ago.
softmax probabilistic neural network generated can not be expressed uncertainty?
In fact, we took a lot of time to calculate the probability softmax of uncertainty, such as active learning query least confident in the policy, margin, entropy. In entropy strategy, the more uniform the greater the probability softmax entropy, we believe that the greater the uncertainty; on the contrary, in one dimension softmax close to 1, the other is close to 0, the minimum uncertainty.
However, softmax value and reliability should not be counter-sample classification result. A model can be uncertain in its predictions even with a high softmax output. [1]
To MNIST classification, for example, when the model validation set above effect sucks when the picture input to the neural network, we can still get a very high value softmax, this time the classification results are not reliable; when model validation set very good effect, even very good on the test set, this time, we will add some picture noise, or a digital handwritten photographed, entered into the network, this time to get a higher value softmax, we believe to be reliable results? This time we can understand, in the known information, the model think they do very good, but the model itself can not be generalized to all samples and space to go, for the data not seen it, it's possible generalization not so strong, this time the model is still based on information known to the data have not seen this strong judgment (softmax a great dimension values), of course, sometimes very good judge, but the judge may sometimes have mistake, but the model does not give much confidence to have this judgment.
The MC dropout can give a predicted value, and gives confidence to this predictive value, which is the advantage of Bayesian depth study lies.
MC dropout Sample Code
import tensorflow as tf
mnist = tf.keras.datasets.mnist
(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
inp = tf.keras.layers.Input(shape=(28, 28))
x = tf.keras.layers.Flatten()(inp)
x = tf.keras.layers.Dense(512, activation=tf.nn.relu)(x)
x = tf.keras.layers.Dropout(0.5)(x, training=True) # dropout 在训练和测试时都将开着
out = tf.keras.layers.Dense(10, activation=tf.nn.softmax)(x)
model = tf.keras.Model(inp, out)
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_train, y_train, epochs=3)
# 在测试过程，dropout 也是打开的，得到的结果将会有波动，而不是完全一致
for _ in range(10):
print(model.predict(x_test[:1]))dropout layer has been in an open state, the testing process is repeated several times.
References
[1] Gal, Y., & Ghahramani, Z. (2015). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. Retrieved from http://arxiv.org/abs/1506.02142
[2] Gal, Y., & Ghahramani, Z. (2015). Dropout as a Bayesian Approximation: Appendix. Retrieved from http://arxiv.org/abs/1506.02157
Two kinds of experimental uncertainty [notes] depth learning (on) - Zhang Zi Yang
Dropout Past and Present - the heart of the machine's
Deep Bayesian Neural Networks -. Stefano Cosentino