Table of Contents
2. Anomaly detection based on automatic encoder
Third, the implementation of anomaly detection Tensorflow
3.2, build a self-coding model
1. Anomaly detection
An abnormality is defined as an event that deviates from the standard, rarely occurs, and does not follow the rest of the "pattern". Examples of exceptions include:
- The stock market plummeted due to world events
- Defective items on factory/conveyor belt
- Contaminated samples in the laboratory
Assuming that our data obeys a normal distribution, the abnormal data are usually located on both sides of the normal distribution curve. As shown below.
As we have seen, these events will occur, but the probability of occurrence is extremely low. From a machine learning perspective, this makes it difficult to detect anomalies-by definition, we have many examples of "standard" events and very few examples of "abnormal" events. Therefore, our data set has a large deviation. When the anomaly we want to detect may only occur 1%, 0.1%, or 0.0001% of the time, how should the machine learning algorithm that works best in the balanced data set work? At this time, we need to use anomaly detection to specifically deal with this type of problem.
Due to the huge imbalance in the labels of our data set, abnormal data rarely occurs by definition, and we have a large amount of normal data. In order to detect abnormal data, traditional machine learning algorithms have derived such as isolated forests, One-class SVMs, Elliptic Envelopes, and local anomaly factor algorithms. I will not introduce them one by one here. Interested students can go to research and study. The main topic here is how to use deep learning to solve this problem.
2. Anomaly detection based on automatic encoder
Autoencoder is an unsupervised neural network, it can:
- Accept a set of input data;
- Encode the data;
- Decode the encoded features to reconstruct the input data.
Usually an autoencoder mainly consists of two parts: encoder and decoder. The encoder accepts input data and converts it into a feature representation. Then, the decoder attempts to reconstruct the compressed features to obtain the input data. When we train the autoencoder in an end-to-end manner, the network can learn a powerful filter that can even denoise the input data.
From the perspective of anomaly detection, the autoencoder is unique in that it can reconstruct the loss. The mean square error (MSE) between the input and the reconstructed data is usually measured when the autoencoder is trained. If the loss is smaller, the reconstructed image is more like the original data.
Suppose we train an autoencoder on the entire MNIST data set, and then provide a number to the autoencoder and reconstruct it. We hope we can reconstruct it to be similar to the input data, and then we will find that the MSE between these two images is relatively low.
If, at this time, we provide a non-digital image, such as an elephant, the MSE of these two images is very high at this time. This is because the autoencoder has never seen an elephant before, and more importantly, has never been trained to reconstruct an elephant, so our MSE is very high.
At this time, if the reconstructed MSE is high, then we may have an outlier.
Third, the implementation of anomaly detection Tensorflow
3.1, data loading
Here we use the mnist data set, the label is 1 as normal data, the label is 1% of the number of data and the label is 3 as abnormal data, and make abnormal data. details as follows:
- Load data set
# 加载MNIST数据集
print("[INFO] loading MNIST dataset...")
((trainX,trainY),(testX,testY))=tf.keras.datasets.mnist.load_data()- Generate abnormal data
def build_unsupervised_dataset(data,labels,validLabel=1,
anomalyLabel=3, contam=0.01, seed=42):
'''制作数据集'''
# 抓取提供的类标签的所有 * 真正 * 索引特定的标签，
# 然后获取图像的索引个标签将成为我们的“异常”
validIdxs=np.where(labels==validLabel)[0]
anomalyLabelIdx=np.where(labels==anomalyLabel)[0]
#随机打乱数据
random.shuffle(validIdxs)
random.shuffle(anomalyLabelIdx)
#计算并设置异常数据的个数
i=int(len(validIdxs)*contam)
anomalyLabelIdx=anomalyLabelIdx[:i]
#提取正常数据和异常数据
validImages=data[validIdxs]
anomalyImages=data[anomalyLabelIdx]
#打包数据并进行数据打乱
images=np.vstack([validImages,anomalyImages])
return images
# 建立少量的无监督图像数据集,污染（即异常）添加到其中
print("[INFO] creating unsupervised dataset...")
images = build_unsupervised_dataset(trainX, trainY, validLabel=1,
anomalyLabel=3, contam=0.01)3.2, build a self-coding model
Here is the code directly:
class ConvAutoencoder:
@staticmethod
def build(width,height,depth=None,filters=(32,64),latentDim=16):
'''
构建自动编码器模型
:param width: 图像的宽度
:param height: 图像的高度
:param depth: 图像的深度
:param filters: 卷积核的尺寸
:param latentDim: 全连接的维数
:return:
'''
#定义解码器
inputs=tf.keras.layers.Input(shape=(height,width,depth))
x=inputs
for filter in filters:
x=tf.keras.layers.Conv2D(filter,kernel_size=(3,3),strides=2,padding='same')(x)
x=tf.keras.layers.LeakyReLU(alpha=0.2)(x)
x=tf.keras.layers.BatchNormalization(axis=-1)(x)
volumeSize=tf.keras.backend.int_shape(x)
x=tf.keras.layers.Flatten()(x)
latent=tf.keras.layers.Dense(latentDim)(x)
encoder=tf.keras.Model(inputs=inputs,outputs=latent,name='encoder')
#定义编码器
latentinputs=tf.keras.layers.Input(shape=(latentDim,))
x=tf.keras.layers.Dense(np.prod(volumeSize[1:]))(latentinputs)
x=tf.keras.layers.Reshape((volumeSize[1],volumeSize[2],volumeSize[3]))(x)
for filter in filters[::-1]:
x=tf.keras.layers.Conv2DTranspose(filter,kernel_size=(3,3),strides=2,padding='same')(x)
x=tf.keras.layers.LeakyReLU(alpha=0.2)(x)
x=tf.keras.layers.BatchNormalization(axis=-1)(x)
x=tf.keras.layers.Conv2DTranspose(depth,(3,3),padding='same')(x)
outputs=tf.keras.layers.Activation('sigmoid')(x)
decoder=tf.keras.Model(latentinputs,outputs,name='decoder')
autoencoder=tf.keras.Model(inputs,decoder(encoder(inputs)),name='autoencoder')
return (encoder,decoder,autoencoder)
autoencoder.summary()The network frame structure is as follows:
3.3, model training
epochs=20
lr=1e-3
batch_size=32
#搭建模型
print("[INFO] building autoencoder...")
(encoder, decoder, autoencoder) = ConvAutoencoder.build(28, 28,1)
#搭建优化器
opt=tf.keras.optimizers.Adam(lr=lr,decay=lr/epochs)
autoencoder.compile(loss='mse',optimizer=opt,metrics=['acc'])
#训练
H=autoencoder.fit(trainX,trainX,validation_data=(testX,testX),epochs=epochs,batch_size=batch_size)
The training process is as follows:
3.4. Model prediction
It mainly uses the self-encoding reconstructed picture and the original picture to calculate the MSE, and then makes the corresponding judgment according to the threshold. The main realization is as follows:
# 加载模型
print("[INFO] loading autoencoder and image data...")
autoencoder = tf.keras.models.load_model("autoencoder.h5")
images = pickle.loads(open("mages.pickle", "rb").read())
#预测图片
images=images.reshape(-1,28,28,1)
images=images.astype('float32')
images/=255
decoded = autoencoder.predict(images)
errors = []
for (image, recon) in zip(images, decoded):
# 计算预测和真实图片之间的均方差1
mse = np.mean((image - recon) ** 2)
errors.append(mse)
# compute the q-th quantile of the errors which serves as our
# threshold to identify anomalies -- any data point that our model
# reconstructed with > threshold error will be marked as an outlier
thresh = np.quantile(errors, 0.999)
idxs = np.where(np.array(errors) >= thresh)[0]
print("[INFO] mse threshold: {}".format(thresh))
print("[INFO] {} outliers found".format(len(idxs)))The test results are as follows:
Detailed code link: https://github.com/kingqiuol/learning_tensorflow/tree/master/cv/Anomaly_detection
Reference link:
https://www.pyimagesearch.com/2020/03/02/anomaly-detection-with-keras-tensorflow-and-deep-learning/
Handwritten digit classification of MNIST dataset based on Keras+CNN