Anomalib image anomaly detection algorithm

1 Introduction

In industrial production, quality assurance is crucial because small defects in production may lead to substandard products and even harm the interests of consumers and businesses. Industrial anomaly detection is a critical task that aims to reliably detect anomalies from large-scale production data, which may be defects, failures, or other anomalies. Here are more details on industrial anomaly detection:

1.1. Problem description

In industrial production, products and production processes are usually monitored and controlled to ensure product quality and production efficiency. However, there are still abnormalities in the production process due to various factors, such as material variation, equipment failure, operating errors, etc. These anomalies may include defects in the product, malfunctions in the production line, abnormal process parameters, and other deviations from normal production.

1.2. Challenges and needs

Industrial anomaly detection faces several challenges that largely distinguish it from traditional supervised learning problems:

• Difficulty in obtaining a large number of abnormal samples: Abnormalities are usually relatively rare in industrial production because they are not the norm. In contrast, data from normal conditions usually dominate, making anomaly detection an unsupervised learning problem.
• The difference between normal samples and abnormal samples is small: Normal products and abnormal products may not have obvious differences in characteristics, and abnormalities may show relatively weak characteristics in details.
• The type of exception cannot be known in advance: In some cases, the type and nature of the exception may not be known a priori, so the algorithm needs to be flexible enough to Detect various types of anomalies.

2. Image anomaly detection definition

2.1 Definition and types of exceptions

Anomaly, also known as outlier, is a common concept in the field of data mining. There have been many works trying to define abnormal data. Generally, common anomaly samples are divided into three categories: point anomalies, context anomalies and cluster anomalies.

• Point anomalies generally manifest as some observed values ​​that seriously deviate from the normal data distribution range, as shown below$The\; two-dimensional\; data\; points\; shown\; in\; (a)$, which deviate from the normal sample points Distribution area$(N_1,N_2)$Point$(O_1,O_2,O_3)$ is the abnormal point.
• The context anomaly is represented by the fact that although the observation value is within the normal data distribution range, it will show significant anomalies when analyzed together with the surrounding data. As shown below$(b)$所示，$t_2$Although the temperature value at the point is still within the normal range, combining the data of the two months before and after it can be found that the point is abnormal data.
• Cluster anomalies, also known as pattern anomalies, are anomaly types that are aggregated from a series of observation results and are different from normal data. In this type of anomaly, any one point may not be an anomaly when viewed alone, but when a series of points appear together, it is an anomaly, as shown below$\left(c\right)$The value of each point individually in the area pointed by the arrow is normal. within the range, but these points come together to form a structure that is completely different from the normal signal pattern.

2.2 Anomalies in image data

The pixel value of each pixel in the image data corresponds to an observation result. Due to the diversity of pixel values ​​in the image, it is difficult to judge whether it is an anomaly by only analyzing the pixel value of a certain point. Therefore, in most image anomaly detection tasks, it is necessary to jointly analyze the image background and pixel information for classification. Most of the detected anomalies are contextual or pattern anomalies: of course, there are no very strict boundaries between such anomaly types. For example, , some methods extract various features of the image and compare them with the features of the normal image to determine whether they are abnormal. This converts the detection of pattern abnormalities in the original image space to the detection of point abnormalities in the feature space. .

Image anomaly detection tasks can be divided into two categories: qualitative anomaly classification and quantitative anomaly localization according to the shape of the anomaly.

• Classification of qualitative anomalies: Similar to the image classification task in traditional image recognition tasks, that is, an overall judgment of whether there is an abnormality is given, without accurately locating the location of the abnormality. As shown below As shown in the upper left figure, the left side represents normal images, and the stone side represents abnormal images. In the first row, the model only uses samples of clothing types in the clothing data set for training, and sample images of other categories (shoes, etc.) are not suitable for the model. They are all abnormal samples that need to be detected, because they are different in terms of texture, structure and semantic information, or as shown in line 2, there is only an overall shift between the transistors in the abnormal image and the normal image. However, there is no local abnormal area on the surface of the transistor, so it is difficult to accurately define the location of the abnormality, and it is more suitable to classify whether it is abnormal or not as a whole.
• Quantitative anomaly positioning: Similar to target detection or image segmentation tasks, it is necessary to obtain the location information of the abnormal area. In this type of image anomaly detection, only a small area of ​​the test image has abnormal patterns. The anomaly localization task can be divided into the following categories according to the specific image background, as shown in the figure below, which is circled Part of it is an abnormal area.
• • Uniform background
    Uniform background refers to some scenes with relatively single content, such as the positioning of local defects on the frosted glass surface as shown in the figure below, or the positioning of winding mountains in dark mountainous images. Highway positioning and anomaly detection in this context are relatively simple detection tasks.
• • Texture background
    Texture background mainly appears in the field of industrial production. According to the texture form, it can be divided into three types: simple texture, periodic texture and random texture. Among them, simple texture refers to some non-uniform texture backgrounds produced on the surface of an originally uniform object due to factors such as lighting and material reflection, such as the steel plate surface image below. Periodic texture refers to various types of images with significant periodicity composed of a large number of repeating units. The most representative ones are the various types of cloth images shown in the figure below. Random texture refers to some image backgrounds composed of irregular structures, such as the sonar and nanomaterial images shown below.
• • Structural background
    Structural background is a broader type of image background. It generally has the characteristics of complex structure, large individual differences and rich semantic information. It needs to be analyzed as a whole rather than just Anomaly detection is performed based on certain local information, such as various medical images as shown in the figure below. The anomaly detection problem in the context of such images is a relatively difficult detection task.
    

3. Current research status of image anomaly detection technology

Generally, the goal of image anomaly detection is to detect abnormal images or local abnormal areas that are different from normal images through unsupervised or semi-supervised learning. In recent years, traditional machine learning methods have been used in the field of image anomaly detection. With the development of deep learning technology, more and more methods are trying to combine neural networks to detect image anomalies. According to whether neural networks are involved in the model construction stage, existing image anomaly detection methods can be divided into two categories: traditional methods and deep learning-based methods.

As shown in the figure below, anomaly detection technology based on traditional methods roughly includes 6 categories:

• Template based matching
• Based on statistical models
• Based on image decomposition
• Based on frequency domain classification
• Reconstruction based on sparse coding
• Built based on classification surface

Methods based on deep learning roughly include 4 categories:

• Based on distance metric
• Built based on classification surface
• Based on image reconstruction
• combined with traditional methods

4.Method

In order to solve the problem of image anomaly detection, a method of unsupervised learning is introduced here. Unsupervised learning uses deep learning technology, such as convolutional neural network (CNN), generative adversarial network (GAN) and autoencoder (AE), etc., for unsupervised anomaly detection and localization. These methods typically leverage deep learning models to learn representations of the data and use these representations to calculate anomaly scores or perform pixel-level anomaly localization. They are part of anomaly detection methods based on deep learning.

Unsupervised image anomaly detection and localization methods can be divided into discriminative methods and generative methods. Some of these algorithms are described in more detail below

Discrimination method:

• CFlow: CFlow is a method based on density ratio, which uses conditional density ratio estimation to identify abnormal points. This method relies on the calculation of density ratios to distinguish between normal and abnormal data points.
• DFKDE: DFKDE (Density-Ratio Based Kernel Density Estimation) is a density ratio-based method that identifies anomalies by estimating the ratio between two conditional density functions point.
• PatchCore: The PatchCore method uses a convolutional neural network (CNN) to learn a representation of the data, and then uses a clustering method in the representation space to locate abnormal areas.
• STFPM: STFPM (Spatial-Temporal Fusion Proximity Map) is an anomaly detection method for video data that combines spatial and temporal information to identify Abnormal area.

Generating method:

• GANomaly: GANomaly combines the ideas of Generative Adversarial Network (GAN) and Autoencoder to detect abnormal data points through generation and reconstruction. Abnormal data points generally cannot be disguised by generative models.
• Draem: Draem (Deep Recurrent Autoencoder for Extreme Multiclass Classification) is a deep recursive autoencoder used to detect and locate outliers. It performs well in large-scale multi-class anomaly detection.
• Padim: Padim (Part-based Anomaly Detection for Images) is an image anomaly detection method that decomposes the image into parts and detects the abnormal parts to achieve abnormal location .
• Reverse Distillation: The Reverse Distillation method uses the idea of ​​knowledge distillation to transfer the knowledge of large models to small models to identify abnormal data points.
• Fast-Flow: Fast-Flow is a method of flow field estimation on images, which locates abnormal areas in the image by detecting pixel-level flow field anomalies.

So how to get started using so many algorithms faster? Here is an introduction to a python anomaly detection library anomalib

github：https://github.com/openvinotoolkit/anomalib

5.Installation and use

Anomalib is a deep learning library designed to collect state-of-the-art anomaly detection algorithms for benchmarking on public and private datasets. Anomalib provides several ready-to-use implementations of anomaly detection algorithms described in recent literature, as well as a set of tools that facilitate the development and implementation of custom models. The library focuses on image-based anomaly detection, where the goal of the algorithm is to identify abnormal images or abnormal pixel areas of images in a dataset.

5.1 Installation

This is a tutorial on the installation and use of the "anomalib" library

PyPI installation

You can easily start using anomalib by using pip.

pip install anomalib

Local installation

It is highly recommended to use a virtual environment when installing anomalib. For example, using Anaconda, you can install anomalib as follows:

yes | conda create -n anomalib_env python=3.10
conda activate anomalib_env
git clone https://github.com/openvinotoolkit/anomalib.git
cd anomalib
pip install -e .

5.2 Training

By default, python tools/train.py runs the PADIM model on the leather class of the MVTec AD (CC BY-NC-SA 4.0) dataset.

python tools/train.py    # 在MVTec AD的leather上训练PADIM

Training a model on a specific dataset and category requires further configuration. Each model has its own configuration file config.yaml, which contains data, model and training configurable parameters. To train a specific model on a specific dataset and category, you need to provide the corresponding configuration file:

python tools/train.py --config <path/to/model/config.yaml>

For example, to train PADIM, you can use the following command:

python tools/train.py --config src/anomalib/models/padim/config.yaml

Alternatively, you can provide the model name as a parameter and the script will automatically find the corresponding configuration file.

python tools/train.py --model padim

Currently available models:

• CFA
• CFlow
• DFKDE
• DFM
• DRAEM
• EfficientAd
• FastFlow
• GANomaly
• DO NOT TELL ME
• PatchCore
• Reverse Distillation
• STFPM

5.3 Feature extraction and (pre-training) backones

Pretrained backbone models come from PyTorch image models (timm), which are wrapped by FeatureExtractor.

Configuring the backbone model can be set in the configuration file, here are two examples:

model:
  name: cflow
  backbone: wide_resnet50_2
  pre_trained: true

5.4 Custom data sets

It is also possible to train on custom folder datasets. To do this, you need to modify the data section of the config.yaml file as follows:

dataset:
  name: <name-of-the-dataset>
  format: folder
  path: <path/to/folder/dataset>
  normal_dir: normal # name of the folder containing normal images.
  abnormal_dir: abnormal # name of the folder containing abnormal images.
  normal_test_dir: null # name of the folder containing normal test images.
  task: segmentation # classification or segmentation
  mask: <path/to/mask/annotations> #optional
  extensions: null
  split_ratio: 0.2 # ratio of the normal images that will be used to create a test split
  image_size: 256
  train_batch_size: 32
  test_batch_size: 32
  num_workers: 8
  normalization: imagenet # data distribution to which the images will be normalized: [none, imagenet]
  test_split_mode: from_dir # options: [from_dir, synthetic]
  val_split_mode: same_as_test # options: [same_as_test, from_test, sythetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)
  transform_config:
    train: null
    val: null
  create_validation_set: true
  tiling:
    apply: false
    tile_size: null
    stride: null
    remove_border_count: 0
    use_random_tiling: False
    random_tile_count: 16

By placing the above configuration into the data section of the config.yaml file, the model will be trained on the custom dataset.

5.5 Reasoning

Anomalib includes multiple inference scripts, including Torch, Lightning, Gradio, and OpenVINO reasoners, for performing inference using trained/exported models. In this section, we describe how to use these scripts for inference.

PyTorch inference

# 要获得有关参数的帮助，请运行：
python tools/inference/torch_inference.py --help

# 示例Torch推理命令：
python tools/inference/torch_inference.py \
    --weights results/padim/mvtec/bottle/run/weights/torch/model.pt \
    --input datasets/MVTec/bottle/test/broken_large/000.png \
    --output results/padim/mvtec/bottle/images

Lightning reasoning

# 要获得有关参数的帮助，请运行：
python tools/inference/lightning_inference.py --help

# 示例Lightning推理命令：
python tools/inference/lightning_inference.py \
    --config src/anomalib/models/padim/config.yaml \
    --weights results/padim/mvtec/bottle/run/weights/model.ckpt \
    --input datasets/MVTec/bottle/test/broken_large/000.png \
    --output results/padim/mvtec/bottle/images

OpenVINO inference

To run OpenVINO inference, you need to first export your PyTorch model into an OpenVINO model. Make sure to set export_mode to "openvino" in the corresponding model config.yaml file.

# 示例OpenVINO配置文件config.yaml
optimization:
  export_mode: "openvino" # 选项: openvino, onnx

# 要获得有关参数的帮助，请运行：
python tools/inference/openvino_inference.py --help

# 示例OpenVINO推理命令：
python tools/inference/openvino_inference.py \
    --weights results/padim/mvtec/bottle/run/openvino/model.bin \
    --metadata results/padim/mvtec/bottle/run/openvino/metadata.json \
    --input datasets/MVTec/bottle/test/broken_large/000.png \
    --output results/padim/mvtec/bottle/images

Make sure to provide the metadata.json path when normalization needs to be applied correctly.

Gradio reasoning

Interact with the trained model using Gradio inference, using the user interface. See our guide for more details.

# 要获得有关参数的帮助，请运行：
python tools/inference/gradio_inference.py --help

# 示例Gradio推理命令：
python tools/inference/gradio_inference.py \
    --weights results/padim/mvtec/bottle/run/weights/model.ckpt \
    --metadata results/padim/mvtec/bottle/run/openvino/metadata.json  \ # 可选
    --share  # 可选，用于共享UI

5.6 Hyperparameter optimization

To run hyperparameter optimization, use the following command:

python tools/hpo/sweep.py \
    --model padim --model_config ./path_to_config.yaml \
    --sweep_config tools/hpo/sweep.yaml

5.7 Benchmark testing

To collect various benchmark data, such as throughput across categories, use the following command:

python tools/benchmarking/benchmark.py \
    --config <relative/absolute path>/<paramfile>.yaml

5.8 Experiment management

Anomalib integrates with various experiment tracing libraries, such as Comet, tensorboard and wandb, through the PyTorch Lightning logger.

Here is a log example of how to enable hyperparameters, metrics, model plots, and image predictions in the test dataset:

visualization:
  log_images: True # log images to the available loggers (if any)
  mode: full # options: ["full", "simple"]

 logging:
  logger: [comet, tensorboard, wandb]
  log_graph: True