Anomaly detection for target recognition - wandering detection project

In-depth research on abnormal behavior detection

1. Research Status of Abnormal Behavior Detection

1.1 Overview of Abnormal Behavior Video Analysis

1.1.1 Abnormal Behavior Analysis Research Framework

Review literature:

A Review on Video-Based Human Activity Recognition
https://github.com/hoya012/awesome-anomaly-detection

In general, human activity recognition can be divided into three levels of representation, which are low-level core technology, middle-level human activity recognition system, and high-level applications, as shown in Figure 1, considering three main processing stages, namely object segmentation , feature extraction and representation, and activity detection and classification algorithms. Human subjects are first segmented from video sequences. Then, features of human objects such as shape, contour, color, pose and body motion can be correctly extracted and represented by a set of features. Subsequently, activity detection or classification algorithms are applied to the extracted features to identify various human activities. Furthermore, in the second-level human activity recognition system, three important recognition systems are discussed, including single-person activity recognition, multi-person interaction and crowd behavior, and abnormal activity recognition. Finally, Level III applications discuss recognized results applied in surveillance settings, entertainment settings, or healthcare systems.

1.1.2 Abnormal Behavior Summary

1.2 Abnormal behavior monitoring research

1.2.1 Multi-Resolution Semantic Activity Representation and Anomaly Discovery in Video

(1) Model : LDS1: YOLOv3 + DeepSORT

(2) Dataset : CAVIAR dataset

(3) Idea : A novel method is proposed to characterize and analyze activities at different resolutions. Semantic information is conveyed according to the resolution at which the activity is observed. Furthermore, multi-resolution activity representations are utilized to detect anomalous activities. This paper proposes a multi-resolution method for automatically characterizing human activities, conveying such activities using semantic terms and detecting anomalous activities, such as people staying in an area unusually for a long time, loitering or taking low-frequency paths. The main innovation of the method is that region-based multi-resolution features are exploited to adjust semantic interpretation when needed and to extract statistics of region occupancy and transfer between active regions to establish statistical thresholds that can distinguish between normal/ unusual activity.

（4）文献：Multiresolution semantic activity characterisation and abnormality discovery in videos

(5) Code : https://github.com/ntienvu/abnormal_detection_video_surveillance

1.2.2 Bayesian Nonparametric Approaches to Anomaly Detection in Video Surveillance

(1) Model :

(2) Dataset :

(3) Idea : We propose a framework for nonparametric data segmentation and multimodal anomaly detection. By building multiple anomaly detection models on different coherent parts of streaming data, our proposed framework is more robust to anomaly detection in large-scale video data. Especially when the cameras are monitored over many days and show large data changes. Our experiments on video data collected over 14 days demonstrate the higher performance of the proposed multi-modal anomaly detector compared to single-modal detectors. The main contributions of this paper are: (1) the proposed Infinite Hidden Markov model for streaming data segmentation, and (2) the introduction of a user interface for feature extraction using Rank-1 robust PCA and Bayesian nonparametric factor analysis for pattern Discovery, which allows users to inspect and browse suspicious anomalies.

（4）文献：Bayesian Nonparametric Approaches to Abnormality Detection in Video Surveillance

1.2.3 Abnormal Event Detection at 150 FPS in MATLAB

(1) Model : sparse combination learning

(2) Dataset : CAVIAR dataset

(3) Idea : We propose an anomalous event detection method via sparse ensemble learning. This approach directly learns sparse combinations, speeding up testing hundreds of times without compromising effectiveness. Our method achieves state-of-the-art results in several datasets. It is related to traditional subspace clustering, but mainly differs from traditional subspace clustering

(4) Reference : Abnormal Event Detection at 150 FPS in MATLAB

(5) Code :

2. Overview of Abnormal Behavior Detection Algorithms

2.1 Algorithm for loitering behavior detection

2.1.1 Deep learning based loitering detection system using multi-camera video surveillance network

(1) Model : LDS1: YOLOv3 + DeepSORT

(2) Dataset : PETS 2007

(3) Idea : A deep learning-based loitering detection system (LDS) with re-identification (ReID) on a multi-camera network is proposed. The proposed LDS mainly includes object detection and tracking, loitering detection, feature extraction, camera switching, and rogue re-identification. The person is detected using Look Only Once (YOLOv3) and tracked using Simple Online Real-Time Tracking (DeepSORT) with a deep association matrix. From trajectory analysis, once the time and displacement thresholds are met, the person is considered a rogue.

（4）文献：Deep Learning based Loitering Detection System using Multi-camera Video Surveillance Network

2.1.2 Loitering detection based on pedestrian activity area classification

(1) Model : GMM+Meanshift

(2) Dataset : PETS 2007

(3) Idea : This paper first uses the size of the pedestrian activity area to give a definition of wandering from a new perspective. Pedestrian loitering behavior is divided into three categories. The proposed algorithm dynamically calculates the rectangle, ellipse and sector of the pedestrian active area by curve fitting based on the trajectory coordinates within a given dwell time threshold. Loitering is identified if restrained pedestrian activity is detected within a certain period of time.

（4）文献：Loitering Detection Based on Pedestrian Activity Area Classification

2.1.3 Entropy Model for Identifying Video Loitering Behavior

(1) Model : Heat map + Entropy Model

(2) Dataset : PETS 2007

(3) Idea : Entropy theory can be used to detect irregular behavior in crowds. Most of the previous work on loitering detection only provides classification of loafers. For this, some time threshold or score is involved. It is not easy to choose this threshold in practice. For example, suspicious behavior of pedestrians staying in the area below this time threshold will not be detected. Also, they make no difference between someone who is moving and someone who is waiting. When analyzing a large number of shots, there may be many potential rogues. Fine-tuning alerts based on time thresholds will be difficult. We've rethought our attempt to classify rogues by providing a list of candidates, with the most likely rogues at the top. This way, the operator can make the final judgment. Our method retrieves rogues and provides other suspicious candidates in an ordered list sorted by suspiciousness. We apply entropy theory to individuals to determine how much they move.

（4）文献：An Entropy Model for Identifying Loitering Behavior in Videos

2.1.4 Markov random walk model for loitering detection

(1) Model : Markov + Random Walk

(2) Dataset : PETS 2007

(3) Idea : Based on the spatio-temporal co-occurrence and distribution of trajectories, we propose a Markov random walk model using appearance and motion features.

（4）文献：A Markov Random Walk Model for Loitering People Detection

2.1.5 Spotting loitering in public transport areas

(1) Model :

(2) Dataset : Bus Stop video

(3) Idea : This paper proposes a vision-based approach to automatically detect people wandering in urban bus stops. Using a still camera view of a bus stop, pedestrians are segmented and tracked throughout the scene. When it finds an unobstructed view of pedestrians, the system takes a personal snapshot. Then, using an appearance-based approach, the snapshots are used to classify individual images into the database. The features used to correlate individual images are based on short-term biometrics that can change but remain valid for short periods of time; the system uses clothing color. A linear discriminative method is applied to color information to enhance the differences between different individuals and minimize the similarity between them in the feature space. To determine whether a given individual is loitering, the timestamps collected with snapshots in its corresponding database class can be used to determine how long a person has existed. An experiment was conducted using a 30 minute video of a busy bus stop with 6 people loitering about it. The results showed that the system successfully classified images of all six people as loitering.

（4）文献：Detection of Loitering Individuals in Public Transportation Areas

2.1.6 Avenue dataset for abnormal event detection

(1) Model :

(2) Dataset : Avenue Dataset ( http://www.cse.cuhk.edu.hk/leojia/projects/detectabnormal/dataset.html )

(3) Thought :

（4）文献：Avenue Dataset for Abnormal Event Detection(http://www.cse.cuhk.edu.hk/leojia/projects/detectabnormal/index.html)

(5) Code : https://github.com/gongruya/abnormality-detection

3. Anomaly Detection Framework and Dataset

3.1 Commonly used video anomaly analysis and detection datasets

[1] A Review on Video-Based Human Activity Recognition

3.2 A discriminative framework for large-scale video anomaly detection

(1) Model : unsupervised

(2) Dataset :

Avenue Dataset(http://www.cse.cuhk.edu.hk/leojia/projects/detectabnormal/dataset.html)
Subway surveillance dataset
Personal Vacation Dataset
UMN Unusual Activity Dataset

(3) Ideas : Contributions include a new anomaly detection framework that is (1) independent of the temporal order of anomalies, and (2) unsupervised, requiring no separate training sequences. We show that our algorithm achieves state-of-the-art results even when we tweak the settings to remove training sequences from standard datasets.

（4）文献：A Discriminative Framework for Anomaly Detection in Large Videos

(5) Code :

https://github.com/alliedel/videofeatures ；
https://github.com/alliedel/anomalyframework_ECCV2016 (matlab & C++);
https://github.com/alliedel/anomalyframework_python (Python & C++)

3.3 Multi-time-scale trajectory prediction for human activity anomaly detection

(1) Model :

（2）数据集：IV Corridor dataset（https://drive.google.com/drive/folders/1Rxm-4fPqhfVPB9HINGtbdvauYpQuCD5i?usp=sharing）

(3) Thought : In the surveillance scene, jumping is a short-term abnormality, while wandering is a long-term abnormality. A single, pre-determined time scale is insufficient to capture a wide variety of anomalies that occur over different temporal durations. In this paper, we propose a multi-timescale model to capture temporal dynamics at different timescales. In particular, the proposed model makes future and past predictions at different time scales for a given input pose trajectory. The model is multi-layered, where intermediate layers are responsible for generating predictions for different time scales. These predictions are combined to detect unusual activity. In addition, we also introduce a research anomalous activity dataset, which contains 4,83,566 annotated frames. Our experiments show that the proposed model can capture anomalous activities of different time lengths and outperforms existing methods.

（4）文献：Multi-timescale Trajectory Prediction for Abnormal Human Activity Detection

(5) Code :

https://rodrigues-royston.github.io/Multi-timescale_Trajectory_Prediction/

3.4 Future Frame Prediction for Anomaly Detection – A New Baseline

(1) Model : YOLOv3 + Future
(2) Dataset :
- UCSD dataset
- Avenue dataset
- ShanghaiTech dataset

(3) Idea : Anomaly detection in video refers to the identification of events that do not conform to expected behavior. However, almost all existing methods address this problem by minimizing the reconstruction error of the training data, which cannot guarantee a large reconstruction error for anomalous events. In this paper, we propose to address the anomaly detection problem within a video prediction framework. To the best of our knowledge, this is the first work to exploit the difference between predicted future frames and their ground truth to detect anomalous events. In order to make higher-quality predictions for future frames of normal events, in addition to the usual appearance (spatial) constraints of intensity and gradients, we also introduce motion (temporal) constraints in video prediction, enforcing both the predicted frame and the ground truth frame It is also the first work to introduce temporal constraints into video prediction tasks. Such spatial and motion constraints facilitate future frame prediction of normal events, and thus facilitate the identification of anomalous events that do not conform to expectations. Extensive experiments on a toy dataset and some publicly available datasets verify the effectiveness of our method in terms of uncertainty about normal events and sensitivity to abnormal events.
（4）文献： Future frame prediction for anomaly detection–a new baseline
(5) Code : https://github.com/StevenLiuWen/ano_pred_cvpr2018

3.5 Online Anomaly Detection of Surveillance Video with Asymptotic Bound of False Alarm Rate

(1) Model : YOLOv3 + Statistic
(2) Dataset :
- UCSD dataset
- Avenue dataset
- ShanghaiTech dataset

(3) Idea : Our proposed algorithm consists of a multi-objective deep learning module and a statistical anomaly detection module . Its efficiency has been demonstrated on several public datasets, and our performance outperforms the state-of-the-art algorithms.
（4）文献：Online Anomaly Detection in Surveillance Videos with Asymptotic Bounds on False Alarm Rate
(5) Code :
- https://github.com/kevaldoshi17/Prediction-based-Video-Anomaly-Detection-
- https://github.com/StevenLiuWen/ano_pred_cvpr2018

3.6 Video Anomaly Event Detection Using Spatiotemporal Autoencoders

(1) Model : LSTM

(2) Dataset :

UCSD dataset
Avenue dataset
Subway dataset

(3) Idea : We propose an eﬁcient method for detecting anomalies in videos. Recent applications of convolutional neural networks have shown the promise of convolutional layers for object detection and recognition, especially in images. However, convolutional neural networks are supervised and require labels as learning signals. We propose a spatio-temporal architecture for anomaly detection in videos including crowded scenes. Our architecture consists of two main parts, one for spatial feature representation and one for learning the temporal evolution of spatial features. Experimental results on Avenue, Subway, and UCSD benchmarks demonstrate that our method achieves detection accuracy comparable to state-of-the-art methods at up to 140 fps.

（4）文献：Abnormal Event Detection in Videos using Spatiotemporal Autoencoder

(5) Code :

4. Loitering Behavior Detection Dataset

PETS 2007 : The dataset is a multi-sensor sequence of increasing scene complexity consisting of three scenarios: loitering, attended baggage handling (theft) and unattended baggage.

Loitering Loitering is defined as a person who enters the scene,and remains within the scene for more than t seconds. For the purposes of PETS 2007, t = 60 seconds.

For other research on abnormal video analysis of the PET2007 dataset, please refer to the literature: Tenth IEEE International Workshop on Performance Evaluation of Tracking and Surveillance (PETS 2007)

UCSD dataset

UCSD: The pedestrian dataset of UCSD consists of two parts, Ped1 and Ped2 . We excluded Ped1 from our experiments due to its significantly lower resolution of 158 x 238 and the lack of consistency in the reported results as some recent works only reported performance on a subset of the entire dataset. The UCSD Ped2 dataset consists of 16 training videos and 12 testing videos, each with a resolution of 240×360. All of the anomalous incidents were caused by vehicles such as bicycles, scooters and wheelchairs crossing pedestrian areas.

An on-line, real-time learning method for detecting anomalies in videos using spatio-temporal compositions.

W. Liu, W. Luo, D. Lian, and S. Gao. Future frame prediction for anomaly detection–a new baseline. 2017.

MIT trajectory dataset

Multiresolution semantic activity characterisation and abnormality discovery in videos.

Avenue dataset

The Zhongda Avenue dataset consists of 16 training videos and 21 testing videos with a frame resolution of 360×640. Anomalous behavior manifests as people throwing objects, loitering, and running.

ShanghaiTech dataset

The ShanghaiTech University Campus dataset is currently one of the largest and most challenging datasets in video anomaly detection, consisting of 330 training videos and 107 testing videos from 13 diﬁerent scenes. It consists of 330 training videos and 107 testing videos from 13 diﬁerent scenes, which distinguishes it from other available datasets. Each video frame has a resolution of 480 x 856.

mall_1, mall_2 dataset

Shopping mall dataset

Subway exit dataset

British subway import and export