Anomaly Detection: Exploring the Mysteries Behind the Deep Layers of Data (Part 1)
1. What is anomaly detection?
Outlier Detection , as the name suggests, is to identify data that is different from normal data and data that is significantly different from expected behavior.
Identifying problems such as credit card fraud, industrial production anomalies, anomalies in network flows (network intrusions), etc., targeting a small number of events.
1.1 Categories of exceptions
Point anomalies refer to a few individual instances that are abnormal and most individual instances are normal, such as the health indicators of normal people and patients;
Conditional anomalies , also known as contextual anomalies, refer to individual instances that are abnormal in a specific situation and are normal in other situations. For example, the temperature suddenly rises or drops at a specific time. In a specific scenario, for fast credit card transactions;
Group anomalies refer to situations where an individual instance in a group collection is abnormal, and the individual instance itself may not be an anomaly. In applications such as intrusion or fraud detection, outlier points correspond to sequences of multiple data points. , rather than a single data point. For example, a set formed by fake accounts in a social network serves as an abnormal subset of the group, but individual nodes in the subset may be as normal as real accounts.
1.2 Classification of anomaly detection tasks
Supervised : both positive and negative examples in the training set are labeled
Unsupervised : The training set has no labels
semi-supervised