Hyperspectral Image Classification
During my postgraduate study, my research direction is deep learning (image classification, object detection). The tutor gave related projects for the classification of hyperspectral images. Since it involves too much unfamiliar knowledge related to spectroscopy that is inconsistent with the research direction, the main purpose of writing this blog post is to review some of the knowledge accumulated a while ago. At the same time Share and exchange knowledge and skills of hyperspectral images and subsequent related deep learning algorithms with everyone.
Table of contents
1. Definition of hyperspectral image
Different from traditional optical RGB three-channel color images, hyperspectral has many channels in the spectral dimension due to its high spectral resolution, and each channel captures a specific wavelength. The most notable feature of hyperspectral images: "integration of maps and spectra", which contains rich spatial and spectral information, and provides spatial domain information and spectral domain information. Among them, in the spatial dimension of the image, the hyperspectral data is similar to the traditional optical color image; in the spectral dimension, the spectral curve characteristics of each pixel on the image are revealed, which can be obtained in any pixel of the hyperspectral image A "continuous" spectral curve. Image fetched from Google web page
2. Based on the principle of spectral classification
The essential characteristics of a substance that distinguish it from other substances are determined by the types and arrangements of the molecules and atoms that make up the substance. When electromagnetic waves are incident on the surface of a material, the internal molecules and atoms vibrate to form absorption and reflection characteristics at specific wavelengths. Most surface features have typical spectral absorption characteristics. According to the different chemical compositions of different surface features, they have different absorption and reflection conditions on the spectrum, which can reflect the difference of substances. The same substance has different performances under different band spectral signals. Based on the difference of the curves, different substances in the hyperspectral image can be classified.
3. Some problems of classifying hyperspectral images based on deep learning
3.1 Band Redundancy
Due to the power of "integration of graphs and spectra", hyperspectral images show an exponential increase in the amount of data as the number of bands in the spectral dimension increases. Therefore, the information redundancy is high, and there will be information overlap between adjacent bands, which will increase the calculation amount of the model, bring great difficulties to image classification and recognition, and bring great difficulties to the accurate classification of network models. performance impact.
3.2 Hughes phenomenon
First explain the concept of this phenomenon: In hyperspectral image analysis, as the number of spectral bands involved in the calculation increases, the classification accuracy of the algorithm appears to "increase first and then decrease". Because the dimensions of hyperspectral images have increased significantly, the number of feature dimensions that the network model needs to extract will also increase, and the number of data set samples of hyperspectral images is less than that of traditional optical images, so it cannot provide enough training samples to generate Hughes phenomenon.
3.3 Intra-class variation and inter-class similarity
Due to the high spatial resolution of HSI, the rich information in the image increases intra-class variation, such as: in hyperspectral images, shaded roofs; also increases the similarity and confusion between classes, such as: road and Roofs are similar in the spectral domain.
3.4 "Same substance with different spectrum" / "Same spectrum with different substance"
The mechanism and action process from ground object to image response are extremely complex, even for the same material, its spectrum will show differences.
4. Preprocessing method
4.1 Band Selection
Select the bands that are interesting or have the characteristics of obvious objects from several bands, that is, select the subset of bands that play a major role from all bands. In this way, the inference time and calculation amount of the model can be reduced, and some bands that are useless to the research can be eliminated. The picture below is an excerpt from a core journal of Peking University . Interested friends can click in and take a look. In the figure, the PCA algorithm is used to reduce the dimensionality of the blueberry hyperspectral image, and the first 7 principal component images are obtained. The pest characteristics shown in the PC1 image are not obvious, and the principal component images after PC4 have obvious noise compared with the previous principal component images. PC2 and PC3 are relatively detailed, so the PC2 and PC3 data sets are selected for subsequent experiments.
4.2 Through mathematical transformation
Data compression is performed on all bands through mathematical transformation, but such a method may change the physical meaning of the original data and is not conducive to algorithm learning.
4.3 Image correction and restoration (for hyperspectral images)
Used to correct for geometric and radiometric anomalies in data caused by sensors, etc. Ambient lighting, atmosphere, environment, and sensor noise may require radiometric correction. Generally, black and white correction is performed on hyperspectral images to remove noise, which can improve the accuracy of subsequent algorithm classification. formula:
4.4 Segmentation to obtain hyperspectral images of objects
For the black-and-white corrected image, the Otsu method can be used to binarize the image, and the morphological algorithm can be used to remove noise points and fill holes to obtain a mask image. Next, an AND operation is performed on the mask image and the corrected spectral image to obtain the hyperspectral image of the object after the background has been removed. After segmentation, multiple scattering (MSC) can be used to process the graphics to eliminate the influence of scattering.
5 main ways of deep feature extraction network
5.1 Preprocessing-based networks
Spectral and spatial features are subjected to feature fusion before being fed into the network model. The general steps are: feature fusion of low-dimensional spectral and spatial features, high-dimensional feature extraction based on deep neural network, and finally classification using a classifier
5.2 Integration-based networks
Instead of taking spectral and spatial features separately and processing them together. Instead, the two-dimensional convolutional neural network is directly used to directly extract joint spectral spatial features directly from the original data. Since hyperspectral data is a 3D cube, 3D convolutions can also be used.
5.3 Post-processing based networks
Two deep neural networks are used to extract spectral features and spatial features respectively, and these two features are fused in the fully connected layer to generate joint spectral and spatial features, and finally a classifier is used for classification.
6. Hyperspectral unmixing
In theory, there should be only one spectral curve for each type of ground object, and the reason why each type of ground object has multiple intra-class spectra:
mainly because the area corresponding to one pixel in the mixed pixel contains two or more characteristic ground object, the final spectral curve displayed by the pixel is actually superimposed by the characteristic spectral information of various surface objects. This superposition is further divided into two types of mixing: linear mixing and nonlinear mixing. The process of hyperspectral unmixing is a problem of decomposing a mixed pixel into endmembers and corresponding abundances. That is, the process of decomposing each such mixed pixel in this image into an endmember m and its corresponding proportional abundance a. In the pixels of
this picture , there are mainly three types of features: land, road, and water. Mixed pixel decomposition is based on the information provided by the image to determine which pure pixels (end members) and how they are mixed (corresponding abundance) in each mixed pixel.