Hundreds of Deep Learning Reading Notes-Video Processing

video processing

video encoder

intra prediction

• • Select the intra-frame coding mode (convolution) through the pixel values ​​of the current image block to be encoded, and then use the selected mode to predict all pixel values ​​​​(HEVC) of the image block to be encoded.
  • Through the surrounding pixel values ​​of the current image block to be encoded, directly predict all pixel values ​​​​of the current image block to be encoded (multi-layer fully connected network).

loop filter

• • It mainly solves the distortion effects such as block effect, ringing effect and color deviation in video reconstruction.
  • Larger reconstruction blocks are selected with overlap, and the reconstruction blocks are enhanced and restored by using a deep convolutional neural network.

Video Surveillance

The balance of video compression and recognition accuracy

• • Compress-Then-Analysis
  • Analysis-Then-Compress
  • Video feature and content joint compression transfer model
  • Face Image Video Compression Algorithm
    • Face Feature Extraction
      • FaceNet
        • The mapping from image pixel space to face feature space is realized by using deep convolutional neural network and ternary loss function.
    • Reconstruction of the basic structure of the face
      • Backbone: Transposed Convolutional Neural Networks
      • Loss function: Linear combination of Mean Absolute Error (MAE) and ReLU layer perceptual error in VGG-19
    • Face Residual Information Compression
      • The original image is compressed with the residual information of the basic layer structure graph.
        • Model based on GDN transformation
        • Traditional Image and Video Compression Algorithms
          • JPEG
          • JPEG2000
          • HEVC

Image Quality Evaluation

Subjective/objective quality evaluation

Full-reference/half-reference/no-reference quality assessment

• • FR-IQA
  • RR-IQA
  • NR-IQA

super-resolution reconstruction

method

• • Interpolation-Based Super-Resolution Reconstruction Method
    • high speed
    • Image details are not well reconstructed (ringing or aliasing occurs)
  • Reconstruction-based super-resolution reconstruction methods
    • frequency domain method
      • Anti-aliasing reconstruction method
    • airspace law
      • Strong ability to include airspace prior constraints
  • Learning-Based Super-Resolution Reconstruction Method
    • algorithm
      • SRCNN
        • Image patch extraction and representation
        • nonlinear mapping
        • reconstruction
      • Raisr
    • Evaluation index
      • Peak Signal-to-Noise Ratio (PSNR)
      • Structural Similarity Index (SSIM)

Improve rebuild speed

• • increase computing resources
  • optimization model
    • The input is changed to the original low-resolution image
    • Use a small convolution kernel (the deconvolution layer enlarges the image, and puts the end to reduce the amount of calculation)

Improve reconstruction effect

• • Deepen the network structure
    • Deep networks can get a larger receptive field
    • Deep networks enable complex nonlinear mappings
  • Optimize the loss function
    • perceptual loss function
      • content loss
      • against loss

Video super-resolution reconstruction

• • Reconstruction with Inter-Frame Correlation
    • motion compensation
      • STN
        • Positioning network (learning the affine transformation parameters from U to V)
        • coordinate generator
        • Sampler
    • affine transformation
    • three-frame fusion

Telecommunication

time series forecasting

• • Differential Integrated Moving Average Autoregressive Model
  • CNN-RNN
    • LSTM
      • Learn about long-term dependencies
    • 3D convolution
      • Learning Geo-Temporal Joint Features
  • GNN
    • Better representation of point-to-point relationships in the network

Adaptive Rate Control

• • difficulty
    • Contradiction of Multiple Optimization Objectives
    • The complexity and variability of the network environment
  • method
    • Bandwidth-Based Rate Adaptive Algorithm
    • Bit rate adaptive algorithm based on client video cache length
    • Pensive

Note: This resource is only for learning and communication, not for private use, otherwise legal responsibility will be investigated. Reprint please attach this link.