Introduction to PaddleVideo and detailed explanation of file directory

Introduction

PaddleVideo aims to create a rich, leading and practical video tool library , aiming to help developers better conduct academic research and industrial practice in the video field .

PaddleVideo is a paddle-based video understanding tool library that supports video data annotation tools, lightweight RGB and skeletal point behavior recognition models, and practical applications such as video labeling and motion detection.

URL: https://github.com/PaddlePaddle/PaddleVideo/tree/develop

characteristic

It supports a variety of video-related cutting-edge algorithms , and builds industrial-level characteristic models PP-TSM and PP-TSMv2 on this basis , and opens up the entire process of data production, model training, compression, and forecast deployment .

license

Posting of projects is subject to Apache 2.0 licenselicense certification.

PaddleVideo file directory overview

• .github/ ISSUE_TEMPLATEFolder: Store GitHub issue templates .

An issue template is a format for standardizing questions or suggestions , which can help developers and users communicate and solve problems more effectively.
Different projects may have different issue templates, which are generally placed in a folder under the project root directory , named starting with ISSUE_TEMPLATE..github

For example, there are two issue templates in the PaddleVideo project:

• Bug report: Used to describe the problems encountered in the study, including problem description, reproduction steps, expected effects, screenshots, environment version notes, etc.
• Feature request: Used to help maintain the effectiveness of the tutorial, including problem descriptions, suggested solutions, other ideas, other content, etc.

• benchmark/TimeSformerFolder: stores the code and configuration of the transformer-based behavior recognition model TimeSformer .

The here benchmarkindicates a way of evaluating, comparing the quality or quantity of something with a known standard.
This benchmarkfolder stores the code and configuration of TimeSformer, a transformer-based behavior recognition model, for performance comparison with other models .

• data: Store example files of various datasets , such as ucf101, ntu-rgb-d, fsd10datasets; and example of various file formats , such as example.pkl, example.avi, example_feat.npy.
• deploy: store deployment -related code and files.
• test_tipc: store test- related codes and files.
• tools: Store tool related codes and files.
• output: Store training output weights, optimizer parameters and other files .
• log: store the training log .
• inference: store the prediction model .

applications folder details

Store video application- related codes and data .

You can quickly find the project you want to do according to the application cases in this folder, such as $FigureSkatingfigureskating$ .

The application case is shown in the figure below:

configs folder details

Contains configuration files for various models and datasets .

The configuration file is a text file in the format used to define the model structure, training parameters, data processing, etc.yaml

PaddleVideo supports the combination of different models and data sets by modifying configuration files , as well as model training, testing, reasoning and other operations .

PaddleVideo is divided into categories to store configuration files of different models and data sets:
PaddleVideo/configs/there are folders such as recognition(behavior recognition), segmentation(video segmentation), localization(action positioning), (action detection) and so on.detection

For example, there are many configuration filesrecognition of models and data sets in the direction of behavior recognition under the folder , as shown in the figure:

docs folder details

Store documentation related files.

Including Quick Start, Installation Instructions, User Guide, Model Library, Scenario Application, etc. These files are all markdownwritten in the format and can be viewed or downloaded directly on GitHub or Gitee.

• PaddleVideo/docs/zh-CN/dataset/Under the folder are description documents for various datasets , including introduction, download, preprocessing, etc. of datasets .

• PaddleVideo/docs/zh-CN/model_zoo/Under the folder are description documents of some cutting-edge algorithms and models , including the introduction, performance, and download links of various models .

• PaddleVideo/docs/zh-CN/benchmark.md: store related documents of model performance comparison , including accuracy and speed of various models on different datasets , etc.

paddlevideo folder details

A folder for storing codes and files related to the video model library .

There are mainly the following files in this folder, as shown in the figure:

utils folder

paddlevideo/utilsThe folder contains some common utility functions and preprocessing methods for loading, preprocessing and postprocessing of auxiliary video data , etc.

tasks folder

tasksThe role of the folder is to store some classes or functions used to define and execute different machine learning tasks .

Different machine learning tasks may require different data sets, models, indicators, training and testing processes, etc., such as image classification tasks, relationship classification tasks, semantic retrieval tasks, intelligent question answering tasks, etc.

tasksThe classes or functions in the folder can build and run corresponding models according to different tasks and data sets , and use the indicators in the folder to evaluate the performance of the model during training or testingmetrics .

loader folder

loaderThe role of the folder is to store some classes or functions for loading and processing data .

Different machine learning tasks may require different data formats, data preprocessing, data augmentation, data sampling, etc., for example,

• Image classification tasks need to load image files, crop, rotate, normalize, etc.,
• Relational classification tasks need to load text files, perform word segmentation, encoding, filling and other operations.

loaderThe classes or functions in the folder can create and use corresponding data loaders according to different tasks and data sets , and provide batch data input during training or testing .

modeling folder

modelingThe role of the folder is to store some classes or functions used to build and define the model .

The folder is as shown in the figure:

Among them, backbonesunder the folder:

headsUnder the folder:

Different machine learning tasks may require different model structures, model parameters, model layers, model activations, etc., for example,

• Image classification tasks require the use of convolutional neural networks, fully connected layers, $softmaxx$ layer,$ReLU$ activation etc.,
• Relation classification tasks need to use recurrent neural network, attention mechanism, linear layer, $tanh$ activation etc.

modelingThe classes or functions in the folder can create and use corresponding models according to different tasks and data sets , and realize the forward propagation and back propagation of the model during training or testing .

solver folder

solverThe role of the folder is to store some classes or functions used to optimize and solve the model .

Different machine learning tasks may require different optimization algorithms, loss functions, regularization terms, learning rate strategies, etc., for example,

• Image classification tasks need to use stochastic gradient descent, cross-entropy loss, weight decay, cosine annealing, etc.,
• Relational classification tasks require the use of adaptive moment estimation, contrastive losses, adversarial training, linear decay, etc.

solverThe classes or functions in the folder can create and use corresponding optimizers according to different tasks and data sets , and update and adjust the parameters of the model during training or testing .

metrics folder

metricsThe folder stores some classes or functions used to evaluate the performance of the model .

Different machine learning tasks may require different metrics to measure the quality of the model, for example,

• Commonly used indicators for classification tasks are accuracy, recall, $F1$ Score,$ROC$ curve, etc.,
• Commonly used indicators for regression tasks include mean square error, root mean square error, and mean absolute error.

metricsThe classes or functions in the folder can define and calculate corresponding indicators according to different tasks and data sets , and update and accumulate indicator values ​​during training or testing , so as to facilitate model selection and optimization.