Taking intelligent video analysis and storage as an example, common application scenarios include video surveillance, sports event analysis, artificial intelligence education, and self-driving cars. For these scenarios, it is necessary to intelligently analyze the video content and add metadata to the video for easy storage and retrieval.
1. Attach the information description of intelligent analysis to a separate file for management
Taking products based on this solution as an example, Hikvision's video intelligent recognition product "Hisilicon Algorithm" is a typical representative of this solution. HiSilicon algorithm can identify the object, color and other information of the video, and store it as independent metadata information. When users need to retrieve related videos, they can realize intelligent search by retrieving these metadata information. In addition, this approach supports online updating of model data to improve recognition accuracy.
advantage:
- Separation of data and video content: Write metadata information into an independent file or database for future management and update.
- Scalability: You can customize the metadata information structure according to specific business needs to facilitate expansion.
- Low Latency: Since data and video content are separated, there is no impact on video transmission delay.
shortcoming:
- Reliability: Since data and video content are separated, the correspondence between the two needs to be guaranteed. If there is a problem with this correspondence, the reliability of the metadata will be affected. In addition, failure to update metadata information in a timely manner may result in reduced recognition accuracy.
2. Add the information of intelligent analysis to SEI
Taking products based on this solution as an example, Weishi Media's intelligent sports event analysis platform adopts the method of adding intelligent analysis data to SEI. The platform can conduct real-time analysis of sports event videos, and add information such as players on the field, game scores, and audience reactions to the SEI for subsequent retrieval and analysis.
advantage:
- Compatible with existing video coding standards: Since the existing SEI mechanism is used, the addition of metadata can be realized under the existing video coding standards.
- Real-time performance: Since the SEI information is transmitted together with the NAL unit, metadata information can be provided in real time.
- Easy to expand: Since SEI is extensible, it can support various types of metadata information.
shortcoming:
- Limited data size: Since the size of SEI data is limited by NAL units, only small metadata information can be added.
- Data parsing complexity: Since SEI information needs to be decoded by a decoder, it is necessary to consider the decoding complexity and corresponding processing flow during analysis and processing.
3. Write the intelligent analysis information into the package format (such as PS, TS, etc.)
Taking products based on this solution as an example, Intel's OpenVINO toolkit can deploy deep learning models to terminal devices based on Intel architecture. The toolkit supports writing intelligently analyzed information into TS data packets for subsequent storage and retrieval.
advantage:
- Compatible with existing formats: Due to the use of the existing TS protocol mechanism, it can be used in existing network transmission and storage scenarios.
- Support a large amount of metadata: Since the TS packet size is not limited, a large amount of metadata information can be added.
- Easy to expand: Since TS supports multiple types of metadata information addition, it can be expanded according to specific business needs.
shortcoming:
- Higher Latency: Due to the inconsistent size of TS packets, it may cause increased transmission delay.
- Storage complexity: Since metadata information is fused with video content, it is necessary to consider how the data is organized and retrieved when storing and managing it.
To sum up, methods such as attaching the information description of the intelligent analysis to a separate file for management, adding it to the SEI, and writing it to the package format have their advantages and disadvantages. Which method to choose depends mainly on the specific application scenarios and requirements. According to different scenarios, it is necessary to comprehensively consider factors such as data storage management, intelligent retrieval, deep learning training and reasoning, so as to choose a suitable solution.