When we watch various live broadcasts, when we have video calls with family and friends, when we use remote monitoring to see how our children are doing in kindergarten, the technology we use is video communication technology.
With the development of modern communication technologies and services, the video communication service integrating language and video transmission has become a hot spot in the development of the communication field. An efficient and stable video communication system is the basis for developing various video communication services.
1. How to build a video communication system
Building a video communication system is a systematic project integrating data acquisition, processing and display technology , data compression codec technology and network transmission technology . A basic video communication system process mainly includes the following three steps:
1) Data collected by the data sender: including multimedia data such as audio, video, and text.
2) Send the compressed data to the receiver over the network.
3) After the data receiver receives the data, it decodes the multimedia data to display and play.
However, in daily business, different businesses have different technical requirements for video communication systems. For example, for video conferences, it is required that the video communication process of two or more parties can be carried out in real time, the delay is as low as possible, and at the same time It is necessary to keep the communication process smooth and not stuck; for video surveillance systems, the requirements for video clarity are higher, important details in the picture can be retained, and delays are more tolerant.
Although different businesses have different emphases, building an efficient and stable video communication system has always been the pursuit of video communication operators.
(Figure 1. Building a video communication system is a comprehensive project that integrates technologies such as audio and video processing, encoding and decoding, and network transmission.)
2. How to build an efficient and stable video communication system
1. Data acquisition technology : It is required to obtain high-quality original data in an efficient and timely manner .
For example, in the real-time video call application, the collected audio generally needs to be preprocessed, mainly including filtering out background noise, detecting and eliminating echoes, etc.; in surveillance video, the collected video will be defogged and improved. Strong light interference, night image enhancement and other processing.
2. Data compression technology: Video compression technology in data compression is the core technology in the entire video communication system.
Since the data volume of the original video is very large, if it is not compressed, it will bring great pressure to transmission and storage. High-efficiency video coding technology can maintain the highest possible quality while meeting the limitation of transmission bandwidth, or obtain the highest possible compression rate under the condition of ensuring a certain quality.
In low-latency services, the requirements for compression speed are also very high. For example, to achieve a frame rate of 20 frames per second, the time required to compress each frame is less than 50 milliseconds, and at the same time, the compression performance must be taken into account. The current mainstream video coding standard H.264/AVC can obtain a good compromise between performance and speed. The next-generation video coding standard is H.265/HEVC. Although its bit rate can save 45% compared to H.264 with the same coding quality, it has not been widely used due to its patent issues and huge computational complexity. application.
In screen sharing and video calling services, the resolution should be set according to different scenarios. The former needs to see details such as screen text and has high resolution requirements; for the latter, since the face is generally flat, the resolution can be set lower. If it is a closed-loop communication system, it is even possible to develop non-standard codec technology or improve existing standard codec technology according to specific applications.
3. Network transmission technology : the key technology of video communication system .
The compressed data needs to be packaged according to a suitable network transmission protocol, the packaged data is transmitted to the receiving end through the network, and the receiving end parses the compressed data according to the network protocol. Depending on the type of service and the type of data to be transmitted, different transmission protocols need to be selected. The two most commonly used transmission protocols are Transmission Control Protocol TCP and User Data Packet Protocol UDP : TCP is a connection-oriented reliable transmission protocol, but the delay is relatively large; UDP is a connectionless, unreliable transmission protocol, but the delay is relatively large. smaller.
In the video communication system, it mainly includes multimedia data (audio, video, text and other data) and control signaling data (login server, user connection changes and other messages). Among them, the control signaling data plays the role of scheduling communication parties, and is very important to the normal operation of the entire system, requiring stable transmission, so the TCP protocol is generally used to achieve the purpose of stable transmission. For multimedia data, due to the high requirement for delay and the little impact of packet loss on the operation of the entire system, the UDP protocol is often used to achieve the purpose of timely transmission.
However, such a choice is not absolute. For example, in a service that does not require high multimedia data delay, TCP can also be used. Or develop a certain protocol according to a specific scenario, such as developing a protocol with a retransmission mechanism based on the UDP protocol, under the premise of satisfying low latency, while avoiding communication interruption caused by the loss of data transmission. In a word, the network transmission technology is very important for the stable and smooth communication, and it needs to be rationally designed in combination with the business itself.
4. Decoding and playback of multimedia data : the last step of the video communication system.
To achieve audio and video synchronization, while avoiding the effect of Huaping. During decoding, it is necessary to detect whether data is lost according to the decoding standard, and take measures such as not displaying or requesting retransmission of video frames that fail to be decoded, so as to avoid incomplete decoding frames, that is, the phenomenon of blurred screens. At the same time, the playback of audio and video should be synchronously played according to the timestamp, otherwise the user experience will be greatly affected. For the decoded audio, due to the influence of sampling rate change, compression distortion, etc., it is often processed by filtering, automatic gain, etc., so as to obtain a round and not harsh sound effect. Of course, in multi-person communication, the multi-channel audio must be mixed first.
3. Factors influencing the construction of an efficient and stable video communication system
The transmission bandwidth is unstable . The network environment is complex and changeable, especially in the mobile network environment, and the transmission bandwidth may change in real time . To achieve smooth video communication in such a network environment, the output bit rate of the encoding needs to be adaptively changed. If the network bandwidth becomes tight for a period of time, and the encoding output bit rate is still high, the data cannot be transmitted in time, and the receiver will be stuck, which will have a great impact on the user experience; and when the network bandwidth has a lot of excess At the same time, if a low bit rate is still used for encoding, it is a waste of bandwidth, and the video quality of the receiver is also poor.
Fourth, the solution
To achieve smooth and high-quality video communication, the encoding end needs to adjust the compression rate according to the real-time network status , which can be achieved by changing the quantization parameter, changing the resolution, or changing the frame rate. The real-time perception of the network state requires the network transmission module to perform statistical monitoring of the network state and provide the real-time network state information to the encoder.
(Figure 2. The network transmission module feeds back the network status to the encoder according to statistical information such as the packet loss rate, and the encoder adjusts the encoding parameters according to the network status to change the output bit rate.)
Of course, it is also possible to design a positive and feedback mechanism at the encoding end and the decoding end to realize the rate control . Specifically, a certain check code can be designed at the encoding end, so that the decoding end can know whether the packet is lost after receiving the message, so as to carry out packet loss statistics. According to the statistical results, the network state is estimated, and then feedback information is sent to the encoder. According to the feedback information, on the one hand, the encoder can resend the lost important data to the decoder; on the other hand, it can also know the network status and implement appropriate rate control measures in time, so that the communication can be carried out smoothly. When this design mode is adopted, the codec system constitutes a complete closed-loop system. At this time, the media data is transparent to the network transmission, so the network transmission module only needs to provide a transmission path.
(Figure 3. The decoding end estimates the network state according to the decoding failure rate, and feeds back the information to the encoder, which can also play the function of real-time control of the code rate. At this time, the network transmission module does not need to care about the transmitted content, but only provides a transmission path. .)
5. Summary:
To ensure the stable and smooth operation of the video communication system, in addition to the performance of each technical module is good enough, reasonable and efficient cooperation between each module is also required. The most important of these are the cooperation between the encoding module and the network transmission module, as well as the positive and feedback mechanisms at the encoding and decoding ends.
In general, building a complete and efficient video communication system is an extremely complex systematic project that requires the close cooperation of multiple key technologies; it is also a project that needs to be developed and optimized reasonably according to the specific requirements of the business.
Contact us, pay attention to Tuya WeChat public account
