In fact, to put it plainly, audio and video development is to master the basic knowledge of images, audio, and video, and learn how to collect, render, process, and transmit a series of development and applications.
Acquisition: it solves the problem of where the data comes from
Rendering: it solves the problem of how the data is displayed
Processing: it solves the problem of how the data is processed
Transmission: it solves the problem of how to share the data
Each category can be dig deep to derive one after another full of technical challenges, such as: how to render images more efficiently, how to improve the compression ratio of audio and video, how to optimize the transmission of audio and video data under weak networks, and so on.
In fact, the technical accumulation of audio and video development is not that difficult. Go to Google with problems and practice with tasks. Everything is not a problem. Let’s explore the four directions mentioned above and explore the knowledge points one by one. It is for us to understand and master.
collection
Collection, it solves the problem of where the data comes from, so where does the data come from?
In fact, no matter which platform, images and videos are originally from the camera, and the audio is originally from the microphone. Therefore, to do audio and video collection, you must master the following technical knowledge:
1. 系统的摄像头采集接口是什么，怎么用 ？
比如：Windows：DirectShow，Linux：V4L2，Android：Camera，iOS：AVCaptureSession 等
2. 系统的摄像头采集的参数怎么配置，都是什么含义 ？
比如：分辨率、帧率、预览方向、对焦、闪光灯 等
3. 系统的摄像头输出的图像/视频数据，是什么格式，不同格式有什么区别 ？
比如：图片：JPEG，视频数据：NV21，NV12，I420 等
4. 系统的麦克风采集接口是什么，怎么用 ？
比如：Windows：DirectShow，Linux：ALSA & OSS，Android：AudioRecord，iOS：Audio Unit 等
6. 系统的麦克风采集参数怎么配置，都是什么含义 ？
比如：采样率，通道号，位宽 等
8. 系统的麦克风输出的音频数据，是什么格式？
比如：PCM
Rendering
Rendering, it solves the problem of how to display the data, then how to display the data?
In fact, no matter which platform, images and videos are ultimately drawn to the view, and audio is ultimately output to the speakers. Therefore, to do audio and video rendering, you must master the following technical knowledge:
- What APIs does the system provide to draw a picture or a frame of YUV image data?
For example:
Windows: DirectDraw, Direct3D, GDI, OpenGL, etc.
Linux: GDI, OpenGL, etc.
Android: ImageView, SurfaceView, TextureView, OpenGL, etc.
iOS: CoreGraphics, OpenGL, etc.
- What APIs does the system provide to play an mp3 or pcm data?
For example:
Windows: DirectSound, etc.
Linux: ALSA & OSS, etc.
Android: AudioTrack, etc.
iOS: AudioQueue, etc.
deal with
Processing, it solves the problem of how to process the data, then, how can the data be processed?
First, let's see what processing can be done on image/audio/video data?
In fact, regardless of the platform, the processing of images, audio and video, in addition to the system API, most of them will rely on some cross-platform third-party libraries. By mastering the principles and usage of these third-party libraries, you can basically meet the daily audio and video Processing work, these libraries include but not limited to:
1. Image processing: OpenGL, OpenCV, libyuv, ffmpeg, etc.
2. Video codec: x264, OpenH264, ffmpeg, etc.
3. Audio processing: speexdsp, ffmpeg, etc.
4. Audio codec: libfaac, opus, speex, ffmpeg, etc.
Therefore, it is very necessary to learn and master the use of these third-party libraries
transmission
Transmission, it solves the problem of how to share data, then how to share data?
Sharing, the most important point is agreement.
I think the reason why the Internet is able to develop so vigorously and connect the whole world closely is actually inseparable from the great contribution of the W3C committee, because no matter what data, it is necessary to interconnect between different countries and different devices. Intercommunication is inseparable from "standards". With "standards", everyone can understand each other.
Therefore, studying audio and video transmission is actually studying protocols. What are the specific protocols?
1. How to package audio and video before transmission, such as: FLV, ts, mpeg4, etc.
2. Live streaming, what are the common protocols, such as: RTMP, RTSP, etc.
3. Live streaming, what are the common protocols, Such as: RTMP, HLS, HDL, RTSP, etc.
4. What are the UDP-based protocols? Such as: RTP/RTCP, QUIC, etc.
The transmission of audio and video in the Internet environment is a very challenging and valuable direction. In order to solve the transmission delay and stall under the weak network, and to improve the user experience, the entire industry is constantly conducting in-depth exploration and optimization.