Let's assume that someone has zero foundation in audio and video, and has not learned any digital signal processing related knowledge. The mathematical foundation is basically high school level, but he is familiar with C/C++ development, at least familiar with the compilation and debugging IDE under a certain platform.
Focus on the two open source projects ffmpeg and webRTC, mainly looking at the video part of ffmpeg and the audio part of webRTC.
First of all, start with video decoding. This is the simplest. The basic code to call ffmpeg video decoding is only about 100 lines. Video decoding only needs to understand h.264. 80%+ scenes are in this encoding format. Understand the basic H .264 sps pps NAL and other file format concepts, understand the principle of YUV image format, understand the conversion of YUV to RGB, so that image rendering can be done, and a simple video player can be realized. (Basically in the first stage)
Then, to learn video coding, it is recommended to start with x264. A simple call to x264 encoding code is less than 100 lines. Encoding is more complicated than decoding in that decoding does not require any parameters to be configured, and encoding has a large number of parameters that need to be configured. At the beginning, just understand a few basic concepts, such as frame rate, common flow control methods ABR CRF, GOP, I/B What does /P Frame mean respectively. (Familiar with X264 encoding again)
The encoding and decoding of the audio part is similar to that of the video. In the early stage, you only need to understand the two encoders EAAC+ and Silk, understand the concepts of sound channel, sampling rate and other concepts, and understand the Wave file format. Can be a simple audio player. (To be an audio player) The
complex part of audio is not the codec, but the sound effect. Take a look at the Audio Processing module in webRTC and understand the following concepts: denoising NS, echo cancellation AEC, silent detection VAD, automatic gain control AGC Although these algorithms built into webRTC are not the best, they can solve 90%+ of the problems and are worth learning. (Learn webrtc audio processing)
After learning this, I need to think about a question, whether I want to take the engineering route or the algorithm route. If it is the algorithm route, then you must learn mathematics. Behind the digital signal processing is a large number of basic mathematical theories. If it is an engineering route, then you have to learn cross-platform development, learn audio and video collection, playback and processing under multiple platforms, and learn assembly language optimization under multiple platforms. (Platform-related)
Audio and video are a very deep field, and you will never finish learning in a lifetime, but this field has the advantage that the speed of knowledge update is relatively slow, the things learned are not easily eliminated, and the accumulation of experience is very important.
Let's take a look at the learning route of audio and video development:
1. The first thing you need to learn is the basic knowledge of audio and video
Some friends have some misunderstandings about audio and video learning, thinking that it takes a lot of theoretical foundation to start practicing. In fact, everyone only needs to understand:
video:
- YUV format
- RGB format
- Frame rate
- Resolution
- H264 IPB frame principle
Audio:
- PCM
- Sampling Rate
- Sampling format
- Bit rate
Wait for these basic knowledge to open the door of audio and video learning.
2. FFmpeg command actual combat
Many friends also have misunderstandings about FFmpeg learning. They start to use various ffmpeg apis. In fact, we can start with the FFmpeg command line. We provide a document of "Windows FFmpeg Command Line Setup". Starting from the FFmpeg command line allows us to better understand what the FFmpeg open source project can do.
FFmpeg video learning materials +1035101242 (VX same number) free to obtain and note FFmpeg materials. Audio and video advanced development qun: Jumping
3. FFmpeg project actual combat
Use FFmpeg to make a player and screen recorder
4. Streaming media combat
Streaming media practice suggests that you learn the order of RTMP->HLS->HTTP-FLV. If you have time, you can learn RTSP again. Why do you say that you have time to learn RTSP again? Because the Internet industry rarely uses RTSP.
5. WebRTC Intermediate Development
6. WebRTC advanced development
7. WebRTC SFU server development
Most of the content that needs to be learned above are skills that audio and video developers must know. If you find it difficult to learn audio and video by yourself, and there is no direction to learn, you can follow the route above to learn.
At last
Here I also share with you the learning materials and learning roadmap of audio and video development. If you need, you can click on Jumping into the group to get
