Here, because a project needs to use the SRS server, and I also want to read the source code of the SRS server later, it is necessary to have a general understanding of the streaming media industry and streaming media servers. And I don't know much about this field, so I will spend some time here to have a simple understanding of some technologies in this field.
Introduction to Streaming Servers
A streaming media server is a server that processes streaming media files. Streaming media server is used in distance education, video on demand, Internet radio, Internet video, etc. A streaming media server is needed in the live broadcast process. A complete live broadcast process includes collection, processing, encoding, packetization, streaming, transmission, transcoding, distribution, decoding, playback and other processes. The streaming media server is mainly responsible for collection, transmission and Play and so on. Streaming media refers to the media form that transmits audio, video and multimedia files in the network in a streaming manner. Compared with the network playback form of downloading and viewing, the typical feature of streaming media is to compress continuous audio and video information and put it on the network server, so that users can watch while downloading without waiting for the entire file to be downloaded.
The development of the new era allows us to see the important role of the video industry in the Internet.
Media and entertainment industry: live broadcast era, RTMP/FLV/HLS, SRS, NGINX-RTMP
Communication and equipment industry: RTC, loT and 5G, GB28181/WebRTC, janus, OWT
With the development of technology and the times, the Internet has shifted from graphics to video, and the boundary between live broadcast and WebRTC has blurred. New videos and new scenarios require new open source servers.
Here we use the main application scenarios of streaming media to describe the main technical architecture
Internet live streaming and Lianmai
| Domain | Tech | OpenSource | Commercial |
|---|---|---|
| Camera (acquisition terminal) | Camera | Camera,Mobile Phone |
| Video Codec (video codec) | H.264 | x264，openh264，FFmpeg | Hardware |
| Audio Codec (audio codec) | AAC | fdkaac，FFmpeg | Hardware |
| Encoder (encoding) | System | OBS，FFmpeg | Haivision |
| Transport | RTMP，FLV，HLS | NGINX-RTMP，SRS | Wowza,AMS |
| Player | System | Mobile Phone, Chrome, Safari, Edge |
| Interactive | RTC | WebRTC |
video codec
H.264:
To put it simply: it is a video coding compression technology, and its biggest advantage is its high data compression ratio. Under the condition of the same image quality, the compression ratio of H.264 is more than twice that of MPEG-2, MPEG-2 1.5~2 times of 4.
Specifically refer to Baidu Encyclopedia and the H.264 principle blog, which I personally think is a good in-depth article.
H.264_Baidu Encyclopedia (baidu.com)
"H264 Coding Principle and Block Diagram" that you can understand after reading
Method to realize
-
x264: It is an open source H.264/MPEG-4 AVC video encoding function library and one of the best lossy video codecs.
-
openh264: also an open source H.264 codec
-
FFmpeg: It can also do codec, but it integrates open-source computer programs that record, digital audio, video, etc. are converted into streams. Compared with the above two encoding technologies, the return is more widely used. **
- For details, please refer to Baidu Encyclopedia: ffmpeg_ Baidu Encyclopedia (baidu.com)
-
Hardware: directly implemented in hardware.
audio codec
AAC：
To put it simply: an advanced audio coding technology, there are two types, one is MPEG-2 AAC, and the other is MPEG-4 AAC.
Refer to Baidu Encyclopedia for details. AAC_Baidu Encyclopedia (baidu.com)
Method to realize
- fdkaac: is an open source MPEG-4 and MPEG-2 AAC codec
- Reference link: fdk_aac | User Manual (t-head.cn)
- FFmpeg: In addition to video encoding, ffmpeg can also perform audio encoding and decoding.
- Hardware: directly implemented in hardware.
related tools
-
Hardware: Haivision has a variety of audio and video codec hardware.
-
OBS: An open source and free video live broadcast software that can broadcast almost all live broadcast platforms.
-
FFmpeg: is a free open source library, a command line tool software.
Transfer Protocol
RTMP:
To put it simply: it is a real-time message transmission protocol based on TCP, so the speed is not fast enough at the beginning, there will be delays, but it is reliable.
For details, refer to Baidu Encyclopedia: rtmp_Baidu Encyclopedia (baidu.com)
FLV：
To put it simply: it is a video format launched by Adobe, a streaming media data storage container format transmitted on the network.
HLS：
To put it simply: it is an HTTP-based media streaming protocol implemented by Apple, which can easily pass through firewalls or proxy servers, and can easily use CDN for distribution acceleration, and the client is also very convenient to implement. It is easy to deploy to HTML5.
CDN：
To put it simply: it is a content distribution network, which adds a new layer of network architecture to the existing Internet, and publishes the content of the website to the edge of the network closest to the user, so that users can obtain the content they need nearby and improve user access. The responsiveness of the website.
Method to realize
NGINX, SRS: Both streaming media servers support the above protocols.
Wowza, AMS: Well-known brand companies belonging to streaming media servers.
playback device
mobile phone, browser, etc.
Interaction (such as Lianmai)
RTC：、
To put it simply: Real-time communication is an abbreviation for real-time audio and video. The bottom layer uses the RTP protocol, and the transport layer is UDP. The RTC technology we often refer to generally refers to WebRTC technology.
For details, refer to this Zhihu: RTC technology (WebRTC) - Zhihu (zhihu.com)
Method to realize
- WebRTC: A real-time communication technology that allows browsers to connect peer-to-peer without third-party plug-ins.
Internet real-time communication
| Domain | Tech | OpenSource | Commercial |
|---|---|---|
| Camera (acquisition terminal) | Camera | Camera,Mobile Phone |
| Video Codec (video codec) | H.264 | x264，openh264，FFmpeg | Hardware |
| Audio Codec (audio codec) | Opus | bopus |
| Encoder/Player (decode, play) | 3A，CC，SVC | WebRTC |
| Transport | RTP，RTCP，SRTP | janus，Mediasoup，OWT，SRS |
| PSTN (Public Switched Telephone Network) | SIP，RTP | Freeswitch |
video codec
- It is the same as the video codec adopted by Internet live broadcast and Lianmai.
audio codec
Opus：
To put it simply: it is a format for lossy sound encoding, but suitable for low-latency instant sound transmission over the network.
For details, please refer to Baidu Encyclopedia: opus (sound encoding format)_Baidu Encyclopedia (baidu.com)
Method to realize:
- libopus: Open source implementation of the Opus codec.
Transfer Protocol
RTP：
To put it simply: it is a real-time transport protocol, usually running RTP on UDP for its multi-channel node and verification services.
For details, please refer to Baidu Encyclopedia: Real-time Transport Protocol_Baidu Encyclopedia (baidu.com)
RTCP：
To put it simply: it is the control protocol of the RTP protocol, which provides data distribution quality feedback information. This is part of the function of RTP as a transport protocol and it is related to flow control and congestion control of other transport protocols .
For details, refer to rtcp_Baidu Encyclopedia (baidu.com)
SRTP：
To put it simply: it is a secure real-time transport protocol, based on RTP, designed to provide encryption, message authentication, integrity assurance, and replay protection for data in real-time transport protocols in unicast and multicast applications.
Method to realize
- janus: Janus is a WebRTC server developed by Meetecho, so in addition to implementing methods for establishing WebRTC media communication with the browser, exchanging JSON messages with it, and relaying RTP/RTCP and messages between the browser and server-side application logic , which does not provide any functionality by itself.
- Mediasoup: mediasoup is a high-performance sfu server fully compatible with webrtc. It consists of a master terminal implemented in ts language and a work module implemented in c++ language based on libuv.
- OWT: The full name of OWT is Open WebRTC Toolkit, which is Intel's end-to-end audio and video SDK based on WebRTC. It is a high-performance, reliable and scalable real-time communication solution.
- SRS: domestic open source streaming media server.
PSTN
**SIP: **Session Initiation Protocol, which is a text-based application layer control protocol for creating, modifying and releasing sessions of one or more participants. SIP is an IP voice session control protocol originating from the Internet. It is flexible, easy to implement, and easy to expand.
Internet Media Center
| Domain | Tech | OpenSource | Commercial |
|---|---|---|
| SRT (Open Source Video Transport Protocol) | UDP,CC | Libsrt,FFmpeg,OBS,SRS |
| GB28181 (national standard agreement) | SIP，RTP | SRS，Camera |
| DVR (Digital Video Recorder) | FLV，MP4，HLS | FFmpeg，NGINX-RTMP，SRS |
| Transcoding | H.264, ACC | FFmpeg, x264, fdkaac |
| security | HTTP Callback | SRS |
| Management | HTTP API | SRS |
| Cluster | System | SRS Edge/Origin Cluster |
An Internet media center basically needs to include both of the above situations.
The above is a simple literacy in the field of audio and video, and lays a foundation for subsequent continuous learning.