To start with Shakespeare's famous quote: To mix or not to mix, that is the question.
In the Lianmai interactive live broadcast plan, whether to mix or not to mix is really a question that needs careful consideration.
This article will share the experience of mixed-streaming summed up in the process of engineering and high-concurrency operation and maintenance of the instant technology Lianmai interactive live broadcast solution.
What is mixed flow?
Before we start discussing this issue, let's first understand what is mixed flow.
Mixing is to mix multiple audio and video streams into a single stream. To be precise, mixing should be called mixing (audio stream) mixing picture (video stream).
For the sake of simplicity, we refer to it as mixed flow here.
The mixing process includes four steps: decoding, mixing, encoding, and pushing. The mixing process includes jitter buffering. The purpose is to align multiple streams and synchronize audio and video. At the same time, buffering is used to counter network jitter, so that good results can be achieved after mixing into one stream.
Stream mixing itself is a relatively computationally expensive operation, and it is also a relatively time-consuming link. It is said that it consumes computing resources because this process involves real-time transcoding of multiple streams, which is more CPU-intensive. It is time-consuming because this link involves jitter buffering, which will cause a certain delay. The time consumed by real-time transcoding itself is generally less than the time of one frame, which is relatively insignificant; however, depending on the network conditions, jitter buffering will consume about 80 milliseconds to 1 second. In the Lianmai interactive live broadcast, the delay caused by jitter buffering is very critical to the viewer's experience.
In the process of mixing streams, the difficulty is how to combat uncertain factors such as network jitter. Dealing with various uncertain factors is the most test of technical ability. When the network is good, try to reduce the delay and complete the processing as quickly as possible. When the network is not good, it is necessary to avoid problems such as stuttering and out-of-sync. There needs to be a better elastic design here to meet the requirements of combating network jitter. It is necessary to ensure that no matter the network is good or the network is jittery, the effect of audio and video streams is always in a stable state, rather than violently jittering with the network jitter, including the delay should also be in a relatively smooth and stable state condition. This is a technical difficulty, and it is also one of the core technologies of the instant technology Lianmai interactive live broadcast solution.
Well, now I basically understand the concept and difficulties of mixed flow. So, do you want to do this thing? If to do, where to do it? Both issues require a trade-off. In order to make the right choice, we need to know the advantages and disadvantages of each option.
This short article will only discuss mixed flow or not. In the next article, we will discuss where to mix flow, so stay tuned.
Advantages and disadvantages of non-mixing
If you don't mix the streams, then during the Lianmai interactive live broadcast, the audio and video streams launched by multiple anchors will be pulled to the viewers for playback independently. This method is called multi-streaming.
The advantage of no mixing
1) Low latency
There is no need to mix streams, which saves the time consumed by stream mixing and significantly reduces the delay.
2) low cost
If it is mixed on the server side, it will consume computing resources. Considering that the computing resources of the server are relatively expensive, if the mixed flow is not used, precious computing resources will be saved and costs will be significantly reduced. Although pulling multiple streams consumes more bandwidth costs than pulling single streams, pulling multiple streams saves computing resource costs, and overall, the cost is reduced.
3) Flexibility
On the audience side, the service side can flexibly control multiple streams to meet diverse service needs. For example, the picture-in-picture size screen can be switched to each other, and the half-split screen can be swapped left and right to improve the user experience of the audience.
Disadvantages of not mixing
As mentioned above, rado streaming consumes more bandwidth. Multiple streams are pushed from the server to the CDN, and then the viewers pull multiple streams from the CDN, which will consume a lot of bandwidth costs. For a game live broadcast platform where bandwidth costs account for a significant percentage of operating costs, it does need to be carefully considered.
Advantages and disadvantages of mixed flow
If the stream is mixed, then during the Lianmai interactive live broadcast, the audio and video streams launched by multiple anchors will be mixed into a single stream and pulled to the audience for playback. This method is called pulling a single stream.
Advantages of mixed flow
We have to look at the advantages of muxing from multiple perspectives, including cost, ease of recording, ease of distribution, and compatibility.
1) Cost
It can be divided into computing resource cost and bandwidth cost. Due to pre-mixing, the cost of computing resources will go up, but because only a single stream is pulled, the cost of bandwidth will go down.
This topic has been briefly discussed above.
If the muxing is done on the server side, the overall cost will still go up. Because the computing resource cost of the server is relatively expensive, the increased computing resource cost is more than the bandwidth cost saved after stream mixing.
If the stream mixing is done on the streaming end, it is equivalent to the platform transferring the computing cost and bandwidth cost to the streaming end (the anchor end), so the overall cost will be reduced, but the service quality will also be reduced, thus paying the "user experience" cost".
Overall, mixing flow does not necessarily save costs.
2) Recordable
If there is a need to record audio and video streams in the business for supervision and spot checks or audience playback, then mixing is required. If you do not mix streams, you can only record one of the audio and video streams when recording, that is, you can only see the picture of one of the anchors. To record full screen, you must mix the stream.
3) Easy to spread
If there is a need for forwarding and dissemination through the audio and video stream address link (HLS) in the business, it also needs to be mixed, because the address link will only point to one audio and video stream. If you don't mix the stream, using the forwarded address link will only play the audio and video stream of one anchor.
4) Compatibility
There is another situation. The live broadcast platform has developed a version of the player in the early days, and this player can only play a single stream. In order to be compatible with this old version of the player, it must also be mixed to generate a single stream for the player to play.
The advantage of mixed streaming is that it can do full-screen recording, which is convenient for forwarding and dissemination, and is compatible with the old system of the live broadcast platform. Whether the operating cost can be reduced mainly depends on whether the stream mixing is done on the server side or on the push stream side.
Disadvantages of mixing
1) High latency
In the process of mixing streams, calculation processing such as jitter buffering and real-time transcoding needs to be performed, which will take time and cause additional delay.
The latency of muxing is higher than that of non- muxing.
If mixing is performed on the streaming end, the delay time includes the time for real-time transcoding, the time for jitter buffering, and the time for waiting for the audio and video streams of other anchors. The streaming end must wait for the audio and video streams of other anchors to arrive before mixing.
If mixing is performed on the server, the delay time includes real-time transcoding processing time and jitter buffering time.
The latency of mixing on the push side is higher than that on the server side.
2) Not flexible
Since the viewer is watching a single stream, multiple audio and video streams have been mixed into a single stream, so the viewer can no longer flexibly control the multiple streams, such as switching the primary and secondary images of picture-in-picture.
3) High cost
Since mixing flow requires additional computing resources, additional operating costs are incurred here.
If the muxing is performed on the server, the cost of muxing is borne by the platform. Although the bandwidth is saved by pulling a single stream after mixing, the computing resources of the server are relatively expensive, and the cost of bandwidth saved is not as much as the cost of additional computing resources. Overall, the cost of mixed flow is still higher than that of non-mixed flow.
If muxing is performed at the push end, the cost of muxing is passed on from the server to the push end. The push end needs to meet the following conditions:
a) The mobile phone configuration requirements of the streaming end (anchor end) are relatively high, the performance requirements of the mobile phone are relatively good, and it must be able to support the computing power required for mixed streaming.
b) The upstream bandwidth of the streaming end has relatively high requirements. The push end pushes two streams and pulls multiple streams from other hosts. The two streams that are pushed include: the original audio and video stream, and the audio and video stream after mixing. The pulled multi-streams are the audio and video streams of other hosts, which must be pulled from the media server.
Because the streaming end needs better mobile phone configuration and better uplink network bandwidth, it needs to pay higher costs.
The disadvantage of muxing is high latency and inflexibility. In terms of cost, it depends on whether the stream is mixed on the server side or on the push stream side. Mixing streams on the server side will result in high costs, and mixing streams on the push side will result in high latency and pay "user experience costs".
To sum up, the mixed-flow and non-mixed-flow schemes each have their own advantages. It is necessary to decide which solution to use according to the specific business needs and usage scenarios of the live broadcast platform. Relatively speaking, the unmixed solution has many outstanding advantages, such as low latency, low cost, and high flexibility.
Instant Technology provides both mixed and non-mixed solutions, flexibly and comprehensively meeting the business needs of various types of live broadcast platforms. In the practice of Instantaneous Technology, it is found that the solution without mixing flow is more popular with mainstream customers, and the effect has been proved by the market. Customers using the unmixed solution include Zanthoxylum and Chestnut Live.
After some discussion, we looked back and found: to mix the flow, or not to mix the flow, this is still a question.
In the process of serving first-line live broadcast customers, Instant Technology has summarized various advantages and disadvantages of mixed-stream and non-mixed-stream solutions. Based on these experiences, the decision to use a mixed-flow or non-mixed-flow solution should be more reliable and grounded.
After making the decision to mix the flow or not to mix the flow, if you want to mix the flow, where should you do it? Is it on the streaming side or the server side? This point will be discussed in detail in the next article without reservation, sharing the technical experience of instant technology.
You can continue to pay attention to the next part of the technical dry goods sharing series brought by Jigou Technology: The Mixing and Mixing Scheme in Lianmai Interactive Live Broadcast: Streaming Side VS Server Side.