Use MediaCodec and RTMP for live streaming

This project is completely open source, the project Github address: AndroidInstantVideo

At present, open source projects or Android live streaming clients on the market mainly use ffmpeg to implement streaming. This article will introduce the use of Android's native video encoding class MediaCodec to implement live streaming.

Data flow and general principle

The live broadcast mentioned here is to send the video data generated by your client to the server in real time. The data on the server is then sent to the playback client in real time.

Take video data as an example:

Obtaining the Camera picture
First, the camera captures the original picture data. We don't care about the format of the original picture data here, because we are using MediaCodec, so we will use
camera.setPreviewTexture(surfaceTexture)
to use the picture obtained by Camera.

The principle here can be ignored. In general terms, the Camera will save the obtained picture as a texture of OpenGL, and we can use this texture to use the Camera picture.

Drawing the picture
After getting the picture, we need to "draw" the picture (texture) to the MediaCodec.

How to draw?
MediaCodec provides a 'white paper', which is a Surface, for us to draw textures onto. The API here is
MediaCodec.createInputSurface()

How to draw? Draw with Canvas. Of course it's not a normal Canvas, I used this open source project android-openGL-canvas .

After the H264 data is
drawn, MediaCodec will help us compress the original picture data into corresponding video data. At present, I compress it into H264 data.
The so-called H264 data is actually just a bunch of byte[] arrays. In the project example, I wrote the H264 data into a file that can be played with some players (eg PotPlayer).

RTMP
I use an open source project that can encapsulate video data into RTMP packets and send them to the server.
LibRtmp-Client-for-Android

To sum up,
the data flow can be seen as
Camera -> SurfaceTexture -> Surface -> MediaCodec -> encode data(byte[]) -> RTMPMuxer -> Server

Audio data:

It is relatively simple, that is, get the original audio data (byte[]) from AudioRecord, encode it into AAC data (also byte[]), and then give it to RTMPMuxer, encapsulate it into RTMP package, and send it to the server

Microphone MIC -> AudioRecord -> voice data (that is byte[]) -> MediaCodec -> encode data (that is byte[]) -> RTMPMuxer -> Server

Muxer

As mentioned above, RTMP packets with video and RTMP packets with audio are encapsulated into RTMP packets of unit H264 and unit AAC respectively and sent to the server. What's the pattern between these packages?
These packets are arranged in chronological order. When MediaCodec returns the encoded data, it will return the timestamp of the encoded data. But note that when encoding into an RTMP packet, the relative timestamp is taken, that is to say, when the timestamp is obtained, the difference with the timestamp of the previous packet needs to be calculated and written into the RTMP packet.

In addition, RTMP streams are essentially audio and video in FLV format, and the function of writing FLV files is also provided here.

renderings

PC player

PC player
Android push stream

process

Video frame image processing

The image processing of video frames mentioned earlier actually uses android-openGL-canvas .

The key code is as follows:

    ...
    streamPublisher.prepareEncoder(streamPublisherParam, new H264Encoder.OnDrawListener() {
        @Override
        public void onGLDraw(ICanvasGL canvasGL, SurfaceTexture surfaceTexture, RawTexture rawTexture, @Nullable SurfaceTexture outsideSurfaceTexture, @Nullable BasicTexture outsideTexture) {
            drawVideoFrame(canvasGL, outsideSurfaceTexture, outsideTexture);

            Loggers.i("DEBUG", "gl draw");
        }
    });
    ...

    private void drawVideoFrame(ICanvasGL canvasGL, @Nullable SurfaceTexture outsideSurfaceTexture, @Nullable BasicTexture outsideTexture) {
        // Here you can do video process
        // 此处可以视频处理，例如加水印等等
        TextureFilter textureFilterLT = new BasicTextureFilter();
        TextureFilter textureFilterRT = new HueFilter(180);
        int width = outsideTexture.getWidth();
        int height = outsideTexture.getHeight();
        canvasGL.drawSurfaceTexture(outsideTexture, outsideSurfaceTexture, 0, 0, width /2, height /2, textureFilterLT);
        canvasGL.drawSurfaceTexture(outsideTexture, outsideSurfaceTexture, 0, height/2, width/2, height, textureFilterRT);

    }
    ...

As shown above, various Filters can be used to process video frame images. All in all, it's possible to draw all kinds of things on the video frame like a Canvas. Of course, if you want to draw text on the image, you can only use bitmap instead.

Bit rate bit/s

When using MediaCodec, you need to set the code rate. This bit rate is calculated based on the video resolution and color format.

    public H264Encoder(int width, int height, int bitRate, int frameRate, int iframeInterval, final EglContextWrapper eglCtx) throws IOException

where bitRate is the bit rate, in bit/s

Some calculation methods can refer to this article:
What bitrate should I use when encoding my video?
Output size Bitrate Filesize
320x240 pixels 400 kbps 3MB / minute
480x270 pixels 700 kbps 5MB / minute
1024 x 576 pixels 1500 kbps 11MB / minute
1280x720 pixels 2500 kbps 19MB / minute
1920x1080 pixels 4000 kbps 30MB / minute

This method is sufficient in most cases, but it is still lacking for complex video processing.
For example,
compare the unprocessed effect of the following image (a texture)

one

For the processing effect in the picture below (the two pictures use the same size texture as the picture above, although I set the display size to be different), the code rate is about 2 times that of the picture above.

test server

If you need to test, please build your own RTMP server. I use the Nginx server I built myself, and the Module I use is nginx-rtmp-module . There is no need to write code to build a server, just type a few lines of commands according to the tutorial.
You can use the open source live broadcast software OBS to compare the playback effects.
All kinds of players can be used, VLC, PotPlayer, ffplay can be used.
I'm using ffplay. Note that because it's just a simple server, you need to open the player connection before starting the push stream.
The command I use is .\ffplay.exe "rtmp://localhost:19305/live/room live=1"

In addition, you can use the following software to view the details of the generated files.

Look at H264 file
H.264 video stream analysis – Lei Xiaohua

h264

Look at aac file
AAC audio stream analysis – Lei Xiaohua
aac

Look at the flv file
FLV package format analysis – Lei Xiaohua

flv

Thanks Thor.

statement

This project is completely open source, and the project Github address: AndroidInstantVideo
This project is a personal open source project. At present, only a simple test has been done. If you want to use it, please test it yourself. If you have any questions, you can submit it to my Github project address.

finally

Your tip is the greatest support for the author (left column)! ! Of course, clicking Star on Github is also a great support haha.