Audio and video development MediaCodec merges audio and video

Android provides a MediaExtractor class that can be used to separate the video track and audio track in the container. The following example shows the use of MediaExtractor and MediaMuxer to achieve video sound change:

private void muxingAudioAndVideo() throws IOException {
    MediaMuxer mMediaMuxer = new MediaMuxer(mOutputVideoPath, 
                MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);

    // 视频的MediaExtractor
    MediaExtractor mVideoExtractor = new MediaExtractor();
    mVideoExtractor.setDataSource(mVideoPath);
    int videoTrackIndex = -1;
    for (int i = 0; i < mVideoExtractor.getTrackCount(); i++) {
        MediaFormat format = mVideoExtractor.getTrackFormat(i);
        if (format.getString(MediaFormat.KEY_MIME).startsWith("video/")) {
            mVideoExtractor.selectTrack(i);
            videoTrackIndex = mMediaMuxer.addTrack(format);
            break;
        }
    }

    // 音频的MediaExtractor
    MediaExtractor mAudioExtractor = new MediaExtractor();
    mAudioExtractor.setDataSource(mAudioPath);
    int audioTrackIndex = -1;
    for (int i = 0; i < mAudioExtractor.getTrackCount(); i++) {
        MediaFormat format = mAudioExtractor.getTrackFormat(i);
        if (format.getString(MediaFormat.KEY_MIME).startsWith("audio/")) {
            mAudioExtractor.selectTrack(i);
            audioTrackIndex = mMediaMuxer.addTrack(format);
        }
    }

    // 添加完所有轨道后start
    mMediaMuxer.start();

    // 封装视频track
    if (-1 != videoTrackIndex) {
        MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
        info.presentationTimeUs = 0;
        ByteBuffer buffer = ByteBuffer.allocate(100 * 1024);
        while (true) {
            int sampleSize = mVideoExtractor.readSampleData(buffer, 0);
            if (sampleSize < 0) {
                break;
            }

            info.offset = 0;
            info.size = sampleSize;
            info.flags = MediaCodec.BUFFER_FLAG_SYNC_FRAME;
            info.presentationTimeUs = mVideoExtractor.getSampleTime();
            mMediaMuxer.writeSampleData(videoTrackIndex, buffer, info);

            mVideoExtractor.advance();
        }
    }

    // 封装音频track
    if (-1 != audioTrackIndex) {
        MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
        info.presentationTimeUs = 0;
        ByteBuffer buffer = ByteBuffer.allocate(100 * 1024);
        while (true) {
            int sampleSize = mAudioExtractor.readSampleData(buffer, 0);
            if (sampleSize < 0) {
                break;
            }

            info.offset = 0;
            info.size = sampleSize;
            info.flags = MediaCodec.BUFFER_FLAG_SYNC_FRAME;
            info.presentationTimeUs = mAudioExtractor.getSampleTime();
            mMediaMuxer.writeSampleData(audioTrackIndex, buffer, info);

            mAudioExtractor.advance();
        }
    }

    // 释放MediaExtractor
    mVideoExtractor.release();
    mAudioExtractor.release();

    // 释放MediaMuxer
    mMediaMuxer.stop();
    mMediaMuxer.release();
}

The interface of MediaExtractor is relatively simple. First, set the data source through setDataSource(). The data source can be a local file address or a network address:

MediaExtractor mVideoExtractor = new MediaExtractor();
mVideoExtractor.setDataSource(mVideoPath);

Then you can get the MediaFormat of each track through getTrackFormat(int index), and get the detailed information of the track through MediaFormat, such as: MimeType, resolution, sampling frequency, frame rate, etc.:

for (int i = 0; i < mVideoExtractor.getTrackCount(); i++) {
    MediaFormat format = mVideoExtractor.getTrackFormat(i);
}

After obtaining the detailed information of the track, select the specified channel through selectTrack(int index):

if (format.getString(MediaFormat.KEY_MIME).startsWith("video/")) {
    mVideoExtractor.selectTrack(i);
    break;
}

After specifying the channel, you can read data from MediaExtractor:

while (true) {
    int sampleSize = mVideoExtractor.readSampleData(buffer, 0);
    if (sampleSize < 0) {
        break;
    }
    // do something

    mVideoExtractor.advance();  // 移动到下一帧
}

After reading, remember to release the resource:

mVideoExtractor.release();

The benefits of this article, free C++ audio and video learning materials package, technical video/code, including (audio and video development, interview questions, FFmpeg, webRTC, rtmp, hls, rtsp, ffplay, codec, push-pull stream, srs)↓↓↓ ↓↓↓See below↓↓Click at the bottom of the article to get it for free↓↓