音视频学习系列第（六）篇---音视频的分离与合成

什么是音视频的分离和合成

分离就是将视频1的声音和图像分别取出来
合成就是将视频1的图像和非视频1的声音组合成一个新的视频

如何进行音视频的分离和合成

安卓提供了两个API来帮助我们完成这个操作
MediaExtractor用于分离视频
MediaMuxer用于合成视频

下面我就来介绍一下这两个API的使用

MediaExtractor

分离音频
1.设置音频源

 MediaExtractor  audioExtractor = new MediaExtractor();
 audioExtractor.setDataSource(audioPath);

2.获取源文件中轨道的数量，并遍历找到我们需要的音频轨

  for (int i = 0; i < audioExtractor.getTrackCount(); i++) {
      MediaFormat format = audioExtractor.getTrackFormat(i);
      if (format.getString(MediaFormat.KEY_MIME).startsWith("audio/")) { srcATrackIndex = i; audioTrackIndex = muxer.addTrack(format); break; } }

分离视频
1.设置视频源

MediaExtractor  videoExtractor = new MediaExtractor();
videoExtractor.setDataSource(videoPath);

2.获取视频源文件中的轨道数，并遍历找到我们所需要的视频轨

for (int i = 0; i < videoExtractor.getTrackCount(); i++) {
       MediaFormat format = videoExtractor.getTrackFormat(i);
       if (format.getString(MediaFormat.KEY_MIME).startsWith("video/")) { srcVTrackIndex = i; videoTrackIndex = muxer.addTrack(format); break; } }

音频和视频额的分离方法一模一样，区别在于MediaFormat类型的不同，MediaFormat封装了媒体的数据格式信息

MediaMuxer

如何合成视频？
1.设置合成后视频的路径和格式

MediaMuxer  muxer = new MediaMuxer(outPath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);

2.将MediaExtractor分离出来的音轨和视轨添加到自己的轨道中

audioTrackIndex = muxer.addTrack(format);
videoTrackIndex = muxer.addTrack(format);

3.添加完所有轨道后start

muxer.start();

4.采集音频源的音轨样本

audioExtractor.selectTrack(srcATrackIndex);   //移动到音频轨上
        if (audioTrackIndex != -1) {
            MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
            info.presentationTimeUs = 0; ByteBuffer buffer = ByteBuffer.allocate(100 * 1024); while (true) { int sampleSize = audioExtractor.readSampleData(buffer, 0); if (sampleSize < 0) { //没有可获取的样本，退出循环 break; } info.offset = 0; info.size = sampleSize; info.flags = MediaCodec.BUFFER_FLAG_SYNC_FRAME; info.presentationTimeUs = audioExtractor.getSampleTime(); muxer.writeSampleData(audioTrackIndex, buffer, info);//将样本写入新的轨道 audioExtractor.advance(); //进入下一个样本 } }

5.采集视频源的视频轨样本

videoExtractor.selectTrack(srcVTrackIndex);   //移动到视频轨上
        if (videoTrackIndex != -1) {
            MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
            info.presentationTimeUs = 0; ByteBuffer buffer = ByteBuffer.allocate(100 * 1024); while (true) { int sampleSize = videoExtractor.readSampleData(buffer, 0); if (sampleSize < 0) { //没有可获取的样本，退出循环 break; } info.offset = 0; info.size = sampleSize; info.flags = MediaCodec.BUFFER_FLAG_SYNC_FRAME; info.presentationTimeUs = videoExtractor.getSampleTime(); muxer.writeSampleData(videoTrackIndex, buffer, info);//将样本写入新的轨道 videoExtractor.advance(); //进入下一个样本 } }

6.停止并释放资源

muxer.stop();
muxer.release();

完整代码

public class MediaUtil {

private static final String TAG = "MediaUtil"; /** * @param audioPath 音频文件路劲 * @param videoPath 视频文件路径 * @param outPath 合成之后的保存路径 */ public static void combineVideo(String audioPath, String videoPath, String outPath) { MediaMuxer muxer = null; MediaExtractor audioExtractor = null; MediaExtractor videoExtractor = null; try { muxer = new MediaMuxer(outPath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4); //找到音频文件的音频轨 audioExtractor = new MediaExtractor(); audioExtractor.setDataSource(audioPath); int srcATrackIndex = -1; //音频源的音频轨 int audioTrackIndex = -1; //音频轨添加到muxer后返回的新的轨道 //在此循环，目的是找到我们需要的音频轨 for (int i = 0; i < audioExtractor.getTrackCount(); i++) { MediaFormat format = audioExtractor.getTrackFormat(i); if (format.getString(MediaFormat.KEY_MIME).startsWith("audio/")) { srcATrackIndex = i; audioTrackIndex = muxer.addTrack(format); break; } } Log.d(TAG,"音频轨源索引："+srcATrackIndex+" 音频轨新索引:"+audioTrackIndex); //找到视频文件的视频轨 videoExtractor = new MediaExtractor(); videoExtractor.setDataSource(videoPath); int srcVTrackIndex = -1; //视频源的视频轨 int videoTrackIndex = -1; //视频轨添加到muxer后返回的新的轨道 for (int i = 0; i < videoExtractor.getTrackCount(); i++) { MediaFormat format = videoExtractor.getTrackFormat(i); if (format.getString(MediaFormat.KEY_MIME).startsWith("video/")) { srcVTrackIndex = i; videoTrackIndex = muxer.addTrack(format); break; } } Log.d(TAG,"视频轨源索引："+srcVTrackIndex+" 视频频轨新索引:"+videoTrackIndex); //添加完所有轨道后start muxer.start(); Log.d(TAG,"开始合成视频..."); //封装音频track audioExtractor.selectTrack(srcATrackIndex); //移动到音频轨上 if (audioTrackIndex != -1) { MediaCodec.BufferInfo info = new MediaCodec.BufferInfo(); info.presentationTimeUs = 0; ByteBuffer buffer = ByteBuffer.allocate(100 * 1024); while (true) { int sampleSize = audioExtractor.readSampleData(buffer, 0); if (sampleSize < 0) { //没有可获取的样本，退出循环 break; } info.offset = 0; info.size = sampleSize; info.flags = MediaCodec.BUFFER_FLAG_SYNC_FRAME; info.presentationTimeUs = audioExtractor.getSampleTime(); muxer.writeSampleData(audioTrackIndex, buffer, info);//将样本写入新的轨道 audioExtractor.advance(); //进入下一个样本 } } Log.d(TAG,"音频轨样本采集完成"); //封装视频track videoExtractor.selectTrack(srcVTrackIndex); //移动到视频轨上 if (videoTrackIndex != -1) { MediaCodec.BufferInfo info = new MediaCodec.BufferInfo(); info.presentationTimeUs = 0; ByteBuffer buffer = ByteBuffer.allocate(100 * 1024); while (true) { int sampleSize = videoExtractor.readSampleData(buffer, 0); if (sampleSize < 0) { //没有可获取的样本，退出循环 break; } info.offset = 0; info.size = sampleSize; info.flags = MediaCodec.BUFFER_FLAG_SYNC_FRAME; info.presentationTimeUs = videoExtractor.getSampleTime(); muxer.writeSampleData(videoTrackIndex, buffer, info);//将样本写入新的轨道 videoExtractor.advance(); //进入下一个样本 } } muxer.stop(); Log.d(TAG,"视频轨样本采集完成"); Log.d(TAG,"视频合成完毕："+outPath); } catch (IOException e) { e.printStackTrace(); Log.d(TAG,"合成出错:"+e.getMessage()); } finally { //释放资源 if(audioExtractor!=null){ audioExtractor.release(); } if(videoExtractor!=null){ videoExtractor.release(); } if(muxer!=null){ muxer.release(); } } } }

踩坑总结

问题1
MPEG4Writer: Unsupported mime 'audio/mpeg'
当音频文件是mp3时会报这个错误 ,需要的格式是AAC，m4a
stackoverflow
我将两个源文件都替换了mp4格式后不再报错了

问题2
WVMExtractor: Failed to open libwvm.so: dlopen failed: library "libwvm.so" not found
小米5 6.0.1机型上出现的问题，意思是缺少so库

由于我就是这款手机，也没测过其他手机有木有这个so库
后续我会用其他手机在测试这个问题

代码地址

MediaUtil类的封装放置在libplayer下的util包下
方法调用放置在app/demo/media/track下