Introduction
MediaCodec is a class provided by Android for encoding and decoding audio and video, that is, the encoder/decoder component. It implements the codec function by accessing the underlying Codec. It is part of the Android media framework, usually used together with MediaExtractor, MediaSync, MediaMuxer, MediaCrypto, MediaDrm, Image, Surface and AudioTrack, and plays an important role in video playback and video compression encoding. For detailed api introduction, please refer to the official document
work process
In terms of the overall process, the MediaCodec codec processes the input data and then generates the output data. This process is asynchronous and uses a set of input and output buffers.
The process is as follows:
- The client requests or receives an empty input buffer (ByteBuffer) from MediaCodec, fills it with data and sends it to MediaCodec for processing.
- After the codec processes the data, it outputs it to an empty output buffer (ByteBuffer).
- The client gets a filled output buffer from the MediaCodec, gets its content and consumes it, then releases it back to the codec.
Introduction to commonly used APIs
-
createDecoderByType
creates a decoder based on MimeType -
configureConfigure
MediaCodec -
getInputBuffer(index)
gets the input stream queue that needs to encode data and returns a ByteBuffer -
dequeueInputBuffer(long timeoutUs)
returns the index of the input buffer filled with valid data, or -1 if no buffer is currently available. If timeoutUs == 0, this method will return immediately, if timeoutUs < 0, wait indefinitely for the availability of the input buffer, if timeoutUs > 0, wait until "timeoutUs" microseconds. -
queueInputBuffer
fills a certain range of data into the input buffer at the specified index -
dequeueOutputBuffer
takes data from the output buffer and returns the data index or some defined state constants -
getOutputBuffer(index)
returns the ByteBuffer of the specified index in the output buffer queue -
releaseOutputBuffer
processing completed, release ByteBuffer data
use
The following is an example of MediaCodec+MediaExrtractor+SurfaceView for video playback:
Note: audio and video need to be played separately
The first step is to initialize MediaExrtractor to obtain video track information from the video file
private fun init() {
try {
//创建MediaExtractor对象
videoExtractor = MediaExtractor()
//设置视频数据源，可以是本地文件也可以是网络文件
//注意，安卓9.0以上不允许http明文链接请求的地址，需要适配
videoExtractor?.setDataSource(videoPath!!)
val count = videoExtractor!!.trackCount //获取轨道数量
//视频
for (i in 0 until count) {
val mediaFormat = videoExtractor!!.getTrackFormat(i)
val mimeType = mediaFormat.getString(MediaFormat.KEY_MIME)
if (mimeType.startsWith("video/")) {
//获取到视频轨道
videoExtractor?.selectTrack(i)
initVideo(mediaFormat)
break
}
}
} catch (e: Exception) {
Log.e("Test", "出错了", e)
}
}
The second step is to initialize the MediaCodec video decoder
private fun initVideo(mediaFormat: MediaFormat) {
val mimeType = mediaFormat.getString(MediaFormat.KEY_MIME)
val codecInfo = getCodecInfo(mimeType)//获取支持的解码格式
if (codecInfo != null) {
handler.sendMessage(Message.obtain(handler, 100, mediaFormat))
//根据MimeType创建解码器
videoCodec = MediaCodec.createDecoderByType(mimeType)
//配置，在此关联SurfaceView
//参数说明：mediaFormat：视频信息，surface:surface容器，crypto:数据加密 flags:解码器/编码器
videoCodec?.configure(mediaFormat, sv_video.holder.surface, null, 0)
videoCodec?.start()//开始解码
} else {
Log.e("Test", "格式错误")
}
}
The third step is to create a thread to perform decoding and playback operations
private val decodeVideoRunnable = Runnable {
try {
//存放目标文件的数据
var byteBuffer: ByteBuffer? = null
//解码后的数据，包含每一个buffer的元数据信息，例如偏差，在相关解码器中有效的数据大小
val info = MediaCodec.BufferInfo()
// videoCodec!!.getOutputBuffers()
var first = false
var startWhen: Long = 0
var isInput = true
while (isDecoding) {
// Thread.sleep(20)//可以控制慢放
if (isInput) {
//1.准备一个输入缓冲区
val inIndex = videoCodec!!.dequeueInputBuffer(TIMEOUT)
if (inIndex >= 0) {
//2.准备填充数据
byteBuffer = videoCodec!!.getInputBuffer(inIndex)
//使用MediaExtractor读取视频数据
val sampleSize = videoExtractor!!.readSampleData(byteBuffer!!, 0)
if (videoExtractor!!.advance() && sampleSize > 0) {
//有数据，数据可用
//3.写入数据到输入缓冲区
videoCodec!!.queueInputBuffer(
inIndex,
0,
sampleSize,
videoExtractor!!.sampleTime,
0
)
} else {
//没有数据了，停止输入处理
videoCodec!!.queueInputBuffer(
inIndex,
0,
0,
0,
MediaCodec.BUFFER_FLAG_END_OF_STREAM
)
isInput = false
}
} else {
continue
}
}
//4 获取一个输出缓冲区，开始解码
val outIndex = videoCodec!!.dequeueOutputBuffer(info, TIMEOUT)
if (outIndex >= 0) {
when (outIndex) {
MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED -> {
Log.d("Test", "INFO_OUTPUT_BUFFERS_CHANGED")
videoCodec!!.getOutputBuffer(outIndex)
}
MediaCodec.INFO_OUTPUT_FORMAT_CHANGED ->
Log.d(
"Test",
"INFO_OUTPUT_FORMAT_CHANGED format : " + videoCodec!!.getOutputFormat()
)
MediaCodec.INFO_TRY_AGAIN_LATER -> {
}
else -> {
if (!first) {
startWhen = System.currentTimeMillis()
first = true
}
try {
val sleepTime: Long =
info.presentationTimeUs / 1000 - (System.currentTimeMillis() - startWhen)
if (sleepTime > 0) Thread.sleep(sleepTime)
} catch (e: InterruptedException) {
e.printStackTrace()
}
//对outputbuffer的处理完后，调用这个函数把buffer重新返回给codec类。
//调用这个api之后，SurfaceView才有图像
videoCodec!!.releaseOutputBuffer(outIndex, true)
}
}
}
}
//解码完毕，释放资源
videoCodec?.stop()
videoCodec?.release()
videoExtractor?.release()
} catch (e: Exception) {
Log.e("Test", "", e)
}
}
So far, the video playback is done. Let’s talk about some audio playback. In order not to conflict, different MediaExtractor and MediaCodec
configurations and start-up processes are actually similar to video decoding. The difference is that there are more AudioTracks.
//解码音频，进行播放
private val decodeAudioRunnable = Runnable {
try {
val inputBuffers: Array<ByteBuffer> = audioCodec!!.getInputBuffers()
var outputBuffers: Array<ByteBuffer> = audioCodec!!.getOutputBuffers()
val info = BufferInfo()
val buffsize = AudioTrack.getMinBufferSize(
sampleRate,
CHANNEL_OUT_STEREO,
ENCODING_PCM_16BIT
)
// 创建AudioTrack对象
var audioTrack: AudioTrack? = AudioTrack(
AudioManager.STREAM_MUSIC, sampleRate,
CHANNEL_OUT_STEREO,
ENCODING_PCM_16BIT,
buffsize,
MODE_STREAM
)
//启动AudioTrack
audioTrack!!.play()
while (isDecoding) {
val inIndex: Int = audioCodec!!.dequeueInputBuffer(TIMEOUT)
if (inIndex >= 0) {
val buffer = inputBuffers[inIndex]
//从MediaExtractor中读取待解数据
val sampleSize = audioExtractor!!.readSampleData(buffer, 0)
if (sampleSize < 0) {
audioCodec!!.queueInputBuffer(
inIndex, 0, 0, 0,
MediaCodec.BUFFER_FLAG_END_OF_STREAM
)
} else {
//向MediaDecoder输入待解码数据
audioCodec!!.queueInputBuffer(
inIndex,
0,
sampleSize,
videoExtractor!!.sampleTime,
0
)
audioExtractor!!.advance()
}
//从输出缓冲区队列取出解码后的数据
val outIndex: Int = audioCodec!!.dequeueOutputBuffer(info, TIMEOUT)
when (outIndex) {
MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED -> {
outputBuffers = audioCodec!!.getOutputBuffers()
}
MediaCodec.INFO_OUTPUT_FORMAT_CHANGED -> {
val format: MediaFormat = audioCodec!!.getOutputFormat()
audioTrack.playbackRate = format.getInteger(MediaFormat.KEY_SAMPLE_RATE)
}
MediaCodec.INFO_TRY_AGAIN_LATER -> Log.d(
"Test",
"dequeueOutputBuffer timed out!"
)
else -> {
val outBuffer = outputBuffers[outIndex]
//Log.v(TAG, "outBuffer: " + outBuffer);
val chunk = ByteArray(info.size)
// Read the buffer all at once
outBuffer[chunk]
//清空buffer,否则下一次得到的还会得到同样的buffer
outBuffer.clear()
// AudioTrack write data
audioTrack.write(chunk, info.offset, info.offset + info.size)
audioCodec!!.releaseOutputBuffer(outIndex, false)
}
}
// 所有帧都解码、播放完之后退出循环
if (info.flags and MediaCodec.BUFFER_FLAG_END_OF_STREAM != 0) {
break
}
}
}
//释放MediaDecoder资源
audioCodec?.stop()
audioCodec?.release()
audioCodec = null
audioExtractor?.release()
audioExtractor = null
//释放AudioTrack资源
audioTrack.stop()
audioTrack.release()
audioTrack = null
} catch (e: Exception) {
Log.e("Test", "", e)
}
}
Finally, let me explain a key point: the determination of the video width and height.
Here I use MediaExtractor to get MediaFormat to get the video width and height. In actual development, the width and height of a player are generally fixed, and then the SurfaceView is scaled and filled.
/**
* 根据视频大小改变SurfaceView大小
*/
private fun changeVideoSize(mediaFormat: MediaFormat) {
var videoWidth = mediaFormat.getInteger(MediaFormat.KEY_WIDTH) //获取高度
var videoHeight = mediaFormat.getInteger(MediaFormat.KEY_HEIGHT) //获取高度
val surfaceWidth = sv_video.measuredWidth
val surfaceHeight = sv_video.measuredHeight
//根据视频尺寸去计算->视频可以在sufaceView中放大的最大倍数。
var maxSize: Double
maxSize =
if (resources.configuration.orientation == ActivityInfo.SCREEN_ORIENTATION_PORTRAIT) {
//竖屏模式下按视频宽度计算放大倍数值
max(videoWidth / surfaceWidth.toDouble(), videoHeight / surfaceHeight.toDouble())
} else {
//横屏模式下按视频高度计算放大倍数值
max(videoWidth / surfaceHeight.toDouble(), videoHeight / surfaceWidth.toDouble())
}
//视频宽高分别/最大倍数值 计算出放大后的视频尺寸
videoWidth = Math.ceil(videoWidth / maxSize).toInt();
videoHeight = Math.ceil(videoHeight / maxSize).toInt();
//将计算出的视频尺寸设置到surfaceView 让视频自动填充。
sv_video.layoutParams = ConstraintLayout.LayoutParams(videoWidth, videoHeight);
}
Please see the complete code: VideoDemo