Audio Capture: AudioRecord
video capture: Camera preview callback YUV data
encoding: MediaCodec
Synthesis packet MP4: MediaMuxer
First, determine a few threads processing tasks
1.audioThread audio capture and encoding
2.videoThread video coding
3.muxerThread synthesis
Sample Code: Kotlin
All detailed code uploaded github, will give an address later, it is an example of Activity Camera1PreviewActivity
Less some validation code, such as the device supports preview format, which mentioned in the previous article, pay attention to their device supports this setting.
In the last issue, it will be prone to write the code does not operate correctly when the next can control whether or not
any of these mistakes
1. Turn the camera on and initialization
Preview interface with SurfaceView, through the front of the camera preview learning should know, do not say
private fun initView() {
surfaceView = findViewById(com.example.mediastudyproject.R.id.surface_view)
surfaceView.holder.addCallback(object : SurfaceHolder.Callback2 {
override fun surfaceRedrawNeeded(holder: SurfaceHolder?) {
}
override fun surfaceChanged(
holder: SurfaceHolder?,
format: Int,
width: Int,
height: Int
) {
isSurfaceAvailiable = true
[email protected] = holder
}
override fun surfaceDestroyed(holder: SurfaceHolder?) {
isSurfaceAvailiable = false
mCamera?.stopPreview()
//这里要把之前设置的预览回调取消，不然关闭app，camera释放了，但是还在回调，会报异常
mCamera?.setPreviewCallback(null)
mCamera?.release()
mCamera = null
}
override fun surfaceCreated(holder: SurfaceHolder?) {
isSurfaceAvailiable = true
[email protected] = holder
thread {
//打开相机
openCamera(Camera.CameraInfo.CAMERA_FACING_BACK)
}
}
})
}
The camera parameter settings
/**
* 初始化并打开相机，我这里默认打开的后置摄像头
*/
private fun openCamera(cameraId: Int) {
mCamera = Camera.open(cameraId)
mCamera?.run {
setPreviewDisplay(holder)
setDisplayOrientation(WindowDegree.getDegree(this@Camera1PreviewActivity))
var cameraInfo = Camera.CameraInfo()
Camera.getCameraInfo(cameraId, cameraInfo)
Log.i("camera1", "相机方向 ${cameraInfo.orientation}")
val parameters = parameters
parameters?.run {
//自动曝光结果给我爆一团黑，不能忍 自己设置
exposureCompensation = maxExposureCompensation
//自动白平衡
autoWhiteBalanceLock = isAutoWhiteBalanceLockSupported
//设置预览大小
appropriatePreviewSizes = getAppropriatePreviewSizes(parameters)
setPreviewSize(appropriatePreviewSizes?.width!!, appropriatePreviewSizes?.height!!)
//设置对焦模式
val supportedFocusModes = supportedFocusModes
if (supportedFocusModes.contains(Camera.Parameters.FOCUS_MODE_AUTO)) {
//设置自动对焦，启动自动对焦是通过Camera的autoFocus方法实现
//如果要连续对焦，这个方法要多次调用，这里就没有调用autoFocus
//想要连续对焦的可以自己实现，通过Handler连续发送消息就行
focusMode = Camera.Parameters.FOCUS_MODE_AUTO
}
previewFormat = ImageFormat.NV21
}
//相机资源回收的时候，注意setPreviewCallBack(null)，将回调移除
setPreviewCallback { data, camera ->
//isRecording是一个开启录制的标志，回调帧数据存放在集合中等待编码器编码
if (isRecording) {
if (data != null) {
Log.i("camera1", "获取视频数据 ${data.size}")
Log.i("camera1", "视频线程是否为 $videoThread")
videoThread.addVideoData(data)
}
}
}
//开始预览
startPreview()
}
}
To avoid the article is too long, some of the code is not posted, can go directly to github view, getAppropriatePreviewSizes (parameters) are not posted.
2. Video processing thread
format YUV video data set is the NV21, Camera1 the API can return to this, but Camera2 is not supported by the video encoding best NV12 data, and finally to convert it, the video thread is mainly to do is get the data converted into NV12 -> encoded as H264 -> write Muxer
/**
*代码没有分离，直接在Activity创建的内部类，想要代码更简洁的可以分开
**/
inner class VideoEncodeThread : Thread() {
//预览的数据就直接添加到这个集合中
private val videoData = LinkedBlockingQueue<ByteArray>()
fun addVideoData(byteArray: ByteArray) {
videoData.offer(byteArray)
}
override fun run() {
super.run()
//创建编码用的MediaFormat，下面贴出
initVideoFormat()
//创建视频编码器MediaCodec
videoCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_AVC)
videoCodec!!.configure(videoMediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
videoCodec!!.start()
//如果未设置结束，就循环编码数据
while (!videoExit) {
val poll = videoData.poll()
if (poll != null) {
encodeVideo(poll, false)
}
}
//发送编码结束标志
encodeVideo(ByteArray(0), true)
//注意释放资源
videoCodec!!.release()
Log.i("camera1", "视频释放")
}
}
Initialization MediaFormat
private fun initVideoFormat() {
videoMediaFormat =
MediaFormat.createVideoFormat(
MediaFormat.MIMETYPE_VIDEO_AVC,
appropriatePreviewSizes!!.width,
appropriatePreviewSizes!!.height
)
//设置颜色类型 5.0新加的颜色格式
videoMediaFormat.setInteger(
MediaFormat.KEY_COLOR_FORMAT,
MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible
)
//设置帧率
videoMediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 30)
//设置比特率
videoMediaFormat.setInteger(
MediaFormat.KEY_BIT_RATE,
appropriatePreviewSizes!!.width * appropriatePreviewSizes!!.height * 5
)
//设置每秒关键帧间隔
videoMediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5)
}
Video coding (synchronous mode)
private fun encodeVideo(data: ByteArray, isFinish: Boolean) {
val videoArray = ByteArray(data.size)
if (!isFinish) {
//NV21转NV12 网上找的，他两不同就是排列方式一个是VUVUVU一个是UVUVUV
//具体看github代码
NV21toI420SemiPlanar(
data,
videoArray,
appropriatePreviewSizes!!.width,
appropriatePreviewSizes!!.height
)
}
val videoInputBuffers = videoCodec!!.inputBuffers
var videoOutputBuffers = videoCodec!!.outputBuffers
//这个TIME_OUT_US设置的是0.01s也就是10000微秒，之前设置成1s，结果视频掉帧
//严重，声音也播放不了，说明这个值不能设置太大
val index = videoCodec!!.dequeueInputBuffer(TIME_OUT_US)
if (index >= 0) {
val byteBuffer = videoInputBuffers[index]
byteBuffer.clear()
byteBuffer.put(videoArray)
if (!isFinish) {
videoCodec!!.queueInputBuffer(index, 0, videoArray.size, System.nanoTime()/1000, 0)
} else {
videoCodec!!.queueInputBuffer(
index,
0,
0,
System.nanoTime()/1000,
MediaCodec.BUFFER_FLAG_END_OF_STREAM
)
}
val bufferInfo = MediaCodec.BufferInfo()
Log.i("camera1", "编码video $index 写入buffer ${videoArray?.size}")
var dequeueIndex = videoCodec!!.dequeueOutputBuffer(bufferInfo, TIME_OUT_US)
//这里需要注意，MediaMuxer要设置的音视频MediaFormat要在这里获取，设置过了就不用重新在更改
//如果不使用在这里获取的MediaFormat，极有可能最后MediaMuxer关闭时候出现关闭失败异常
if (dequeueIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
if (MuxThread.videoMediaFormat == null)
MuxThread.videoMediaFormat = videoCodec!!.outputFormat
}
if (dequeueIndex == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
videoOutputBuffers = videoCodec!!.outputBuffers
}
while (dequeueIndex >= 0) {
val outputBuffer = videoOutputBuffers[dequeueIndex]
//由于配置性信息在之前的MediaFormat已经包含，这里就不需要写入MediaMuxer了
if (bufferInfo.flags and MediaCodec.BUFFER_FLAG_CODEC_CONFIG != 0) {
bufferInfo.size = 0
}
//将编码数据加入队列等待Muxer写入
if (bufferInfo.size != 0) {
muxerThread?.addVideoData(outputBuffer, bufferInfo)
}
Log.i(
"camera1",
"编码后video $dequeueIndex buffer.size ${bufferInfo.size} buff.position ${outputBuffer.position()}"
)
videoCodec!!.releaseOutputBuffer(dequeueIndex, false)
//检查是否结束
if (bufferInfo.flags and MediaCodec.BUFFER_FLAG_END_OF_STREAM != 0) {
break
} else{
dequeueIndex = videoCodec!!.dequeueOutputBuffer(bufferInfo, TIME_OUT_US)
}
}
}
}
3. Audio threads
audio thread needs to do two things, get audio data -> encoded as AAC -> ready to write Muxer, process video and
similar, there is not much to explain the steps
AudioRecord recording ready
inner class AudioThread : Thread() {
private val audioData = LinkedBlockingQueue<ByteArray>()
fun addVideoData(byteArray: ByteArray) {
audioData.offer(byteArray)
}
override fun run() {
super.run()
prepareAudioRecord()
}
}
/**
* 准备初始化AudioRecord
*/
private fun prepareAudioRecord() {
initAudioFormat()
audioCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_AUDIO_AAC)
audioCodec!!.configure(audioMediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
audioCodec!!.start()
//创建audiorecord对象，配置文件都在AudioCongfig中，minsize是根据系统方法算出，请查看github
audioRecorder = AudioRecord(
MediaRecorder.AudioSource.MIC, AudioConfig.SAMPLE_RATE,
AudioConfig.CHANNEL_CONFIG, AudioConfig.AUDIO_FORMAT, minSize
)
if (audioRecorder!!.state == AudioRecord.STATE_INITIALIZED) {
audioRecorder?.run {
startRecording()
val byteArray = ByteArray(SAMPLES_PER_FRAME)
var read = read(byteArray, 0, SAMPLES_PER_FRAME)
while (read > 0 && isRecording) {
Log.i("camera1", "读取到的音频 $read")
//音频数据的时间戳需要在读取的时候去获得，getPTSUs是获取当前系统纳秒表示时间
encodeAudio(byteArray, read, getPTSUs())
//读取的字节大小如果使用minSize，也就是计算得到的最小大小，编码合成后
//播放会没有声音，时间戳就不对，很可能这个大小的数据超过一帧数据大小，
//有待研究，1024和2048都能播放
read = read(byteArray, 0, SAMPLES_PER_FRAME)
}
audioRecorder!!.release()
//发送EOS编码结束信息
encodeAudio(ByteArray(0), 0, getPTSUs())
Log.i("camera1", "音频释放")
audioCodec!!.release()
}
}
}
Audio Coding (synchronous mode)
/***
* @param 音频数据个数
*/
private fun encodeAudio(audioArray: ByteArray?, read: Int, timeStamp: Long) {
val index = audioCodec!!.dequeueInputBuffer(TIME_OUT_US)
val audioInputBuffers = audioCodec!!.inputBuffers
if (index >= 0) {
val byteBuffer = audioInputBuffers[index]
byteBuffer.clear()
byteBuffer.put(audioArray, 0, read)
if (read != 0) {
audioCodec!!.queueInputBuffer(index, 0, read, timeStamp, 0)
} else {
audioCodec!!.queueInputBuffer(
index,
0,
read,
timeStamp,
MediaCodec.BUFFER_FLAG_END_OF_STREAM
)
}
val bufferInfo = MediaCodec.BufferInfo()
Log.i("camera1", "编码audio $index 写入buffer ${audioArray?.size}")
var dequeueIndex = audioCodec!!.dequeueOutputBuffer(bufferInfo, TIME_OUT_US)
if (dequeueIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
if (MuxThread.audioMediaFormat == null) {
MuxThread.audioMediaFormat = audioCodec!!.outputFormat
}
}
var audioOutputBuffers = audioCodec!!.outputBuffers
if (dequeueIndex == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
audioOutputBuffers = audioCodec!!.outputBuffers
}
while (dequeueIndex >= 0) {
val outputBuffer = audioOutputBuffers[dequeueIndex]
Log.i(
"camera1",
"编码后audio $dequeueIndex buffer.size ${bufferInfo.size} buff.position ${outputBuffer.position()}"
)
if (bufferInfo.flags and MediaCodec.BUFFER_FLAG_CODEC_CONFIG != 0) {
bufferInfo.size = 0
}
if (bufferInfo.size != 0) {
Log.i("camera1","音频时间戳 ${bufferInfo.presentationTimeUs /1000}")
muxerThread?.addAudioData(outputBuffer, bufferInfo)
}
audioCodec!!.releaseOutputBuffer(dequeueIndex, false)
if (bufferInfo.flags and MediaCodec.BUFFER_FLAG_END_OF_STREAM != 0) {
break
} else {
dequeueIndex = audioCodec!!.dequeueOutputBuffer(bufferInfo, TIME_OUT_US)
}
}
}
}
Video encoding process and basically the same
4.MediaMuxer synthetic thread
MediaMuxer separate thread I made, and create a class, his task is to
create MediaMuxer Object -> Get audio and video MediaFormat to add audio and video track -> Opens synthesis ->
Gets a collection of data, write
class MuxThread(val context: Context) : Thread() {
private val audioData = LinkedBlockingQueue<EncodeData>()
private val videoData = LinkedBlockingQueue<EncodeData>()
companion object {
var muxIsReady = false
var audioMediaFormat: MediaFormat? = null
var videoMediaFormat: MediaFormat? = null
var muxExit = false
}
private lateinit var mediaMuxer: MediaMuxer
fun addAudioData(byteBuffer: ByteBuffer, bufferInfo: MediaCodec.BufferInfo) {
audioData.offer(EncodeData(byteBuffer, bufferInfo))
}
fun addVideoData(byteBuffer: ByteBuffer, bufferInfo: MediaCodec.BufferInfo) {
videoData.offer(EncodeData(byteBuffer, bufferInfo))
}
private fun initMuxer() {
val file = File(context.filesDir, "muxer.mp4")
if (!file.exists()) {
file.createNewFile()
}
mediaMuxer = MediaMuxer(
file.path,
MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4
)
audioAddTrack = mediaMuxer.addTrack(audioMediaFormat)
videoAddTrack = mediaMuxer.addTrack(videoMediaFormat)
//注意添加轨道，必须在start之前进行
mediaMuxer.start()
muxIsReady = true
}
private fun muxerParamtersIsReady() = audioMediaFormat != null && videoMediaFormat != null
override fun run() {
super.run()
//判断音视频MediaFormat是否都获取到了
while (!muxerParamtersIsReady()) {
}
//初始化，添加音视频轨道，开启合成
initMuxer()
Log.i("camera1", "当前记录状态 $isRecording ")
while (!muxExit) {
if (audioAddTrack != -1) {
if (audioData.isNotEmpty()) {
val poll = audioData.poll()
Log.i("camera1", "混合写入音频 ${poll.bufferInfo.size} ")
mediaMuxer.writeSampleData(audioAddTrack, poll.buffer, poll.bufferInfo)
}
}
if (videoAddTrack != -1) {
if (videoData.isNotEmpty()) {
val poll = videoData.poll()
Log.i("camera1", "混合写入视频 ${poll.bufferInfo.size} ")
mediaMuxer.writeSampleData(videoAddTrack, poll.buffer, poll.bufferInfo)
}
}
}
//写入完成，释放
mediaMuxer.stop()
mediaMuxer.release()
Log.i("camera1", "合成器释放")
Log.i("camera1", "未写入音频 ${audioData.size}")
Log.i("camera1", "未写入视频 ${videoData.size}")
}
}
These places are the main processes of this series, where the following points in mind to write, and it is likely to cause a program error
1. audio recording and coding, provided the read size can not be calculated using the minimum size, otherwise there will be play
no sound, 1024 or 2048 bytes encode a correct result can be obtained
2.MediaCodec encoding of available Buffer waiting time is not too large, otherwise there will encode the video after skipping
serious, there is no audio sound
3.MediaMuxer acquired MediaFormat MediaCodec preferably in the encoding process, by substituting the above-described
way to obtain the presented code, otherwise possible missing specific data, failure to close the abnormal MediaMuxer
4.MediaMuxer add audio and video tracks, must be completed before the start
SetPreviewCallback 5.Camera Camera settings at the time of the release of resources, but also to have it released by
setPreviewCallback (null), otherwise it will report Camera still being used, after the call to release Camera
abnormal
6. Set the size of the preview data, the system must be given, supported by the system size, Camera1 by
parameters.getSupportedPreviewSizes acquisition preview is sized system does not support, the video recording
is possible problems
Address github project, code Camera1PreviewActivity
References
to ask each AAC frame for a long time
In MediaMuxer and Android MediaCodec use cases - audio + video