Foreword
Almost two months did not write a blog, recently changed jobs, always wanted something before the audio and video aspects of the research study, finally have the opportunity to achieve, so recently would have been writing this series. I just groping for some time, what is the problem we discuss correction together.
The first chapter is the concept of literacy articles, if you do not know some common API inevitable after the difficult start.
text
Due to the special FFmpeg fire, so I always thought that little audio-video and native Android API relationship, in fact, this understanding is wrong, FFmpeg advantage is to solve the different versions of the Android API performance gaps and problems, try on different models achieve a similar effect on performance, certainly better than the third-party libraries to update the system version update more convenient.
So if API for native has a skilled master, then over to the FFmpeg is a small problem. Today we understand the concept of some common API.
MediaPlayer
Top audio and video playback API can be used to play audio and video files, especially common API.
// 创建MediaPlayer
val mediaPlayer = MediaPlayer()
// 设置要播放的文件路径
mediaPlayer.setDataSource(path)
// 准备
mediaPlayer.prepare()
// 播放
mediaPlayer.start()
There are some pause, stop approach, not presented here, I believe we have used.
MediaRecoder
Audio and video recording of the upper layer API, through some simple configuration, you can record audio and video saved to the specified file path directly.
// 创建MediaRecorder
val mediaRecorder = MediaRecorder()
// 录制声音的来源，具体参考AudioSource
mediaRecorder.setAudioSource(MediaRecorder.AudioSource.MIC);
// 录制视频的来源
mediaRecorder.setVideoSource(MediaRecorder.VideoSource.SURFACE)
// 输出的文件编码格式
mediaRecorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4)
// 保存的文件路径
mediaRecorder.setOutputFile(videoRecorderFile)
// 设置录音编码器
// 注意设置的录制音频编码格式与视频的编码格式是否匹配
mediaRecorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB)
// 音频录制的其他设置
mediaRecorder.setAudioEncodingBitRate(60)
mediaRecorder.setAudioSamplingRate(14400)
// 视频录制的其他设置
mediaRecorder.setVideoSize(getScreenWidth(), getScreenHeight())
mediaRecorder.setVideoEncodingBitRate(2 * getScreenWidth().times(getScreenHeight()))
mediaRecorder.setVideoFrameRate(60)
mediaRecorder.setVideoEncoder(MediaRecorder.VideoEncoder.H264)
mediaRecorder.prepare()
mediaRecorder.start()
// 停止录制
mediaRecorder!!.stop()
mediaRecorder!!.release()
They are basic configuration, but if it is a video recording and Surface generally used together with the recorded content obtained from the camera, there is also a corresponding method MediaRecorder settings:
// 设置摄像头
mediaRecorder.setCamera(camera!!)
// 设置录制的角度，如果与摄像头不符，会出现视频角度不对的问题
mediaRecorder.setOrientationHint(90);
// 设置录音和录制视频的来源
mediaRecorder.setAudioSource(MediaRecorder.AudioSource.CAMCORDER)
mediaRecorder.setVideoSource(MediaRecorder.VideoSource.CAMERA)
// 设置录制的质量
mediaRecorder.setProfile(CamcorderProfile.get(CamcorderProfile.QUALITY_HIGH))
....
There are many online Demo, you can go to search, then we will write yourself a write Demo.
AudioRecord
Recorded audio API, the audio data stream in the form of output is not encoded, that is, the original PCM data, directly save the recording can not be used directly out of the player for playback.
// 创建AudioRecord
AudioRecord(
// 录制音频的来源，参数与MediaRecoder相同
AudioSource,
// 采样率，
sampleRateInH,
// 声音的频道
CHANNEL,
// 编码位数
ENCODING,
// 录制一帧的最小Buffer大小
AudioRecord.getMinBufferSize(
sampleRateInH,
CHANNEL,
ENCODING
)
)
As can be seen, AudioRecorder compared MediaRecoder to be more careful if you have to have some special audio processing, such as voice changed, you need to use to AudioRecorder, then save your own coding.
Then we look at the argument constructor method:
AudioSource: Recorded sound source, with MediaRecoder is the same.
·
SampleRateInH: sampling rate, the popular talk sampling frequency is the number of computer signal samples per second collection, the unit is Hz. The higher the sound is definitely more clear, there is commonly used size:
8000Hz sampling rate used by the phone, for people who speak enough
sound to get 11025Hz is called a telephone quality, basically allows you to distinguish the voice of caller
22050Hz sampling radio broadcasting rate, broadcast quality
32000Hz miniDV digital video camcorder, DAT (LPmode) sampling rate used
44100Hz audio CD, also commonly used in audio MPEG-1 (VCD, SVCD, MP3) sampling rate used in
the development of 47250Hz NipponColumbia (Denon) is the world's first commercial PCM recorder used sampling rate
digital audio 48000Hz miniDV, digital TV, DVD, DAT, films and professional audio used in the sampling rate
.
the cHANNEL: is the stereo video, left channel, right channel
.
the eNCODING: this code refers to the save digits, generally use 16-bit.
.
AudioRecord.getMinBufferSize: minimum size of each frame, if it is a read operation, it is necessary to use.
MediaCodec
Audio and video encoders and decoders, this should be the most important series of API, in order to match his use, have used other API MediaFormat and so on.
// 这里仅仅是一个简单的例子，MediaCodec的使用要更复杂一些
// 创建一个avc格式的视频编码器
mediaCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_AVC)
// 详细配置MediaFormat
val mediaFormat = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, width, height)
mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, 125000)
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 15)
mediaFormat.setInteger(
MediaFormat.KEY_COLOR_FORMAT,
MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface
)
mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5)
// 使用配置好的MediaFormat
mediaCodec!!.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
// 开始编码
mediaCodec!!.start()
In practical applications, MediaCodec more complicated to use, simple to understand here is, in our case after slowly analysis.
MediaExtractor
For the extraction of audio and video, such as a video file to extract audio saved as an audio file.
// 创建一个分离器，还没有指定要分离的部分
val mediaExtractor = MediaExtractor()
// 文件路径
mediaExtractor.setDataSource(file.absolutePath)
// trackCount表示有多少可以分离的轨道，常用的音轨和视轨
val trackNum = mediaExtractor.trackCount
for (i in 0 until trackNum) {
// 对应轨道的格式
val mediaFormat = mediaExtractor.getTrackFormat(i)
// 音频：“audio/”，视频：“video/”
val format = mediaFormat.getString("mime")
if (format.startsWith(prefix)) {
// 选择需要的轨道，接下来就可以提取这个轨道的内容了
videoMediaExtractor.selectTrack(videoIndex)
}
}
// 读取内容
val frameSize = mediaExtractor.readSampleData(mReadBuffer, 0)
...
// 释放资源
mediaExtractor.release()
MediaMuxer
Audio and video combiner, video and audio synthesis, the synthetic video, etc., commonly used in conjunction with just MediaExtractor
mediaMuxer = MediaMuxer(outPath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4)
val mReadBuffer = ByteBuffer.allocate(MAX_BUFF_SIZE)
// 选择音轨
val audioMediaExtractor = MediaExtractor()
audioMediaExtractor.setDataSource(path)
val audioIndex = findTrackIndex(audioMediaExtractor, "audio/")
audioMediaExtractor.selectTrack(audioIndex)
// 读取帧数据
val frameSize = mediaExtractor.readSampleData(mReadBuffer, 0)
mReadBuffer.rewind()
// 将数据写入到合成文件中
mediaMuxer!!.writeSampleData(outTrackIndex, mReadBuffer, bufferInfo)
// 释放
mediaMuxer!!.release()
to sum up
Today we understand the common API Android native, the API is sufficient to meet the needs of our development, then we have to complete the following tasks:
- MediaRecoder record audio and video
- MediaPlayer to play audio and video
- MediaCodec play audio and video
- AudioRecoder PCM audio recording and playback
- MediaCodec record video
- Extracting video and audio synthesis
- Breakpoints recording and video compositing
(Above tasks in no particular order, the order may become)