In the previous article, we used FFmpeg+ANativeWindwo to realize video decoding and rendering, and the video screen has been displayed on SurfaceView. Students who have not read the previous article suggest reading it first: Android FFmpeg Development (2), Realizing Video Decoding and Rendering
In this article, we will decode and render the audio stream, so that we can achieve the effect of a more complete video player. The specific technical selection is as follows:
- Decode audio stream with FFmpeg
- Play audio PCM data using OpenSL ES
1. FFmpeg decodes the audio stream
The audio process is similar to the video decoding process as a whole. For the specific process, you can read the previous article. This article talks about the unique details related to audio decoding.
First, there is format conversion. In the previous article, we also did format conversion (YUV->RGBA) for video decoding. Similarly, for audio decoding, we also need to perform format conversion according to our expected format, as follows:
AVCodecContext *codecContext = getCodecContext();
mSwrContext = swr_alloc();
av_opt_set_int(mSwrContext, "in_channel_layout", codecContext->channel_layout, 0);
av_opt_set_int(mSwrContext, "out_channel_layout", AV_CH_LAYOUT_STEREO, 0);
av_opt_set_int(mSwrContext, "in_sample_rate", codecContext->sample_rate, 0);
av_opt_set_int(mSwrContext, "out_sample_rate", 44100, 0);
av_opt_set_int(mSwrContext, "in_sample_fmt", codecContext->sample_fmt, 0);
av_opt_set_int(mSwrContext, "out_sample_fmt", AV_SAMPLE_FMT_S16, 0);
swr_init(mSwrContext);
// resample
mNbSample = av_rescale_rnd(ACC_NB_SAMPLES, AUDIO_DST_SAMPLE_RATE,
codecContext->sample_rate,AV_ROUND_UP);
mDstFrameDataSize = av_samples_get_buffer_size(NULL, AUDIO_DST_CHANNEL_COUNTS,
mNbSample, DST_SAMPLE_FORMAT, 1);
// 分配OpenSL播放音频的帧内存
mAudioOutBuffer = (uint8_t *) malloc(mDstFrameDataSize);
We need to pass the original audio format (including channel, sampling rate, sampling format) and target audio format (including channel, sampling rate, sampling format) to the av_opt_set_int method, and then call swr_init to initialize the SwrContext context. Finally, call swr_convert and pass in SwrContext to get the audio in the target format. As follows:
void AudioDecoder::onFrameAvailable(AVFrame *frame) {
LOGD("AudioDecoder::onFrameAvailable frame=%p, frame->nb_samples=%d\n", frame, frame->nb_samples);
if (mAudioRender) {
// 将解码出来音频帧进行重采样，采样后数据存入mAudioOutBuffer中
int result = swr_convert(mSwrContext, &mAudioOutBuffer,mDstFrameDataSize / 2,
(const uint8_t **)frame->data, frame->nb_samples);
if (result > 0) {
// 利用OpenSL实现音频流渲染
mAudioRender->renderAudioFrame(mAudioOutBuffer, mDstFrameDataSize);
}
}
}
Finally, the audio PCM data to be rendered is stored in the buffer specified by mAudioOutBuffer. Next, give this batch of data to OpenSL ES, and let OpenSL ES realize the playback of audio PCM.
2. OpenSL ES rendering audio
Android's NDK provides the C interface of OpenSL ES, which can provide very powerful audio processing. Android official document: OpenSL ES
Before using the API of OpenSL ES, you need to import the header file of OpenSL ES, the code is as follows:
#include <SLES/OpenSLES.h>
#include <SLES/OpenSLES_Android.h>
Then write CMakeLists.txt, and link the so library corresponding to OpenSL ES in the link phase:
target_link_libraries( # Specifies the target library.
hello-ffmpeg
# Links the target library to the log library
# included in the NDK.
${log-lib}
android
OpenSLES
ffmpeg)
At this point, we can use the API of OpenSL ES. The specific steps are as follows:
1) Create an engine object interface
int OpenSLRender::createEngine() {
SLresult result = SL_RESULT_SUCCESS;
do {
result = slCreateEngine(&mEngineObj, 0, nullptr,
0, nullptr, nullptr);
if (result != SL_RESULT_SUCCESS) {
LOGD("OpenSLRender::createEngine slCreateEngine fail. result=%d\n", result);
break;
}
result = (*mEngineObj)->Realize(mEngineObj, SL_BOOLEAN_FALSE);
if (result != SL_RESULT_SUCCESS) {
LOGD("OpenSLRender::createEngine Realize fail. result=%d\n", result);
break;
}
result = (*mEngineObj)->GetInterface(mEngineObj, SL_IID_ENGINE, &mEngineEngine);
if (result != SL_RESULT_SUCCESS) {
LOGD("OpenSLRender::createEngine GetInterface fail. result=%d\n", result);
break;
}
} while (false);
return result;
}
2) Next create a mixer object
int OpenSLRender::createOutputMixer() {
SLresult result = SL_RESULT_SUCCESS;
do {
const SLInterfaceID mids[1] = {
SL_IID_ENVIRONMENTALREVERB};
const SLboolean mreg[1] = {
SL_BOOLEAN_FALSE};
result = (*mEngineEngine)->CreateOutputMix(mEngineEngine, &mOutputMixObj, 1, mids, mreg);
if (result != SL_RESULT_SUCCESS) {
LOGD("OpenSLRender::createOutputMixer CreateOutputMix fail. result=%d\n", result);
break;
}
result = (*mOutputMixObj)->Realize(mOutputMixObj, SL_BOOLEAN_FALSE);
if (result != SL_RESULT_SUCCESS) {
LOGD("OpenSLRender::createOutputMixer Realize fail. result=%d\n", result);
break;
}
} while (false);
return result;
}
3) Create an audio playback object and set a callback for the audio playback bufferQueue
int OpenSLRender::createAudioPlayer() {
SLDataLocator_AndroidSimpleBufferQueue android_queue = {
SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE, 2};
SLDataFormat_PCM pcm = {
SL_DATAFORMAT_PCM, // format type
(SLuint32)2, // channel count
SL_SAMPLINGRATE_44_1, // 44100HZ
SL_PCMSAMPLEFORMAT_FIXED_16, // bits per sample
SL_PCMSAMPLEFORMAT_FIXED_16, // container size
SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT, // channel mask
SL_BYTEORDER_LITTLEENDIAN // 小端字节
};
SLDataSource slDataSource = {
&android_queue, &pcm};
SLDataLocator_OutputMix outputMix = {
SL_DATALOCATOR_OUTPUTMIX, mOutputMixObj};
SLDataSink slDataSink = {
&outputMix, nullptr};
const SLInterfaceID ids[3] = {
SL_IID_BUFFERQUEUE, SL_IID_EFFECTSEND, SL_IID_VOLUME};
const SLboolean req[3] = {
SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE};
SLresult result;
do {
result = (*mEngineEngine)->CreateAudioPlayer(mEngineEngine, &mAudioPlayerObj, &slDataSource,
&slDataSink, 3, ids, req);
if (result != SL_RESULT_SUCCESS) {
LOGD("OpenSLRender::createAudioPlayer CreateAudioPlayer fail. result=%d\n", result);
break;
}
result = (*mAudioPlayerObj)->Realize(mAudioPlayerObj, SL_BOOLEAN_FALSE);
if (result != SL_RESULT_SUCCESS) {
LOGD("OpenSLRender::createAudioPlayer Realize mAudioPlayerObj fail. result=%d\n",
result);
break;
}
result = (*mAudioPlayerObj)->GetInterface(mAudioPlayerObj, SL_IID_PLAY, &mAudioPlayerPlay);
if (result != SL_RESULT_SUCCESS) {
LOGD("OpenSLRender::createAudioPlayer GetInterface SL_IID_PLAY fail. result=%d\n",
result);
break;
}
result = (*mAudioPlayerObj)->GetInterface(mAudioPlayerObj, SL_IID_BUFFERQUEUE,
&mBufferQueue);
if (result != SL_RESULT_SUCCESS) {
LOGD("OpenSLRender::createAudioPlayer GetInterface SL_IID_BUFFERQUEUE fail. result=%d\n",
result);
break;
}
result = (*mBufferQueue)->RegisterCallback(mBufferQueue, audioPlayerCallback, this);
if (result != SL_RESULT_SUCCESS) {
LOGD("OpenSLRender::createAudioPlayer RegisterCallback fail. result=%d\n", result);
break;
}
result = (*mAudioPlayerObj)->GetInterface(mAudioPlayerObj, SL_IID_VOLUME,
&mAudioPlayerVolume);
if (result != SL_RESULT_SUCCESS) {
LOGD("OpenSLRender::createAudioPlayer GetInterface SL_IID_VOLUME fail. result=%d\n",
result);
break;
}
} while (false);
return result;
}
One step in the above core process is to register a callback function for SLAndroidSimpleBufferQueueItf. This callback function will be called when OpenSL ES needs data to play. We need to fill the PCM data in the callback function.
4) Fill the PCM data, fill the audio PCM data through the Enqueue method of the SLAndroidSimpleBufferQueueItf interface, as follows:
// SLAndroidSimpleBufferQueueItf
(*mBufferQueue)->Enqueue(mBufferQueue, audioFrame->data, (SLuint32)audioFrame->dataSize);
5) Play
// SLPlayItf
(*mAudioPlayerPlay)->SetPlayState(mAudioPlayerPlay, SL_PLAYSTATE_PLAYING);
6) Finally, resources are released
void OpenSLRender::unInit() {
LOGD("OpenSLRender::unInit");
if (mAudioPlayerPlay) {
(*mAudioPlayerPlay)->SetPlayState(mAudioPlayerPlay, SL_PLAYSTATE_STOPPED);
mAudioPlayerPlay = nullptr;
}
if (mAudioPlayerObj) {
(*mAudioPlayerObj)->Destroy(mAudioPlayerObj);
mAudioPlayerObj = nullptr;
mBufferQueue = nullptr;
}
if (mOutputMixObj) {
(*mOutputMixObj)->Destroy(mOutputMixObj);
mOutputMixObj = nullptr;
}
if (mEngineObj) {
(*mEngineObj)->Destroy(mEngineObj);
mEngineObj = nullptr;
mEngineEngine = nullptr;
}
}
3. Summary
This article uses FFmepg+OpenSL ES to realize audio stream decoding and playback, so far a simple video player has been made.
Source link
git clone [email protected]:lorienzhang/HelloFFmpeg.git
# 检出 v3 tag 进行查看
git checkout v3