FFmpeg Development (3): Implementation of Audio Player

OpenSL basics

The OpenSLES interface can directly load the Native layer to process audio data, reducing the data copying from the Java layer to the Native layer in the process of collection, playback and encoding and decoding.
OpenSLES document: OpenSL_ES_Specification1.1.pdf

advantage

1. C language interface, use NDK, flipping depth optimization, such as NEON optimization.
2. There is no garbage collection mechanism, you need to implement garbage collection yourself
3. Support PCM data acquisition
4. Support PCM data playback

Objects 和 Interfaces

1. The official defines a series of Interfaces for each Objects
2. First get the interface through GetInterface, and then access the function function through Interface
3. The corresponding functional interface can be obtained through the Interface ID

state mechanism

The state machine diagram in the document:

Each object has some basic methods: Realize, Resume, GetState, Destory

As shown in the figure above, the system will allocate resources correctly only after the object is realized.

Common structure

1.Engine Object and SLEngineltf (management interface, used to create other objects)

2. Media Object (multimedia abstract function)

3.Data Source and Data Sink (input source and output source)

FFmpeg decode audio data

The general processing method is: use FFmpeg to decode the audio stream of an Mp4 file, then use libswresample to convert the decoded PCM audio data into target format data, resample to ensure that the audio sampling rate is consistent with the device driver sampling rate, so that Audio plays correctly. Finally, use OpenSLES to play.

Play source code:

 //创建引擎对象
    result = slCreateEngine(&engineObject, 0, NULL, 0, NULL, NULL);
    if(result != SL_RESULT_SUCCESS)
    {
        return false;
    }

    //初始化引擎对象
    result = (*engineObject)->Realize(engineObject, SL_BOOLEAN_FALSE);
    if(result != SL_RESULT_SUCCESS)
    {
        return false;
    }

    result = (*engineObject)->GetInterface(engineObject, SL_IID_ENGINE, &engineEngine);
    if(result != SL_RESULT_SUCCESS)
    {
        return false;
    }

    //创建并初始化混音器
    SLInterfaceID ids1[1] = {SL_IID_OUTPUTMIX};
    SLboolean reqs1[1] = {SL_BOOLEAN_FALSE};
    result = (*engineEngine)->CreateOutputMix(engineEngine, &outputMixObject, 0, ids1, reqs1);
    if(result != SL_RESULT_SUCCESS)
    {
        return false;
    }

    result = (*outputMixObject)->Realize(outputMixObject, SL_BOOLEAN_FALSE);
    if(result != SL_RESULT_SUCCESS)
    {
        return false;
    }

    // 创建并初始化播放器
    SLDataLocator_AndroidSimpleBufferQueue bufferQueue = {SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE, 2};
    SLDataFormat_PCM pcmFormat = {SL_DATAFORMAT_PCM, 2, SL_SAMPLINGRATE_44_1, SL_PCMSAMPLEFORMAT_FIXED_16, SL_PCMSAMPLEFORMAT_FIXED_16,
                                  SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT, SL_BYTEORDER_LITTLEENDIAN};

    SLDataSource audioSrc = {&bufferQueue, &pcmFormat};

    SLDataLocator_OutputMix locOutputMix = {SL_DATALOCATOR_OUTPUTMIX, outputMixObject};
    SLDataSink audioSink = {&locOutputMix, NULL};

    SLInterfaceID ids2[2] = {SL_IID_BUFFERQUEUE, SL_IID_VOLUME};
    SLboolean reqs2[2] = {SL_BOOLEAN_TRUE, SL_BOOLEAN_FALSE};

    result = (*engineEngine)->CreateAudioPlayer(engineEngine, &playerObject, &audioSrc, &audioSink, 2, ids2, reqs2);
    if(result != SL_RESULT_SUCCESS)
    {
        return false;
    }

    result = (*playerObject)->Realize(playerObject, SL_BOOLEAN_FALSE);
    if(result != SL_RESULT_SUCCESS)
    {
        return false;
    }

    result = (*playerObject)->GetInterface(playerObject, SL_IID_PLAY, &playerPlay);
    if(result != SL_RESULT_SUCCESS)
    {
        return false;
    }

    result = (*playerObject)->GetInterface(playerObject, SL_IID_BUFFERQUEUE, &playerBufferQueue);
    if(result != SL_RESULT_SUCCESS)
    {
        return false;
    }

    result = (*playerBufferQueue)->RegisterCallback(playerBufferQueue, audioCallback, this);
    if(result != SL_RESULT_SUCCESS)
    {
        return false;
    }

//    result = (*playerObject)->GetInterface(playerObject, SL_IID_VOLUME, &bqPlayerVolume);
//    if(result != SL_RESULT_SUCCESS)
//    {
//        return false;
//    }

    result = (*playerPlay)->SetPlayState(playerPlay, SL_PLAYSTATE_PAUSED);
    if(result != SL_RESULT_SUCCESS)
    {
        return false;
    }

    (*playerBufferQueue)->Enqueue(playerBufferQueue, emptyBuffer, EMPTY_BUFFER_SAMPLES * 2 * sizeof(int16_t));

    return true;

RegisterCallback(playerBufferQueue, audioCallback, this);Used to register the callback function, the principle of the callback function:

void OpenSLESPlayer::processAudio() {
    //注意这个是另外一条采集线程回调，
    unique_lock<mutex> providerLock(providerMu);
    if(provider != NULL)
    {
        AudioFrame *data = provider->getAudioFrame();
        if(data == NULL)
        {
            LOGE("get a NULL audio frame");
            (*playerBufferQueue)->Enqueue(playerBufferQueue, emptyBuffer, EMPTY_BUFFER_SAMPLES * 2 * sizeof(int16_t));
        } else
        {
//            LOGD("audio frame sample count = %d, pts = %ld", data->sampleCount, data->pts);
            memcpy(emptyBuffer, data->data, data->sampleCount * 2 * sizeof(int16_t));
            // 取完数据，需要调用Enqueue触发下一次数据回调
            (*playerBufferQueue)->Enqueue(playerBufferQueue, emptyBuffer, data->sampleCount * 2 * sizeof(int16_t));
            provider->putBackUsed(data);
        }
    }
    providerLock.unlock();
}

Original link: FFmpeg Development (3): Implementation of Audio Player_comochris' Blog-CSDN Blog

★The business card at the end of the article can receive audio and video development learning materials for free, including (FFmpeg, webRTC, rtmp, hls, rtsp, ffplay, srs) and audio and video learning roadmaps, etc.

see below!