This series of articles Master link: Thematic sub-directory Android Framework Class Audio Subsystem
Summary and description of key points in this chapter:
This chapter mainly focuses on ➕ AudioRecord process analysis and multi-app simultaneous recording scheme in the upper left recording part of the mind map above. It mainly explains the AudioRecord process analysis and framework, as well as the multi-APP simultaneous recording solution.
1 AudioRecord test program core source code analysis
The core logic of the AudioRecord test program in the previous chapter is organized as follows:
//step1 创建AudioRecord
pAudioRecord = new android::AudioRecord();
//step2 set操作
pAudioRecord->set( inputSource, sampleRateInHz,audioFormat, \
channelConfig,0,NULL, NULL,0,true,0);
//step3 启动对应的线程
pAudioRecord->start()
//step4 循环读取音频数据
while (!g_bQuitAudioRecordThread)
{
//读取数据
pAudioRecord->read(inBuffer, bufferSizeInBytes);
//存储操作
//...
}
//step5 停止对应线程，相对较为简单，忽略分析
pAudioRecord->stop()Here is a detailed analysis of the previous four steps: create, set operation, start operation, read operation
1.1 AudioRecord creation
The AudioRecord constructor code is implemented as follows:
AudioRecord::AudioRecord()
: mStatus(NO_INIT), mSessionId(AUDIO_SESSION_ALLOCATE),
mPreviousPriority(ANDROID_PRIORITY_NORMAL), mPreviousSchedulingGroup(SP_DEFAULT)
{
}
AudioRecord::AudioRecord(
audio_source_t inputSource,
uint32_t sampleRate,
audio_format_t format,
audio_channel_mask_t channelMask,
size_t frameCount,
callback_t cbf,
void* user,
uint32_t notificationFrames,
int sessionId,
transfer_type transferType,
audio_input_flags_t flags,
const audio_attributes_t* pAttributes)
: mStatus(NO_INIT), mSessionId(AUDIO_SESSION_ALLOCATE),
mPreviousPriority(ANDROID_PRIORITY_NORMAL),
mPreviousSchedulingGroup(SP_DEFAULT),
mProxy(NULL)
{
mStatus = set(inputSource, sampleRate, format, channelMask, frameCount, cbf, user,
notificationFrames, false /*threadCanCallJava*/, sessionId, transferType, flags,
pAttributes);
}Note that there are two types of constructors here, one is to initialize some parameters, and then call the set method to set other parameters. The other type is to directly set all the parameters, that is, directly call the set operation internally, which is very similar to AudioTrack.
1.2 pAudioRecord-> set operation
The code of set is implemented as follows:
status_t AudioRecord::set(
audio_source_t inputSource,
//...
const audio_attributes_t* pAttributes)
{
//...
if (sessionId == AUDIO_SESSION_ALLOCATE) {
mSessionId = AudioSystem::newAudioUniqueId();
} else {
mSessionId = sessionId;
}
ALOGV("set(): mSessionId %d", mSessionId);
mFlags = flags;
mCbf = cbf;
if (cbf != NULL) {
mAudioRecordThread = new AudioRecordThread(*this, threadCanCallJava);
mAudioRecordThread->run("AudioRecord", ANDROID_PRIORITY_AUDIO);
}
// create the IAudioRecord
status_t status = openRecord_l(0 /*epoch*/);
//...
return NO_ERROR;
}Here focus on analysis of openRecord_l implementation, the code is as follows:
status_t AudioRecord::openRecord_l(size_t epoch)
{
status_t status;
const sp<IAudioFlinger>& audioFlinger = AudioSystem::get_audio_flinger();
if (audioFlinger == 0) {
ALOGE("Could not get audioflinger");
return NO_INIT;
}
//...
audio_io_handle_t input;
//关键点1 getInput操作
status = AudioSystem::getInputForAttr(&mAttributes, &input, (audio_session_t)mSessionId,
mSampleRate, mFormat, mChannelMask, mFlags);
//...
sp<IMemory> iMem; // for cblk
sp<IMemory> bufferMem;
//关键点2：AudioFlinger的openRecord操作
sp<IAudioRecord> record = audioFlinger->openRecord(input,
mSampleRate, mFormat,
mChannelMask,
&temp,
&trackFlags,
tid,
&mSessionId,
¬ificationFrames,
iMem,
bufferMem,
&status);
//...
// update proxy 共享内存相关
mProxy = new AudioRecordClientProxy(cblk, buffers, mFrameCount, mFrameSize);
mProxy->setEpoch(epoch);
mProxy->setMinimum(mNotificationFramesAct);
//...
return NO_ERROR;
}
release:
AudioSystem::releaseInput(input, (audio_session_t)mSessionId);
if (status == NO_ERROR) {
status = NO_INIT;
}
return status;
}@ 1 AudioSystem's getInputForAttr code is implemented as follows:
status_t AudioSystem::getInputForAttr(const audio_attributes_t *attr,
audio_io_handle_t *input,
audio_session_t session,
uint32_t samplingRate,
audio_format_t format,
audio_channel_mask_t channelMask,
audio_input_flags_t flags)
{
const sp<IAudioPolicyService>& aps = AudioSystem::get_audio_policy_service();
if (aps == 0) return NO_INIT;
return aps->getInputForAttr(attr, input, session, samplingRate, format, channelMask, flags);
}Continue to analyze the getInputForAttr method of AudioPolicyManager, the code implementation is as follows:
status_t AudioPolicyManager::getInputForAttr(const audio_attributes_t *attr,
//...
input_type_t *inputType)
{
*input = AUDIO_IO_HANDLE_NONE;
*inputType = API_INPUT_INVALID;
audio_devices_t device;
// handle legacy remote submix case where the address was not always specified
String8 address = String8("");
bool isSoundTrigger = false;
audio_source_t inputSource = attr->source;
audio_source_t halInputSource;
AudioMix *policyMix = NULL;
if (inputSource == AUDIO_SOURCE_DEFAULT) {
inputSource = AUDIO_SOURCE_MIC;
}
halInputSource = inputSource;
if (inputSource == AUDIO_SOURCE_REMOTE_SUBMIX &&
strncmp(attr->tags, "addr=", strlen("addr=")) == 0) {
//...
} else {
//根据APP端传入的音源找到对应的device。
device = getDeviceAndMixForInputSource(inputSource, &policyMix);
//...
if (inputSource == AUDIO_SOURCE_HOTWORD) {
ssize_t index = mSoundTriggerSessions.indexOfKey(session);
if (index >= 0) {
*input = mSoundTriggerSessions.valueFor(session);
isSoundTrigger = true;
flags = (audio_input_flags_t)(flags | AUDIO_INPUT_FLAG_HW_HOTWORD);
ALOGV("SoundTrigger capture on session %d input %d", session, *input);
} else {
halInputSource = AUDIO_SOURCE_VOICE_RECOGNITION;
}
}
}
//根据device找到对应的IOProfile
sp<IOProfile> profile = getInputProfile(device, address,
samplingRate, format, channelMask,
flags);
//...
//根据profile找到对应的Module（so文件），对应一个声卡
//这里调用AudioFlinger的openInput方法
status_t status = mpClientInterface->openInput(profile->mModule->mHandle,
input,
&config,
&device,
address,
halInputSource,
flags);
//...
//根据profile创建一个inputDesc
sp<AudioInputDescriptor> inputDesc = new AudioInputDescriptor(profile);
//inputDesc初始化...
//将input索引和inputDesc绑定
addInput(*input, inputDesc);
mpClientInterface->onAudioPortListUpdate();
return NO_ERROR;
}Here continue to analyze the OpenInput method of AudioFlinger, the code implementation is as follows:
status_t AudioFlinger::openInput(audio_module_handle_t module,
//...
audio_input_flags_t flags)
{
Mutex::Autolock _l(mLock);
//...
sp<RecordThread> thread = openInput_l(module, input, config, *device, address, source, flags);
//...
return NO_INIT;
}Continue to analyze the implementation of openInput_l, the code is as follows:
sp<AudioFlinger::RecordThread> AudioFlinger::openInput_l(audio_module_handle_t module,
audio_io_handle_t *input,
audio_config_t *config,
audio_devices_t device,
const String8& address,
audio_source_t source,
audio_input_flags_t flags)
{
AudioHwDevice *inHwDev = findSuitableHwDev_l(module, device);
//...
if (*input == AUDIO_IO_HANDLE_NONE) {
*input = nextUniqueId();
}
audio_config_t halconfig = *config;
audio_hw_device_t *inHwHal = inHwDev->hwDevice();
audio_stream_in_t *inStream = NULL;
status_t status = inHwHal->open_input_stream(inHwHal, *input, device, &halconfig,
&inStream, flags, address.string(), source);
// If the input could not be opened with the requested parameters and we can handle the
// conversion internally, try to open again with the proposed parameters. The AudioFlinger can
// resample the input and do mono to stereo or stereo to mono conversions on 16 bit PCM inputs.
if (status == BAD_VALUE &&
config->format == halconfig.format && halconfig.format == AUDIO_FORMAT_PCM_16_BIT &&
(halconfig.sample_rate <= 2 * config->sample_rate) &&
(audio_channel_count_from_in_mask(halconfig.channel_mask) <= FCC_2) &&
(audio_channel_count_from_in_mask(config->channel_mask) <= FCC_2)) {
inStream = NULL;
status = inHwHal->open_input_stream(inHwHal, *input, device, &halconfig,
&inStream, flags, address.string(), source);
// FIXME log this new status; HAL should not propose any further changes
}
if (status == NO_ERROR && inStream != NULL) {
AudioStreamIn *inputStream = new AudioStreamIn(inHwDev, inStream);
//创建RecordThread录音线程
sp<RecordThread> thread = new RecordThread(this,
inputStream,
*input,
primaryOutputDevice_l(),
device
);
mRecordThreads.add(*input, thread);
return thread;
}
*input = AUDIO_IO_HANDLE_NONE;
return 0;
}Here will eventually create a RecordThread thread, binding the corresponding input index and device. @ 2 audioFlinger-> openRecord code is implemented as follows:
sp<IAudioRecord> AudioFlinger::openRecord(
audio_io_handle_t input,
//...
status_t *status)
{
sp<RecordThread::RecordTrack> recordTrack;
sp<RecordHandle> recordHandle;
sp<Client> client;
status_t lStatus;
int lSessionId;
//...
{
Mutex::Autolock _l(mLock);
RecordThread *thread = checkRecordThread_l(input);
//...
// TODO: the uid should be passed in as a parameter to openRecord
recordTrack = thread->createRecordTrack_l(client, sampleRate, format, channelMask,
frameCount, lSessionId, notificationFrames,
IPCThreadState::self()->getCallingUid(),
flags, tid, &lStatus);
//...
}
//...
cblk = recordTrack->getCblk();//获取共享内存头
buffers = recordTrack->getBuffers();//获取共享内存Buffer
// return handle to client
recordHandle = new RecordHandle(recordTrack);
Exit:
*status = lStatus;
return recordHandle;
}Continue to analyze the implementation of createRecordTrack_l, the code is as follows:
// RecordThread::createRecordTrack_l() must be called with AudioFlinger::mLock held
sp<AudioFlinger::RecordThread::RecordTrack> AudioFlinger::RecordThread::createRecordTrack_l(
const sp<AudioFlinger::Client>& client,
//...
status_t *status)
{
size_t frameCount = *pFrameCount;
sp<RecordTrack> track;
status_t lStatus;
//...
lStatus = initCheck();
//...
{ // scope for mLock
Mutex::Autolock _l(mLock);
track = new RecordTrack(this, client, sampleRate,
format, channelMask, frameCount, NULL, sessionId, uid,
*flags, TrackBase::TYPE_DEFAULT);
lStatus = track->initCheck();
//...
mTracks.add(track);
//...
}
lStatus = NO_ERROR;
Exit:
*status = lStatus;
return track;
}One of the most concerned links here is that recordTrack will be created through RecordThread, similar to playbackThread to create your own Track.
1.3 pAudioRecord-> start () operation
The code implementation of AudioRecord :: start is as follows:
status_t AudioRecord::start(AudioSystem::sync_event_t event, int triggerSession)
{
//...
// reset current position as seen by client to 0
mProxy->setEpoch(mProxy->getEpoch() - mProxy->getPosition());
mRefreshRemaining = true;
mNewPosition = mProxy->getPosition() + mUpdatePeriod;
int32_t flags = android_atomic_acquire_load(&mCblk->mFlags);
status_t status = NO_ERROR;
if (!(flags & CBLK_INVALID)) {
ALOGV("mAudioRecord->start()");
status = mAudioRecord->start(event, triggerSession);
if (status == DEAD_OBJECT) {
flags |= CBLK_INVALID;
}
}
//...
return status;
}Here we pay attention to the implementation of mAudioRecord-> start (mAudioRecord is the type of RecordHandle returned by audioFlinger-> openRecord, and RecordHandle is initialized using recordTrack). In fact, the method of AudioFlinger :: RecordThread :: RecordTrack :: start is called.
status_t AudioFlinger::RecordThread::RecordTrack::start(AudioSystem::sync_event_t event,
int triggerSession)
{
sp<ThreadBase> thread = mThread.promote();
if (thread != 0) {
RecordThread *recordThread = (RecordThread *)thread.get();
return recordThread->start(this, event, triggerSession);
} else {
return BAD_VALUE;
}
}The start method of recordThread is called here, the code is as follows:
status_t AudioFlinger::RecordThread::start(RecordThread::RecordTrack* recordTrack,
AudioSystem::sync_event_t event,
int triggerSession)
{
ALOGV("RecordThread::start event %d, triggerSession %d", event, triggerSession);
sp<ThreadBase> strongMe = this;
status_t status = NO_ERROR;
{
//...
recordTrack->mState = TrackBase::STARTING_1;
mActiveTracks.add(recordTrack);
mActiveTracksGen++;
status_t status = NO_ERROR;
if (recordTrack->isExternalTrack()) {
mLock.unlock();
status = AudioSystem::startInput(mId, (audio_session_t)recordTrack->sessionId());
mLock.lock();
// FIXME should verify that recordTrack is still in mActiveTracks
if (status != NO_ERROR) {
mActiveTracks.remove(recordTrack);
mActiveTracksGen++;
recordTrack->clearSyncStartEvent();
ALOGV("RecordThread::start error %d", status);
return status;
}
}
//...
recordTrack->mState = TrackBase::STARTING_2;
// signal thread to start
mWaitWorkCV.broadcast();//唤醒线程执行
//...
return status;
}
//...
}Here the thread will wake up later. The main content of the thread is to get data from the HAL, and then pass the data to the APP's AudioRecord through the internal RecordTrack. So here is concerned about the implementation of AudioSystem :: startInput. The code implementation of AudioSystem :: startInput is as follows:
status_t AudioSystem::startInput(audio_io_handle_t input,
audio_session_t session)
{
const sp<IAudioPolicyService>& aps = AudioSystem::get_audio_policy_service();
if (aps == 0) return PERMISSION_DENIED;
return aps->startInput(input, session);
}Continue to analyze the startInput implementation of AudioPolicyManager, the code is as follows:
status_t AudioPolicyManager::startInput(audio_io_handle_t input,
audio_session_t session)
{
ssize_t index = mInputs.indexOfKey(input);
sp<AudioInputDescriptor> inputDesc = mInputs.valueAt(index);
index = inputDesc->mSessions.indexOf(session);
//...
// virtual input devices are compatible with other input devices
if (!isVirtualInputDevice(inputDesc->mDevice)) {
// for a non-virtual input device, check if there is another (non-virtual) active input
audio_io_handle_t activeInput = getActiveInput();//获取活跃的input，有APP在录音
/*关键逻辑：
*当前正在录音 / 想启动录音的input，如果不是同一个，则返回error
*也正是因为如此，Android的APP不可以同时录音
*/
if (activeInput != 0 && activeInput != input) {
sp<AudioInputDescriptor> activeDesc = mInputs.valueFor(activeInput);
if (activeDesc->mInputSource == AUDIO_SOURCE_HOTWORD) {
stopInput(activeInput, activeDesc->mSessions.itemAt(0));
releaseInput(activeInput, activeDesc->mSessions.itemAt(0));
} else {
return INVALID_OPERATION;
}
}
}
//...
inputDesc->mRefCount++;
return NO_ERROR;
}The thread RecordThread was started during the start process. At the same time, we also found that Android APP can not record at the same time.
1.4 pAudioRecord-> read operation
AudioRecord read method, the code is implemented as follows:
ssize_t AudioRecord::read(void* buffer, size_t userSize)
{
if (mTransfer != TRANSFER_SYNC) {
return INVALID_OPERATION;
}
if (ssize_t(userSize) < 0 || (buffer == NULL && userSize != 0)) {
// sanity-check. user is most-likely passing an error code, and it would
// make the return value ambiguous (actualSize vs error).
ALOGE("AudioRecord::read(buffer=%p, size=%zu (%zu)", buffer, userSize, userSize);
return BAD_VALUE;
}
ssize_t read = 0;
Buffer audioBuffer;
while (userSize >= mFrameSize) {
audioBuffer.frameCount = userSize / mFrameSize;
status_t err = obtainBuffer(&audioBuffer, &ClientProxy::kForever);
if (err < 0) {
if (read > 0) {
break;
}
return ssize_t(err);
}
size_t bytesRead = audioBuffer.size;
memcpy(buffer, audioBuffer.i8, bytesRead);
buffer = ((char *) buffer) + bytesRead;
userSize -= bytesRead;
read += bytesRead;
releaseBuffer(&audioBuffer);
}
return read;
}
Here is also using obtainBuffer and releaseBuffer to operate shared memory, read data from audioBuffer to Buffer.
2 Brief description of the AudioRecord framework
@ 1 AudioRecord framework process
- APP creates and sets AudioRecord, specifies the sound source inputSource (for example: AUDIO_SOURCE_MIC), and also specifies the sampling rate, channel number, format and other parameters.
- AudioPolicyManager determines the recording device according to the parameters such as inputSource: device, finds the profile according to the device (produced by audio_policy.conf),
finds the module according to the profile, that is, corresponds to a sound card, and then loads the HAL file corresponding to the sound card. - AudioFlinger creates a RecordThread, and later the thread will read sound data from the above device
- Create a corresponding RecordTrack for the AudioRecord of the APP within the RecordThread. The AudioRecord of the APP and the RecordTrack inside the RecordThread transfer data through shared memory
- RecordThread gets data from HAL (call openInput () in HAL file to open an input channel), and then pass the data to AudioRecord of APP through internal RecordTrack
@ 2 Why can't Android record multiple apps at the same time?
In the native code, an AudioRecord of the APP will result in the creation of a RecordThread. There may be multiple RecordThreads on a device. Only one RecordThread can be running at any time, so there can only be one APP recording, not multiple APPs at the same time. recording
3 Multi APP simultaneous recording solution
3.1 Solution principle
The AudioRecord created by the APP will result in the creation of a RecordThread. There may be multiple RecordThreads on a device. Only one RecordThread can be running at any time, so there can only be one APP recording, not multiple APPs.
In fact, at the AudioFlinger level, multiple APPs are supported for simultaneous recording, and there can be multiple RecordTracks in a RecordThread. Only in the AudioPolicyManager (policy layer) this layer has restrictions (limit the number of threads, there can only be one), as long as we guarantee that there is only one RecordThread on a device (here also refers to the PlaybackThread method), when multiple APPs When accessing, the same thread is still accessed, just use different RecordTrack (this mechanism is perfect, can be used directly, without modifying the code), then multiple APPs can be recorded simultaneously.
3.2 Solution implementation
Modify AudioPolicyManager.cpp as follows (+ part is the newly added code):
status_t AudioPolicyManager::getInputForAttr(const audio_attributes_t *attr,
audio_io_handle_t *input,
audio_session_t session,
uint32_t samplingRate,
audio_format_t format,
audio_channel_mask_t channelMask,
audio_input_flags_t flags,
input_type_t *inputType)
{
//...
config.channel_mask = channelMask;
config.format = format;
+ /* check wether have an AudioInputDescriptor use the same profile */
+ for (size_t input_index = 0; input_index < mInputs.size(); input_index++) {
+ sp<AudioInputDescriptor> desc;
+ desc = mInputs.valueAt(input_index);
+ if (desc->mProfile == profile) {
+ desc->mOpenRefCount++; // 引用计数+1
+ desc->mSessions.add(session); // session
+ return desc->mIoHandle;
+ }
+ }
+
status_t status = mpClientInterface->openInput(profile->mModule->mHandle,
input,
&config,
&device,
address,
halInputSource,
flags);
//...
}
