This series of articles Master link: Thematic sub-directory Android Framework Class Audio Subsystem
Summary and description of key points in this chapter:
This chapter mainly focuses on the analysis of PlaybackThread in the upper left of the mind map above. It mainly analyzes the mixing principle and mixing process.
1 Explanation of mixing principle
When multiple applications play, each APP will create an AudioTrack, and each AudioTrack will transfer data through the track of shared memory and playback thread. The audio data format sent by each application may be different, and the sound card only supports a fixed number Format, unless the format sent is the format supported by the sound card itself, otherwise it needs to be resampled / mixed (some operations of mAudioMixer in the playback thread convert the audio format that is not supported by the hardware to the audio format supported by the hardware. This process is called heavy sampling). Here we interpret the mAudioMixer variable type AudioMixer in MixerThread. The reference code of AudioMixer is as follows:
class MixerThread : public PlaybackThread {
{
public:
MixerThread(const sp<AudioFlinger>& audioFlinger,
AudioStreamOut* output,
audio_io_handle_t id,
audio_devices_t device,
type_t type = MIXER);
virtual ~MixerThread();
//...
AudioMixer* mAudioMixer; // normal mixer
//...
};The AudioMixer class here is implemented as follows:
class AudioMixer
{
public:
AudioMixer(size_t frameCount, uint32_t sampleRate,uint32_t maxNumTracks = MAX_NUM_TRACKS);
//...
//给track使用的hook函数整理
/*进行重采样*/
static void track__genericResample(track_t* t, int32_t* out, size_t numFrames, int32_t* temp,
int32_t* aux);
/*不做任何处理*/
static void track__nop(track_t* t, int32_t* out, size_t numFrames, int32_t* temp, int32_t* aux);
static void track__16BitsStereo(track_t* t, int32_t* out, size_t numFrames, int32_t* temp,
int32_t* aux);
static void track__16BitsMono(track_t* t, int32_t* out, size_t numFrames, int32_t* temp,
int32_t* aux);
//...
// multi-format track hooks
template <int MIXTYPE, typename TO, typename TI, typename TA>
static void track__Resample(track_t* t, TO* out, size_t frameCount,
TO* temp __unused, TA* aux);
template <int MIXTYPE, typename TO, typename TI, typename TA>
static void track__NoResample(track_t* t, TO* out, size_t frameCount,
TO* temp __unused, TA* aux);
//...
//给mState使用的hook函数整理
static void process__validate(state_t* state, int64_t pts);
/* no-op,如果静音了不做任何处理*/
static void process__nop(state_t* state, int64_t pts);
/*如果传递过来的数据，是声卡直接支持的格式，则不需要重新采样*/
static void process__genericNoResampling(state_t* state, int64_t pts);
/*如果需要重新采样，则会调用该函数*/
static void process__genericResampling(state_t* state, int64_t pts);
static void process__OneTrack16BitsStereoNoResampling(state_t* state,int64_t pts);
//...
// hook types
enum {
PROCESSTYPE_NORESAMPLEONETRACK,
};
enum {
TRACKTYPE_NOP,
TRACKTYPE_RESAMPLE,
TRACKTYPE_NORESAMPLE,
TRACKTYPE_NORESAMPLEMONO,
};
//...
state_t mState __attribute__((aligned(32)));
//...
}There is a member mstate in mAudioMixer, and the state_t structure is defined as follows:
// pad to 32-bytes to fill cache line
struct state_t {
uint32_t enabledTracks;
uint32_t needsChanged;
size_t frameCount;
process_hook_t hook; // one of process__*, never NULL
int32_t *outputTemp;
int32_t *resampleTemp;
NBLog::Writer* mLog;
int32_t reserved[1];
// FIXME allocate dynamically to save some memory when maxNumTracks < MAX_NUM_TRACKS
track_t tracks[MAX_NUM_TRACKS] __attribute__((aligned(32)));
};mstate contains a hook function that points to different processing functions. For different situations, hooks will point to different functions. If the pointer of the hook function points to process__genericNoResampling, there will be a temporary cache area of outputTemp, and there are also tracks (variables in the mState member), each track corresponds to the AudioTrack of the application, and the definition of the track_t structure is as follows:
struct track_t {
uint32_t needs;
union {
int16_t volume[MAX_NUM_VOLUMES]; // U4.12 fixed point (top bit should be zero)
int32_t volumeRL;
};
int32_t prevVolume[MAX_NUM_VOLUMES];
int32_t volumeInc[MAX_NUM_VOLUMES];
int32_t auxInc;
int32_t prevAuxLevel;
int16_t auxLevel; // 0 <= auxLevel <= MAX_GAIN_INT, but signed for mul performance
uint16_t frameCount;
uint8_t channelCount; // 1 or 2, redundant with (needs & NEEDS_CHANNEL_COUNT__MASK)
uint8_t unused_padding; // formerly format, was always 16
uint16_t enabled; // actually bool
audio_channel_mask_t channelMask;
AudioBufferProvider* bufferProvider;
// 16-byte boundary
mutable AudioBufferProvider::Buffer buffer; // 8 bytes
hook_t hook; //指向不同的处理函数
const void* in; // current location in buffer
// 16-byte boundary
AudioResampler* resampler; //重采样器
uint32_t sampleRate;
int32_t* mainBuffer;//保存重采样之后的最终数据
int32_t* auxBuffer;
//...
bool needsRamp() { return (volumeInc[0] | volumeInc[1] | auxInc) != 0; }
bool setResampler(uint32_t trackSampleRate, uint32_t devSampleRate);
bool doesResample() const { return resampler != NULL; }
void resetResampler() { if (resampler != NULL) resampler->reset(); }
void adjustVolumeRamp(bool aux, bool useFloat = false);
size_t getUnreleasedFrames() const { return resampler != NULL ?
resampler->getUnreleasedFrames() : 0; };
};
Through the above understanding of AudioMixer, state_t, track_t and other types, we can see two kinds of hooks:
- hook in state_t mState: total hook
- hook in track_t track: single hook
AudioTrack transfers data to MixerThread (PlaybackThread subclass) through shared memory, analyzes the format of mTracks in MixerThread to see if it is a format supported by the hardware, sets different hook functions (determine whether to resample) according to different situations, and the setting is completed After that, set the total hook function (in mState), summarized as follows:
- Determine the hook: analyze the data of mState.tracks [x] one by one, determine the tracks [x] .hook according to its format, and then determine the total mState.hook
- Call hook: Just call the total mState.hook, it will call each mState.tracks [x] .hook
The destination of the data after resampling: there is original data in the shared memory. As mentioned earlier in outputTemp, the processed data of each track enters the temporary buffer area (such as resampling). So outputTemp stores the data superimposed after each track is processed. These temporary data are finally saved in mainBuffer. The mainBuffer in each track points to the mixed mMixerBuffer in PlaybackThread. The data in mMixerBuffer can be played, but it is not sent directly to the hardware, but to mSinkbuffer. Of course, mEffectBuffer can also be used to process sound effects in the android system. They will finally send the buffer to the hardware mSinkbuffer. If the sound effect is used, the source may be mEffectBuffer and mMixerBuffer.
2 Analysis of mixing source code
Here, the analysis is mainly based on MixerThread, starting from the playback thread of the parent class of MixerThread, reviewing the source code of AudioPloicyService startup and AudioFlinger startup, and the creation process analysis stack of playbacktrhead is as follows:
AudioPolicyService::onFirstRef(...)
->AudioPolicyService::createAudioPolicyManager(...);
-->AudioPolicyManager(...);
--->AudioFlinger::loadHwModule(...);
--->AudioFlinger::openOutput(...);
---->AudioFlinger::openOutput_l(...);
----->创建thread(MixerThread(多数)、OffloadThread、DirectOutputThread中的一个)
----->mPlaybackThreads.add(*output, thread);将thread加入到mPlaybackThreads中That is, from the beginning of the creation of AudioPolicyService, threads corresponding to output (one of MixerThread (most), OffloadThread, and DirectOutputThread) are created. Therefore, from the analysis of the playbackthread thread, the constructor code is as follows:
AudioFlinger::PlaybackThread::PlaybackThread(const sp<AudioFlinger>& audioFlinger,
//...
mLatchDValid(false), mLatchQValid(false)
{
//设置mName
snprintf(mName, kNameLength, "AudioOut_%X", id);
//...
}
The code implementation in onFirstRef is as follows:
void AudioFlinger::PlaybackThread::onFirstRef()
{
run(mName, ANDROID_PRIORITY_URGENT_AUDIO);
}After creating the playbackthread class (as well as the subclass MixerThread, etc.), the thread starts here. In Android thread, threadloop is the real execution body of the thread, and the code is implemented as follows:
bool AudioFlinger::PlaybackThread::threadLoop()
{
//...
while (!exitPending())
{
cpuStats.sample(myName);
Vector< sp<EffectChain> > effectChains;
{ // scope for mLock
Mutex::Autolock _l(mLock);
/*处理配置信息*/
processConfigEvents_l();
//...
//如果数据为0，则让声卡进入休眠状态
if ((!mActiveTracks.size() && systemTime() > standbyTime) ||
isSuspended()) {
// put audio hardware into standby after short delay
if (shouldStandby_l()) {
//声卡休眠
threadLoop_standby();
mStandby = true;
}
if (!mActiveTracks.size() && mConfigEvents.isEmpty()) {
IPCThreadState::self()->flushCommands();
//...
//线程休眠点，直到AudioTrack发送广播唤醒
mWaitWorkCV.wait(mLock);
//...
continue;
}
}
//关键点1：混音前的准备工作
mMixerStatus = prepareTracks_l(&tracksToRemove);
//...
lockEffectChains_l(effectChains);
} // mLock scope ends
if (mBytesRemaining == 0) {
mCurrentWriteLength = 0;
if (mMixerStatus == MIXER_TRACKS_READY) {
//关键点2：混音
threadLoop_mix();
} else if ((mMixerStatus != MIXER_DRAIN_TRACK)
&& (mMixerStatus != MIXER_DRAIN_ALL)) {
threadLoop_sleepTime();
if (sleepTime == 0) {
mCurrentWriteLength = mSinkBufferSize;
}
}
if (mMixerBufferValid) {
void *buffer = mEffectBufferValid ? mEffectBuffer : mSinkBuffer;
audio_format_t format = mEffectBufferValid ? mEffectBufferFormat : mFormat;
//把数据从thread.mMixerBuffer复制到thread.mSinkBuffer
memcpy_by_audio_format(buffer, format, mMixerBuffer, mMixerBufferFormat,
mNormalFrameCount * mChannelCount);
}
//...
}
//...
if (mEffectBufferValid) {
//把数据从thread.mEffectBuffer复制到thread.mSinkBuffer
memcpy_by_audio_format(mSinkBuffer, mFormat, mEffectBuffer, mEffectBufferFormat,
mNormalFrameCount * mChannelCount);
}
// enable changes in effect chain
unlockEffectChains(effectChains);
if (!waitingAsyncCallback()) {
// sleepTime == 0 means we must write to audio hardware
if (sleepTime == 0) {
if (mBytesRemaining) {
//关键点3：音频输出
ssize_t ret = threadLoop_write();
//...
} else if ((mMixerStatus == MIXER_DRAIN_TRACK) ||
(mMixerStatus == MIXER_DRAIN_ALL)) {
threadLoop_drain();
}
//...
} else {
usleep(sleepTime);
}
}
//...
}
threadLoop_exit();
if (!mStandby) {
threadLoop_standby();
mStandby = true;
}
//...
return false;
}
playbackthread is responsible for creating threads, but the key methods to be analyzed here are all implemented in MixerThread. The three key methods analyzed are prepareTracks_l, threadLoop_mix, and threadLoop_write.
2.1 analysis of prepareTracks_l
The code implementation of MixerThread :: prepareTracks_l is as follows:
AudioFlinger::PlaybackThread::mixer_state AudioFlinger::MixerThread::prepareTracks_l(
Vector< sp<Track> > *tracksToRemove)
{
//默认为空闲状态
mixer_state mixerStatus = MIXER_IDLE;
size_t count = mActiveTracks.size();
//...
//对于所有在mActiveTracks里面的Track，都需要进行设置
for (size_t i=0 ; i<count ; i++) {
const sp<Track> t = mActiveTracks[i].promote();
if (t == 0) {
continue;
}
// this const just means the local variable doesn't change
Track* const track = t.get();
// fastTrack不会在这里进行混音,略过
if (track->isFastTrack()) {
//...
}
{ // local variable scope to avoid goto warning
audio_track_cblk_t* cblk = track->cblk();
/* 获取track的name，这是个索引，AudioMixer会最多维护32个track，分别对应int的32个位，
* 如果track的name还没定下来的话，会自行选择一个空位
*/
int name = track->name();
size_t desiredFrames;
uint32_t sr = track->sampleRate();
if (sr == mSampleRate) {
desiredFrames = mNormalFrameCount;
} else {
desiredFrames = (mNormalFrameCount * sr) / mSampleRate + 1 + 1;
desiredFrames += mAudioMixer->getUnreleasedFrames(track->name());
}
uint32_t minFrames = 1;
if ((track->sharedBuffer() == 0) && !track->isStopped() && !track->isPausing() &&
(mMixerStatusIgnoringFastTracks == MIXER_TRACKS_READY)) {
minFrames = desiredFrames;
}
//混音的状态下，frameReady = 1,那么会进入下面的条件，进行AudioMixer参数设置
size_t framesReady = track->framesReady();
if ((framesReady >= minFrames) && track->isReady() &&
!track->isPaused() && !track->isTerminated())
{
//音量参数设置
//...
//设置AudioMixer参数
mAudioMixer->setBufferProvider(name, track);//源Buffer
mAudioMixer->enable(name);//使能该track，可以混音
//音轨 左 右 aux
mAudioMixer->setParameter(name, param, AudioMixer::VOLUME0, &vlf);
mAudioMixer->setParameter(name, param, AudioMixer::VOLUME1, &vrf);
mAudioMixer->setParameter(name, param, AudioMixer::AUXLEVEL, &vaf);
//音频格式
mAudioMixer->setParameter(
name,
AudioMixer::TRACK,
AudioMixer::FORMAT, (void *)track->format());
//音轨mask，哪个需要或者不需要混音
mAudioMixer->setParameter(
name,
AudioMixer::TRACK,
AudioMixer::CHANNEL_MASK, (void *)(uintptr_t)track->channelMask());
mAudioMixer->setParameter(
name,
AudioMixer::TRACK,
AudioMixer::MIXER_CHANNEL_MASK, (void *)(uintptr_t)mChannelMask);
// limit track sample rate to 2 x output sample rate, which changes at re-configuration
uint32_t maxSampleRate = mSampleRate * AUDIO_RESAMPLER_DOWN_RATIO_MAX;
uint32_t reqSampleRate = track->mAudioTrackServerProxy->getSampleRate();
if (reqSampleRate == 0) {
reqSampleRate = mSampleRate;
} else if (reqSampleRate > maxSampleRate) {
reqSampleRate = maxSampleRate;
}
/*进行重采样
*注意：安卓的MixerThread会对所有的track进行重采样
*那么在混音的时候会调用重采样的混音方法。
*/
mAudioMixer->setParameter(
name,
AudioMixer::RESAMPLE,
AudioMixer::SAMPLE_RATE,
(void *)(uintptr_t)reqSampleRate);
if (mMixerBufferEnabled
&& (track->mainBuffer() == mSinkBuffer
|| track->mainBuffer() == mMixerBuffer)) {
mAudioMixer->setParameter(
name,
AudioMixer::TRACK,
AudioMixer::MIXER_FORMAT, (void *)mMixerBufferFormat);
//目的buffer
mAudioMixer->setParameter(
name,
AudioMixer::TRACK,
AudioMixer::MAIN_BUFFER, (void *)mMixerBuffer);
// TODO: override track->mainBuffer()?
mMixerBufferValid = true;
} else {
//...
}
//aux buffer
mAudioMixer->setParameter(
name,
AudioMixer::TRACK,
AudioMixer::AUX_BUFFER, (void *)track->auxBuffer());
// reset retry count
track->mRetryCount = kMaxTrackRetries;
if (mMixerStatusIgnoringFastTracks != MIXER_TRACKS_READY ||
mixerStatus != MIXER_TRACKS_ENABLED) {
//状态为 ready表示可以混音
mixerStatus = MIXER_TRACKS_READY;
}
} else {
//...
}
} // local variable scope to avoid goto warning
track_is_ready: ;
}
//...
//从mActiveTracks删除需要移除的track
removeTracks_l(*tracksToRemove);
//...
if (fastTracks > 0) {
//正常混音准备时，这里返回的是MIXER_TRACK_READY
mixerStatus = MIXER_TRACKS_READY;
}
return mixerStatus;
}
In the process of preparing the mix, there are mainly a few things:
- Set the parameters required for mixing, including: volume, source buffer, destination buffer, audio format, whether to resample, etc.
- Delete tracks added to tracksToRemove
- Returns the current status mMixerStatus
2.2 threadLoop_mix analysis
Only when the mMixerStatus value returned by prepareTrack_l is MIXER_TRACK_READY, can we enter threadLoop_mix for mixing. The code is implemented as follows:
void AudioFlinger::MixerThread::threadLoop_mix()
{
int64_t pts;
status_t status = INVALID_OPERATION;
//获取timestamps，即输出时间戳，用于seek到源buffer的某个位置进行混音
if (mNormalSink != 0) {
status = mNormalSink->getNextWriteTimestamp(&pts);
} else {
status = mOutputSink->getNextWriteTimestamp(&pts);
}
if (status != NO_ERROR) {
pts = AudioBufferProvider::kInvalidPTS;
}
//AudioMixer混音
mAudioMixer->process(pts);
//混音了多少音频数据
mCurrentWriteLength = mSinkBufferSize;
if ((sleepTime == 0) && (sleepTimeShift > 0)) {
sleepTimeShift--;
}
//等不需要睡眠时直接输出音频
sleepTime = 0;
//待机时间更新
standbyTime = systemTime() + standbyDelay;
}
With the parameters set by prepareTrack_l, all you need to do in threadLoop_mix is to call the AudioMixer process method to mix.
2.3 threadLoop_write分析
threadLoop_write is used for audio output after mixing, and the code is implemented as follows:
ssize_t AudioFlinger::MixerThread::threadLoop_write()
{
if (mFastMixer != 0) {
//...fastMixer处理
}
return PlaybackThread::threadLoop_write();
}Continue to analyze PlaybackThread :: threadLoop_write (), the code implementation is as follows:
// shared by MIXER and DIRECT, overridden by DUPLICATING
ssize_t AudioFlinger::PlaybackThread::threadLoop_write()
{
// FIXME rewrite to reduce number of system calls
mLastWriteTime = systemTime();
mInWrite = true;
ssize_t bytesWritten;
const size_t offset = mCurrentWriteLength - mBytesRemaining;
// If an NBAIO sink is present, use it to write the normal mixer's submix
if (mNormalSink != 0) {
//将Buffer写到声卡上
ssize_t framesWritten = mNormalSink->write((char *)mSinkBuffer + offset, count);
//...
// otherwise use the HAL / AudioStreamOut directly
} else {
//如果用fastMixer的话其实会走该分支，先忽略
// Direct output and offload threads
bytesWritten = mOutput->stream->write(mOutput->stream,
(char *)mSinkBuffer + offset, mBytesRemaining);
//...
}
//...
mNumWrites++;
mInWrite = false;
mStandby = false;
return bytesWritten;//返回输出的音频数据量
}2.4 Process summary
Organize the work done in threadloop as follows:
@1 prepareTracks_l :
- 确定enabled track, disabled track
- For enabled track, set the parameters in mState.tracks [x]
@ 2 threadLoop_mix: processing data (such as resampling), mixing
- Determine the hook: analyze the data of mState.tracks [x] one by one, determine the tracks [x] .hook according to its format, and then determine the total mState.hook
- Call hook: Just call the total mState.hook, it will call each mState.tracks [x] .hook
- The mixed data will be placed in the temporary BUFFER of mState.outputTemp
- Then convert the format and store it in thread.mMixerBuffer
@ 3 memcpy_by_audio_format: copy data from thread.mMixerBuffer or thread.mEffectBuffer to thread.mSinkBuffer
@ 4 threadLoop_write: write thread.mSinkBuffer to sound card
@ 5 threadLoop_exit
