Design and implementation of libnbaio library in Android

The libnbaio ( abbreviation for Non-Blocking Audio I/O) library in Android is mainly designed for non-blocking audio I/O, but now it also contains blocking implementations of some interfaces, so its main role has become Provides a unified read and write interface for various audio I/O facilities. The libnbaio library is mainly used in the implementation of audioflinger and some audio HAL modules. The main components provided by the libnbaioPipe library are N MonoPipe, which Pipesupports a single writer, and N readers, which MonoPipesupports a single writer and a single reader.

PipeThe supported features are as follows:

Without a mutex, it is safe to use SCHED_NORMAL and SCHED_FIFO between threads.

• Write:

  • non-blocking
  • If there is not enough data, return the actual number of frames transmitted.
  • If the speed of consuming data is not enough, the data is overwritten

• Read:

  • non-blocking
  • If there is not enough data, return the actual number of frames transmitted.
  • If the reading speed cannot keep up, data will be lost

MonoPipeThe supported features are as follows:

Without a mutex, it is safe to use SCHED_NORMAL and SCHED_FIFO between threads.

• Write:

  • Whether blocking is optional
  • If configured to block, will wait until space is available before returning
  • If configured as non-blocking, the actual number of frames transmitted will be returned and the data will not be overwritten

• Read:

  • non-blocking
  • If there is not enough data, return the actual number of frames transmitted.
  • Never lose data

Here is a look at the design and implementation of the libnbaio library. The code analysis is based on the android-12.1.0_r27 version.

The main abstractions provided by the libnbaioNBAIO_Sink library are and NBAIO_Source, which are used to write audio data and read audio data respectively. The inheritance hierarchy of each part of the libnbaio library is as follows:

LibsndfileSourceand LibsndfileSinkrespectively encapsulate files opened by libsndfile in SFM_READand SFM_WRITEmode to implement NBAIO_Sourceand NBAIO_Sink. These two components, due to incompatible licenses of libsndfile , will not be included in the Android system by default. There is no place to use them in the entire Android system. With the development of the Android system, they have probably been in a state of disrepair and are unusable.

AudioStreamInSourceand AudioStreamOutSinkencapsulates Audio HAL's StreamInHalInterfaceand StreamOutHalInterfaceinterface implementations NBAIO_Sourcerespectively NBAIO_Sink. They are not multi-thread safe.

PipeUsed in PipeReaderconjunction with , they are used to write audio data and read audio data respectively. MonoPipeUsed in MonoPipeReaderconjunction with , they are used to write audio data and read audio data respectively.

SourceAudioBufferProviderEncapsulated NBAIO_Sourceand ultimately inherited from AudioBufferProvider, it is mainly used to connect those components of the libnbaio library to other audio modules of the Android system, such as mixers, etc.

MonoPipe, MonoPipeReader, Pipeand PipeReaderrealize the structure of related components as shown below:

In terms of basic structure, MonoPipeand MonoPipeReaderand Pipeand PipeReaderare basically the same, but they implement different reading and writing strategies respectively. PipeAllocate a piece of memory to audio_utils_fifomanagement for exchanging data; reading and writing data are ultimately carried out through audio_utils_fifo; Pipecontains an audio_utils_fifo_writerobject, audio_utils_fifo_writerthe object holds audio_utils_fifoa reference to the object, audio_utils_fifo_writerused to audio_utils_fifowrite audio data to; PipeReaderholds Pipea reference to the object, and contains an audio_utils_fifo_readerobject ; audio_utils_fifo_readerThe object holds a reference to the object Pipeof , used to read audio data from; and exchanges data through a shared object, which plays the role of writer and reader of audio data respectively. and have the same basic structure and data exchange methods as and . From a data flow perspective, these components have the following structure:audio_utils_fifoaudio_utils_fifoPipePipeReaderaudio_utils_fifoMonoPipeMonoPipeReaderPipePipeReader

Let's take a look at the specific implementations of MonoPipeand MonoPipeReaderand and Pipeand PipeReaderso on. PipeThe class definition (located frameworks/av/media/libnbaio/include/media/nbaio/Pipe.h) is as follows:

class Pipe : public NBAIO_Sink {

    friend class PipeReader;

public:
    // maxFrames will be rounded up to a power of 2, and all slots are available. Must be >= 2.
    // buffer is an optional parameter specifying the virtual address of the pipe buffer,
    // which must be of size roundup(maxFrames) * Format_frameSize(format) bytes.
    Pipe(size_t maxFrames, const NBAIO_Format& format, void *buffer = NULL);

    // If a buffer was specified in the constructor, it is not automatically freed by destructor.
    virtual ~Pipe();

    // NBAIO_Port interface

    //virtual ssize_t negotiate(const NBAIO_Format offers[], size_t numOffers,
    //                          NBAIO_Format counterOffers[], size_t& numCounterOffers);
    //virtual NBAIO_Format format() const;

    // NBAIO_Sink interface

    //virtual int64_t framesWritten() const;
    //virtual int64_t framesUnderrun() const;
    //virtual int64_t underruns() const;

    // The write side of a pipe permits overruns; flow control is the caller's responsibility.
    // It doesn't return +infinity because that would guarantee an overrun.
    virtual ssize_t availableToWrite() { return mMaxFrames; }

    virtual ssize_t write(const void *buffer, size_t count);
    //virtual ssize_t writeVia(writeVia_t via, size_t total, void *user, size_t block);

private:
    const size_t    mMaxFrames;     // always a power of 2
    void * const    mBuffer;
    audio_utils_fifo        mFifo;
    audio_utils_fifo_writer mFifoWriter;
    volatile int32_t mReaders;      // number of PipeReader clients currently attached to this Pipe
    const bool      mFreeBufferInDestructor;
};

PipeWhen constructing the object, you need to pass in the maximum number of frames, the audio data format, and an optional buffer for exchanging data. The maximum number of frames will be aligned upward to a power of 2. The size of the data buffer must be the maximum number of frames after upward alignment roundup(maxFrames) * frame size Format_frameSize(format) bytes. PipeWhen the object is constructed, if no allocated memory block is passed in, a piece of memory will be allocated. When Pipethe object is destroyed, the allocated memory will be released. PipeThe constructor and destructor are defined (located frameworks/av/media/libnbaio/Pipe.cpp) as follows:

Pipe::Pipe(size_t maxFrames, const NBAIO_Format& format, void *buffer) :
        NBAIO_Sink(format),
        // TODO fifo now supports non-power-of-2 buffer sizes, so could remove the roundup
        mMaxFrames(roundup(maxFrames)),
        mBuffer(buffer == NULL ? malloc(mMaxFrames * Format_frameSize(format)) : buffer),
        mFifo(mMaxFrames, Format_frameSize(format), mBuffer, false /*throttlesWriter*/),
        mFifoWriter(mFifo),
        mReaders(0),
        mFreeBufferInDestructor(buffer == NULL)
{
}

Pipe::~Pipe()
{
    ALOG_ASSERT(android_atomic_acquire_load(&mReaders) == 0);
    if (mFreeBufferInDestructor) {
        free(mBuffer);
    }
}

PipeThe thread-safe semantics of read and write operations of and PipeReaderare mainly implemented by the underlying audio_utils_fifo_reader, audio_utils_fifo_writerand audio_utils_fifo. PipeThe write()function implementation (located frameworks/av/media/libnbaio/Pipe.cpp) is as follows:

ssize_t Pipe::write(const void *buffer, size_t count)
{
    // count == 0 is unlikely and not worth checking for
    if (CC_UNLIKELY(!mNegotiated)) {
        return NEGOTIATE;
    }
    ssize_t actual = mFifoWriter.write(buffer, count);
    ALOG_ASSERT(actual <= count);
    if (actual <= 0) {
        return actual;
    }
    mFramesWritten += (size_t) actual;
    return actual;
}

The definition of the class used Pipeto read data from the write buffer is as follows:PipeReaderframeworks/av/media/libnbaio/include/media/nbaio/PipeReader.h

class PipeReader : public NBAIO_Source {

public:

    // Construct a PipeReader and associate it with a Pipe
    // FIXME make this constructor a factory method of Pipe.
    PipeReader(Pipe& pipe);
    virtual ~PipeReader();

    // NBAIO_Port interface

    //virtual ssize_t negotiate(const NBAIO_Format offers[], size_t numOffers,
    //                          NBAIO_Format counterOffers[], size_t& numCounterOffers);
    //virtual NBAIO_Format format() const;

    // NBAIO_Source interface

    //virtual size_t framesRead() const;
    virtual int64_t framesOverrun() { return mFramesOverrun; }
    virtual int64_t overruns()  { return mOverruns; }

    virtual ssize_t availableToRead();

    virtual ssize_t read(void *buffer, size_t count);

    virtual ssize_t flush();

    // NBAIO_Source end

#if 0   // until necessary
    Pipe& pipe() const { return mPipe; }
#endif

private:
    Pipe&       mPipe;
    audio_utils_fifo_reader mFifoReader;
    int64_t     mFramesOverrun;
    int64_t     mOverruns;
};

PipeReaderOnly safe for a single thread. PipeReaderDifferent threads can read the same data through different objects Pipe, but different threads cannot read data through the same PipeReaderobject. PipeReaderEach member function definition (located frameworks/av/media/libnbaio/PipeReader.cpp) is as follows:

PipeReader::PipeReader(Pipe& pipe) :
        NBAIO_Source(pipe.mFormat),
        mPipe(pipe), mFifoReader(mPipe.mFifo, false /*throttlesWriter*/, false /*flush*/),
        mFramesOverrun(0),
        mOverruns(0)
{
    android_atomic_inc(&pipe.mReaders);
}

PipeReader::~PipeReader()
{
#if !LOG_NDEBUG
    int32_t readers =
#else
    (void)
#endif
            android_atomic_dec(&mPipe.mReaders);
    ALOG_ASSERT(readers > 0);
}

ssize_t PipeReader::availableToRead()
{
    if (CC_UNLIKELY(!mNegotiated)) {
        return NEGOTIATE;
    }
    size_t lost;
    ssize_t avail = mFifoReader.available(&lost);
    if (avail == -EOVERFLOW || lost > 0) {
        mFramesOverrun += lost;
        ++mOverruns;
        avail = OVERRUN;
    }
    return avail;
}

ssize_t PipeReader::read(void *buffer, size_t count)
{
    size_t lost;
    ssize_t actual = mFifoReader.read(buffer, count, NULL /*timeout*/, &lost);
    ALOG_ASSERT(actual <= count);
    if (actual == -EOVERFLOW || lost > 0) {
        mFramesOverrun += lost;
        ++mOverruns;
        actual = OVERRUN;
    }
    if (actual <= 0) {
        return actual;
    }
    mFramesRead += (size_t) actual;
    return actual;
}

ssize_t PipeReader::flush()
{
    if (CC_UNLIKELY(!mNegotiated)) {
        return NEGOTIATE;
    }
    size_t lost;
    ssize_t flushed = mFifoReader.flush(&lost);
    if (flushed == -EOVERFLOW || lost > 0) {
        mFramesOverrun += lost;
        ++mOverruns;
        flushed = OVERRUN;
    }
    if (flushed <= 0) {
        return flushed;
    }
    mFramesRead += (size_t) flushed;  // we consider flushed frames as read, but not lost frames
    return flushed;
}

PipeReaderWhen the read operation finds that the written data overflows, even if part of the data is read, an error will be returned. It obtains the amount of readable data and the flush()operation has similar semantics.

MonoPipeThe audio data writing semantics of write()are implemented in its operation, which function is defined (located frameworks/av/media/libnbaio/MonoPipe.cpp) as follows:

ssize_t MonoPipe::write(const void *buffer, size_t count)
{
    if (CC_UNLIKELY(!mNegotiated)) {
        return NEGOTIATE;
    }
    size_t totalFramesWritten = 0;
    while (count > 0) {
        ssize_t actual = mFifoWriter.write(buffer, count);
        ALOG_ASSERT(actual <= count);
        if (actual < 0) {
            if (totalFramesWritten == 0) {
                return actual;
            }
            break;
        }
        size_t written = (size_t) actual;
        totalFramesWritten += written;
        if (!mWriteCanBlock || mIsShutdown) {
            break;
        }
        count -= written;
        buffer = (char *) buffer + (written * mFrameSize);
        // TODO Replace this whole section by audio_util_fifo's setpoint feature.
        // Simulate blocking I/O by sleeping at different rates, depending on a throttle.
        // The throttle tries to keep the mean pipe depth near the setpoint, with a slight jitter.
        uint32_t ns;
        if (written > 0) {
            ssize_t avail = mFifoWriter.available();
            ALOG_ASSERT(avail <= mMaxFrames);
            if (avail < 0) {
                // don't return avail as status, because totalFramesWritten > 0
                break;
            }
            size_t filled = mMaxFrames - (size_t) avail;
            // FIXME cache these values to avoid re-computation
            if (filled <= mSetpoint / 2) {
                // pipe is (nearly) empty, fill quickly
                ns = written * ( 500000000 / Format_sampleRate(mFormat));
            } else if (filled <= (mSetpoint * 3) / 4) {
                // pipe is below setpoint, fill at slightly faster rate
                ns = written * ( 750000000 / Format_sampleRate(mFormat));
            } else if (filled <= (mSetpoint * 5) / 4) {
                // pipe is at setpoint, fill at nominal rate
                ns = written * (1000000000 / Format_sampleRate(mFormat));
            } else if (filled <= (mSetpoint * 3) / 2) {
                // pipe is above setpoint, fill at slightly slower rate
                ns = written * (1150000000 / Format_sampleRate(mFormat));
            } else if (filled <= (mSetpoint * 7) / 4) {
                // pipe is overflowing, fill slowly
                ns = written * (1350000000 / Format_sampleRate(mFormat));
            } else {
                // pipe is severely overflowing
                ns = written * (1750000000 / Format_sampleRate(mFormat));
            }
        } else {
            ns = count * (1350000000 / Format_sampleRate(mFormat));
        }
        if (ns > 999999999) {
            ns = 999999999;
        }
        struct timespec nowTs;
        bool nowTsValid = !clock_gettime(CLOCK_MONOTONIC, &nowTs);
        // deduct the elapsed time since previous write() completed
        if (nowTsValid && mWriteTsValid) {
            time_t sec = nowTs.tv_sec - mWriteTs.tv_sec;
            long nsec = nowTs.tv_nsec - mWriteTs.tv_nsec;
            ALOGE_IF(sec < 0 || (sec == 0 && nsec < 0),
                    "clock_gettime(CLOCK_MONOTONIC) failed: was %ld.%09ld but now %ld.%09ld",
                    mWriteTs.tv_sec, mWriteTs.tv_nsec, nowTs.tv_sec, nowTs.tv_nsec);
            if (nsec < 0) {
                --sec;
                nsec += 1000000000;
            }
            if (sec == 0) {
                if ((long) ns > nsec) {
                    ns -= nsec;
                } else {
                    ns = 0;
                }
            }
        }
        if (ns > 0) {
            const struct timespec req = {0, static_cast<long>(ns)};
            nanosleep(&req, NULL);
        }
        // record the time that this write() completed
        if (nowTsValid) {
            mWriteTs = nowTs;
            if ((mWriteTs.tv_nsec += ns) >= 1000000000) {
                mWriteTs.tv_nsec -= 1000000000;
                ++mWriteTs.tv_sec;
            }
        }
        mWriteTsValid = nowTsValid;
    }
    mFramesWritten += totalFramesWritten;
    return totalFramesWritten;
}

If it is written synchronously and there is not enough space to write all the data, the sleep time will be calculated based on the water level of the data that has been written in the current buffer, and it will sleep and wait. Otherwise, the number of audio frames actually written will be returned. MonoPipeReaderThe read operation does not care about the overflow of written data. The relevant operations are implemented (located frameworks/av/media/libnbaio/MonoPipeReader.cpp) as follows:

ssize_t MonoPipeReader::availableToRead()
{
    if (CC_UNLIKELY(!mNegotiated)) {
        return NEGOTIATE;
    }
    ssize_t ret = mFifoReader.available();
    ALOG_ASSERT(ret <= mPipe->mMaxFrames);
    return ret;
}

ssize_t MonoPipeReader::read(void *buffer, size_t count)
{
    // count == 0 is unlikely and not worth checking for explicitly; will be handled automatically
    ssize_t actual = mFifoReader.read(buffer, count);
    ALOG_ASSERT(actual <= count);
    if (CC_UNLIKELY(actual <= 0)) {
        return actual;
    }
    mFramesRead += (size_t) actual;
    return actual;
}

AudioStreamInSourceIs StreamInHalInterfacethe encapsulation of , it reads data through the latter, and the data format read will StreamInHalInterfacebe in accordance with the data format obtained from . AudioStreamInSourceThe definitions (located) of each member function frameworks/av/media/libnbaio/AudioStreamInSource.cppare as follows:

AudioStreamInSource::AudioStreamInSource(sp<StreamInHalInterface> stream) :
        NBAIO_Source(),
        mStream(stream),
        mStreamBufferSizeBytes(0),
        mFramesOverrun(0),
        mOverruns(0)
{
    ALOG_ASSERT(stream != 0);
}

AudioStreamInSource::~AudioStreamInSource()
{
    mStream.clear();
}

ssize_t AudioStreamInSource::negotiate(const NBAIO_Format offers[], size_t numOffers,
                                      NBAIO_Format counterOffers[], size_t& numCounterOffers)
{
    if (!Format_isValid(mFormat)) {
        status_t result;
        result = mStream->getBufferSize(&mStreamBufferSizeBytes);
        if (result != OK) return result;
        audio_config_base_t config = AUDIO_CONFIG_BASE_INITIALIZER;
        result = mStream->getAudioProperties(&config);
        if (result != OK) return result;
        mFormat = Format_from_SR_C(config.sample_rate,
                audio_channel_count_from_in_mask(config.channel_mask), config.format);
        mFrameSize = Format_frameSize(mFormat);
    }
    return NBAIO_Source::negotiate(offers, numOffers, counterOffers, numCounterOffers);
}

int64_t AudioStreamInSource::framesOverrun()
{
    uint32_t framesOverrun;
    status_t result = mStream->getInputFramesLost(&framesOverrun);
    if (result == OK && framesOverrun > 0) {
        mFramesOverrun += framesOverrun;
        // FIXME only increment for contiguous ranges
        ++mOverruns;
    } else if (result != OK) {
        ALOGE("Error when retrieving lost frames count from HAL: %d", result);
    }
    return mFramesOverrun;
}

ssize_t AudioStreamInSource::read(void *buffer, size_t count)
{
    if (CC_UNLIKELY(!Format_isValid(mFormat))) {
        return NEGOTIATE;
    }
    size_t bytesRead;
    status_t result = mStream->read(buffer, count * mFrameSize, &bytesRead);
    if (result == OK && bytesRead > 0) {
        size_t framesRead = bytesRead / mFrameSize;
        mFramesRead += framesRead;
        return framesRead;
    } else {
        ALOGE_IF(result != OK, "Error while reading data from HAL: %d", result);
        return bytesRead;
    }
}

AudioStreamInSourceProvides an interface to obtain the number of overflow data from Audio HAL. AudioStreamInSourceThe amount of writable data is always the size of the entire buffer of the Audio HAL stream.

AudioStreamOutSinkIs StreamOutHalInterfacethe encapsulation of , it writes data through the latter, and the data format written will StreamOutHalInterfacebe in accordance with the data format obtained from . AudioStreamOutSinkThe definitions (located) of each member function frameworks/av/media/libnbaio/AudioStreamOutSink.cppare as follows:

AudioStreamOutSink::AudioStreamOutSink(sp<StreamOutHalInterface> stream) :
        NBAIO_Sink(),
        mStream(stream),
        mStreamBufferSizeBytes(0)
{
    ALOG_ASSERT(stream != 0);
}

AudioStreamOutSink::~AudioStreamOutSink()
{
    mStream.clear();
}

ssize_t AudioStreamOutSink::negotiate(const NBAIO_Format offers[], size_t numOffers,
                                      NBAIO_Format counterOffers[], size_t& numCounterOffers)
{
    if (!Format_isValid(mFormat)) {
        status_t result;
        result = mStream->getBufferSize(&mStreamBufferSizeBytes);
        if (result != OK) return result;
        audio_config_base_t config = AUDIO_CONFIG_BASE_INITIALIZER;
        result = mStream->getAudioProperties(&config);
        if (result != OK) return result;
        mFormat = Format_from_SR_C(config.sample_rate,
                audio_channel_count_from_out_mask(config.channel_mask), config.format);
        mFrameSize = Format_frameSize(mFormat);
    }
    return NBAIO_Sink::negotiate(offers, numOffers, counterOffers, numCounterOffers);
}

ssize_t AudioStreamOutSink::write(const void *buffer, size_t count)
{
    if (!mNegotiated) {
        return NEGOTIATE;
    }
    ALOG_ASSERT(Format_isValid(mFormat));
    size_t written;
    status_t ret = mStream->write(buffer, count * mFrameSize, &written);
    if (ret == OK && written > 0) {
        written /= mFrameSize;
        mFramesWritten += written;
        return written;
    } else {
        // FIXME verify HAL implementations are returning the correct error codes e.g. WOULD_BLOCK
        ALOGE_IF(ret != OK, "Error while writing data to HAL: %d", ret);
        return ret;
    }
}

status_t AudioStreamOutSink::getTimestamp(ExtendedTimestamp &timestamp)
{
    uint64_t position64;
    struct timespec time;
    if (mStream->getPresentationPosition(&position64, &time) != OK) {
        return INVALID_OPERATION;
    }
    timestamp.mPosition[ExtendedTimestamp::LOCATION_KERNEL] = position64;
    timestamp.mTimeNs[ExtendedTimestamp::LOCATION_KERNEL] = audio_utils_ns_from_timespec(&time);
    return OK;
}

AudioStreamInSourceand AudioStreamOutSinkprovides negotiate()operations, which are mainly used to complete some initialization actions and negotiate with the user the specific audio data format used.

Done.