I. Overview
Android's Audio Hardware Abstraction Layer (HAL) connects the higher-level audio-specific framework APIs in android.media to the underlying audio drivers and hardware. This section describes implementation notes and tips for improving performance.
The Android Audio Architecture defines how audio functionality is implemented and indicates the associated source code involved in the implementation.
application framework
The application framework contains application code that uses the android.media APIs to interact with the audio hardware. Internally, this code calls the appropriate JNI glue classes to access native code that interacts with the audio hardware.
- Source code directory: frameworks/base/media/java/android/media/
- AudioManager: Audio manager, including volume management, AudioFocus management, audio device management, mode management;
- Recording: AudioRecord, MediaRecorder;
- Play: AudioTrack, MedaiPlayer, SoundPool, ToneGenerator;
- Codec: MediaCodec, audio and video data codec interface.
JNI
The JNI code associated with android.media calls lower-level native code to access the audio hardware. JNI is located in frameworks/base/core/jni/ and frameworks/base/media/jni.
Native framework native framework
The native framework provides a native package equivalent to the android.media package, which calls the Binder IPC proxy to access the audio-specific services of the media server. Native framework code is located in frameworks/av/media/libmedia.
The native framework code is located in frameworks/av/media/libmedia or frameworks/av/media/libaudioclient (different versions, location changes).
Binder IPC
Binder IPC proxies are used to facilitate communication across process boundaries. Proxies are located in frameworks/av/media/libmedia and start with the letter "I".
Audio Server media server
The Audio system is responsible for audio data stream transmission and control functions in Android, and is also responsible for the management of audio devices. This part is used as the input/output layer of Android's Audio system. It is generally responsible for playing PCM sound output and obtaining PCM sound from the outside, as well as managing sound devices and settings (note: the decoding function is not implemented here. The decoding of audio and video in the android system is It is completed by opencore or stagefright, and the interface of the audio system is called after decoding to create and play the audio stream). Audio services existed in mediaserver before Android N (7.0), and Android N began to exist in the form of audioserver. These audio services are the actual code that interacts with the HAL implementation. The media server is located in frameworks/av/services/audioflinger and frameworks/av/services/audiopolicy.
Audio service includes AudioFlinger and AudioPolicyService:
- AudioFlinger: It is mainly responsible for the management of audio streaming devices, processing and transmission of audio streaming data, volume calculation, resampling, mixing, sound effects, etc.
- AudioPolicyService: Mainly responsible for audio policy, volume adjustment, device selection, audio channel selection, etc.
HAL
The HAL defines standard interfaces that audio services call and that you must implement for the audio hardware to function properly. The audio HAL interface is located in hardware/libhardware/include/hardware. See audio.h for details.
kernel driver
Audio drivers are used to interact with your hardware and HAL implementation. You can use Advanced Linux Sound Architecture (ALSA), Open Sound System (OSS), or a custom driver (HAL is driver-independent).
Note: If you are using ALSA, it is recommended to use external/tinyalsa for the user portion of the driver, as it has a compatible license (the standard user-mode library is GPL-licensed).
2. Evolution of audio system architecture
A good system architecture needs to reduce the coupling between the upper layer and the specific hardware as much as possible. This is not only the design purpose of the operating system, but also the audio system. The prototype framework of the audio system can be simply represented by the following figure:
In this figure, except for the Audio driver of Linux itself, the entire Android audio implementation is regarded as User. Therefore, we can think that the Audio Driver is the "isolation board" between the upper layer and the hardware. However, if the audio system is simply designed using the framework shown in the figure above, it is not a small burden to use audio functions for upper-layer applications. Obviously, the Android development team will further refine the "User" part according to its actual situation. How should it be refined? If it is for us to refine, how should we do it?
First of all, as an operating system, it must provide available APIs for application developers to call. The application developed by APP developers is called APP, and the API we provide is called Framework. What's the problem if the Framework interacts directly with the driver?
-
The first is the coupling problem, the interface and the implementation are coupled. Any changes in the hardware layer need to be adapted to the interface layer. We add a hardware adaptation layer;
-
The problem of unified management of resources, if multiple APPs call the same API to use hardware resources, how to allocate them? Adding a unified resource manager is actually the Audio Lib layer corresponding to the Android system.
After refinement, we found that the entire structure corresponds to several hierarchical structures of Android, including the application layer, framework layer, library layer, and HAL layer, as shown in the following figure:
2.1 Lib layer
Most of the classes in the framework layer are actually just the "intermediary" for the application to use the Android library file, it is just a shell. Because Android applications are written in java language, they need the most direct java interface support. If our Android system supports another language runtime, we can provide another language interface support (such as Go), which is the framework One of the meanings of layer existence. But as "intermediaries", they don't really implement specific functions, or only implement part of them, but focus on the core library to complete. For example, the above AudioTrack, AudioRecorder, MediaPlayer and MediaRecorder can find corresponding classes in the library, most of which are written in C++ language.
Let's think about this problem from another clue: the API we provide is called by the application layer, and then this API finally runs in the process of the application. If multiple applications use this function at the same time, it will conflict; allowing any process to operate hardware is also a dangerous behavior. Then the truth surfaced: we need a process with permission management and hardware interaction, and we need to call a certain hardware service to deal with my service. This is the very common C/S structure of the Android system and the main reason for the existence of Binder. The Server in the Android system is a system service, such as ServiceManager, LocationManagerService, ActivityManagerService, etc., as well as SurfaceFlinger that manages image synthesis, and the audio services AudioFlinger and AudioPolicyService we introduced today. Their code is placed in frameworks/av/services/audioflinger, and the main library generated is called libaudioflinger.
It is also mentioned here that another clue to analyze the source code is to use the process as a clue besides the module as a clue. A library does not represent a process, but a process depends on a library to run. Although some classes are implemented in the same library, it does not mean that they will be called in the same process. For example, both AudioFlinger and AudioPolicyService reside in a system process called mediaserver; AudioTrack/AudioRecorder and MediaPlayer/MediaRecorder are only part of the application process, and they communicate with other system processes such as audioflinger through binder services
2.2 HAL layer
As the name implies, the hardware abstraction layer is a layer that is independently encapsulated to adapt to different hardware. The task of the audio hardware abstraction layer is to truly associate AudioFlinger/AudioPolicyService with hardware devices, but it must provide a flexible structure to cope with changes.
From a design point of view, the hardware abstraction layer is the object that AudioFlinger directly accesses. There are two considerations here:
-
On the one hand, AudioFlinger does not directly call the underlying driver;
-
On the other hand, the modules on the upper layer of AudioFlinger (including the MediaPlayerService on the same layer as it) only need to interact with it to implement audio-related functions.
AudioFlinger and HAL are the core layers of the decoupling of the entire architecture. The differences between audio devices are smoothed out through libraries such as audio.primary in the HAL layer. No matter how the hardware changes, there is no need to modify the upper layer implementation on a large scale to ensure that the upper layer API exposed by the system to the outside world No modification is required to achieve high cohesion and low coupling. For manufacturers, the focus of customization is how to implement efficiently in this part of the library.
For example, the Audio system in the Android system used to depend on ALSA-lib, but later changed to tinyalsa. Such a change should not cause damage to the upper layer. Therefore, Audio HAL provides a unified interface to define the communication method between it and AudioFlinger/AudioPolicyService. This is the purpose of audio_hw_device, audio_stream_in and audio_stream_out, etc. Most of these Struct data types are just definitions of function pointers, which are handles . When AudioFlinger/AudioPolicyService is initialized, they will look for the most matching implementation in the system (these implementations reside in various libraries named audio.primary.*, audio.a2dp.*) to fill these "shells", It can be understood as a "polymorphic" implementation.
3. Two main audio driver architectures under the Linux platform receive
3.1 OSS(Open Sound System)
Early Linux versions used the OSS framework, which is also an audio system widely used in Unix and Unix-like systems. OSS can refer to the OSS interface itself, or it can be used to indicate the implementation of the interface. OSS was written by Hannu Savolainen at 4Front Technologies. Due to the intellectual property issues involved, the support and improvement of OSS in the later period is not very good, which is one of the reasons why the Linux kernel finally abandoned OSS.
In addition, OSS has also been questioned in some aspects, such as:
- Insufficient support for new audio features;
- Lack of support for latest kernel features and more.
Of course, as an early implementation of unified audio processing operations under Unix, OSS itself is relatively successful. It conforms to the design concept of "everything is a file", and as a system framework, it only specifies the interaction between the application program and the audio driver of the operating system, so each system can be customized and developed according to actual needs. In general, OSS uses the device nodes shown in the following table:
| device node | illustrate |
|---|---|
| /dev/dsp | Write data to this file à output to the external speaker to read data from this file à record from Microphone |
| /dev/mixer | Mixer, used to make related settings for audio equipment, such as volume adjustment |
| /dev/midi00 | First MIDI port, also midi01, midi02, etc. |
| /dev/sequencer | Used to access the synthesizer (synthesizer), often used in the generation of effects such as games |
3.2 ALSA(Advanced Linux Sound Architecture)
ALSA is a framework proposed by the Linux community to replace OSS. It is a system with completely open source code (following GNU GPL and GNU LGPL). After ALSA was officially introduced in Kernel 2.5, OSS was gradually excluded from the kernel. Of course, OSS itself is still being maintained, but it is no longer used by Kernel.
Compared with OSS, ALSA provides more and more complex API interfaces, so the development difficulty is relatively increased. To this end, ALSA provides a tool library for developers to help them better use ALSA API. According to the official documentation, ALSA has the following features:
- Efficient support for most types of audio interfaces (whether consumer or professional multi-channel sound cards)
- Highly Modular Sound Driver
- SMP and thread-safe design
- Provides alsa-lib in user space to simplify application writing
- Compatible with OSS API so that old OSS programs can run correctly in the system
ALSA is mainly composed of several parts as shown in the following table:
| Elements | Description |
|---|---|
| alsa-driver | kernel driver package |
| alsa-tools | Contains a series of utility programs |
| alsa-utils | Contains many useful applets such as alsactl: for saving device settings amixer: is a command line program for volume and other sound control alsamixer's ncurses version of accconnect and aseqview: for making MIDI connections, and for checking the list of connected ports ,aplay and arecord: Two command-line programs for playing and recording audio in various formats |
| alsa-plugins | Plug-in packages, such as jack, pulse, maemo |
| alsa-us | Simulation package for compatible OSS |
| pyalsa | alsa lib for compiling the python version |
| alsa-lib | user space function |
| alsa-firmware | Audio Firmware Support Package |
| The main file nodes of Alsa are as follows: |
- Information Interface (/proc/asound)
- Control Interface (/dev/snd/controlCX)
- Mixer Interface (/dev/snd/mixerCXDX)
- PCM Interface (/dev/snd/pcmCXDX)
- Raw MIDI Interface (/dev/snd/midiCXDX)
- Sequencer Interface (/dev/snd/seq)
- Timer Interface (/dev/snd/timer)
Android's TinyALSA is based on the transformation of Linux ALSA. Looking at the word "Tiny", we should be able to guess that this is a reduced version of ALSA. In fact, similar practices can be seen in other places in the Android system - what should I do if I want to use open source projects but think the project is too large and cumbersome? Then you can only lose weight, so many Tiny-XXX appeared.
In the early versions, the audio architecture of the Android system was mainly based on ALSA, and its upper layer implementation can be regarded as an "application" of ALSA. Later, due to some deficiencies in ALSA, later versions of Android no longer depended on the implementation of the user space layer provided by ALSA. The HAL layer ultimately relies on the alsa-lib library to interact with the driver layer.