multimedia tunnel

Multimedia Tunneling enables compressed video data to be passed directly to the display through a hardware video decoder without having to be processed by application code or Android framework code. Device-specific code underneath the Android stack determines which video frames to send to the display and when to send them by comparing the video frame rendering timestamp to one of the following types of internal clocks:

• For on-demand video playback in Android 5 or higher, AudioTrackClock & App Audio demo timestamp synchronization into

• For live playback on Android 11 or higher, the Program Reference Clock (PCR) or System Clock (STC) driven by the Tuner )

background

Legacy video playback on Android notifies the application when compressed video frames are decoded. The application then publishes the decoded video frame to the display to be rendered at the same system clock time as the corresponding audio frame RetrievehistoryAudioTimestamps instances to calculate the correct time.

Because tunneled video playback bypasses application code and reduces the number of processes acting on the video, it can provide more efficient video rendering depending on the OEM implementation. It also provides more accurate video pacing and synchronization for the selected clock (PRC, STC, or audio) by avoiding timing issues introduced by potential discrepancies between the time Android requests to render the video and the time of the real hardware vsync. However, tunneling can also reduce support for GPU effects such as blurring or rounded corners in picture-in-picture (PiP) windows because buffers are bypassed Android graphics stack.

The image below shows how tunneling simplifies the video playback process.

Figure 1.Comparison of traditional and tunnel video playback processes

For application developers

Since most application developers integrate libraries for playback implementations, in most cases the implementation simply requires reconfiguring the library for tunnel playback. For a low-level implementation of a tunneled video player, use the following instructions.

For on-demand video playback in Android 5 or higher:

1. Create anSurfaceView instance.

2. Create anaudioSessionId instance.

3. Create and instances using the audioSessionId instance created in step 2. AudioTrackMediaCodec

4. Queue audio data toAudioTrack using the rendering timestamp of the first audio frame in the audio data.

For live streaming on Android 11 or higher:

1. Create anSurfaceView instance.

2. Gets an instance fromTuner. avSyncHwId

3. Create and instances using the avSyncHwId instance created in step 2. AudioTrackMediaCodec

The API call flow is shown in the following code snippet:

aab.setContentType(AudioAttributes.CONTENT_TYPE_MOVIE);

// configure for audio clock sync
aab.setFlag(AudioAttributes.FLAG_HW_AV_SYNC);
// or, for tuner clock sync (Android 11 or higher)
new tunerConfig = TunerConfiguration(0, avSyncId);
aab.setTunerConfiguration(tunerConfig);

if (codecName == null) {
  return FAILURE;
}

// configure for audio clock sync
mf.setInteger(MediaFormat.KEY_AUDIO_SESSION_ID, audioSessionId);
// or, for tuner clock sync (Android 11 or higher)
mf.setInteger(MediaFormat.KEY_HARDWARE_AV_SYNC_ID, avSyncId);

On-demand video playback behavior

Because tunneled on-demand video playback is implicitly bound toAudioTrackplayback, the behavior of tunneled video playback may depend on the behavior of audio playback.

• On most devices, by default, video frames are not rendered until audio playback begins. However, an application may need to render video frames before starting audio playback, for example, to display the current video position to the user when searching.

  • To indicate that the first queued video frame should be rendered immediately after decoding, set the PARAMETER_KEY_TUNNEL_PEEK parameter to < a i=3>. When compressed video frames are reordered in the queue (e.g. when there are B frames), this means that the first displayed video frame should always be I frame. 1

  • If you do not want the first queued video frame to be rendered before audio playback begins, set this parameter to0 .

  • If this parameter is not set, the OEM determines the device's behavior.

• When audio data is not provided toAudioTrack and the buffer is empty (audio underrun), video playback stops until more audio data is written because the audio clock is no longer go ahead.

• During playback, discontinuities that cannot be corrected by the application may appear in the audio presentation timestamps. When this occurs, the OEM corrects negative gaps by stopping the current video frame, and corrects positive gaps by discarding video frames or inserting silent audio frames (depending on OEM implementation). For inserted silent audio frames, the AudioTimestamp frame position will not increase.

For device manufacturers

Configuration

OEMs should create a separate video decoder to support tunneled video playback. This decoder should advertise its ability to tunnel in themedia_codecs.xmlfile:

<Feature name="tunneled-playback" required="true"/>

When a tunnelMediaCodec instance is configured with an audio session ID, it queries AudioFlinger for thisHW_AV_SYNC ID:< /span>

if (entry.getKey().equals(MediaFormat.KEY_AUDIO_SESSION_ID)) {
    int sessionId = 0;
    try {
        sessionId = (Integer)entry.getValue();
    }
    catch (Exception e) {
        throw new IllegalArgumentException("Wrong Session ID Parameter!");
    }
    keys[i] = "audio-hw-sync";
    values[i] = AudioSystem.getAudioHwSyncForSession(sessionId);
}

During this query, AudioFlinger retrieves the HW_AV_SYNC ID from the primary audio device and internally Associated with audio session ID:

audio_hw_device_t *dev = mPrimaryHardwareDev->hwDevice();
char *reply = dev->get_parameters(dev, AUDIO_PARAMETER_HW_AV_SYNC);
AudioParameter param = AudioParameter(String8(reply));
int hwAVSyncId;
param.getInt(String8(AUDIO_PARAMETER_HW_AV_SYNC), hwAVSyncId);

If an AudioTrack instance has been created, the HW_AV_SYNC ID will be passed to the output stream with the same audio session ID. If it has not been created, the ID is passed to the output stream during creation. This is done by the player thread:AudioTrackHW_AV_SYNC

mOutput->stream->common.set_parameters(&mOutput->stream->common, AUDIO_PARAMETER_STREAM_HW_AV_SYNC, hwAVSyncId);

HW_AV_SYNC The ID, whether corresponding to the audio output stream or the Tuner configuration, is passed into the OMX or Codec2 component so that OEM code can associate the codec with the corresponding audio output stream Or Tuner stream associated.

During component configuration, the OMX or Codec2 component should return a sideband handle that can be used to associate the codec with the Hardware Composer (HWC) layer. When the application associates a surface with MediaCodec, this sideband handle is passed down to the HWC via SurfaceFlinger, which configures the layer as Sidebandlayer.

err = native_window_set_sideband_stream(nativeWindow.get(), sidebandHandle);
if (err != OK) {
  ALOGE("native_window_set_sideband_stream(%p) failed! (err %d).", sidebandHandle, err);
  return err;
}

The HWC is responsible for receiving new image buffers from the codec output at the appropriate time, synchronizing to the relevant audio output stream or tuner program reference clock, compositing the buffer with the current contents of other layers, and displaying the resulting image. This is independent of the normal preparation and setup cycle. Prepare and setup are only called when other layers change or when properties of the sideband layer change (such as position or size).

OMX

The tunnel decoder component should support the following:

• SetOMX.google.android.index.configureVideoTunnelMode extended parameter, which uses the ConfigureVideoTunnelModeParams structure to pass in the HW_AV_SYNC ID associated with the audio output device.

• Configuration OMX_IndexConfigAndroidTunnelPeek parameter that tells the codec to render or not render the first decoded video frame regardless of whether audio playback has started.

• SendOMX_EventOnFirstTunnelFrameReady event when the first tunnel video frame has been decoded and is ready for rendering.

AOSP is implemented throughOMXNodeInstance and the tunnel is configured inACodec mode, as shown in the following code snippet:

OMX_INDEXTYPE index;
OMX_STRING name = const_cast<OMX_STRING>(
        "OMX.google.android.index.configureVideoTunnelMode");

OMX_ERRORTYPE err = OMX_GetExtensionIndex(mHandle, name, &index);

ConfigureVideoTunnelModeParams tunnelParams;
InitOMXParams(&tunnelParams);
tunnelParams.nPortIndex = portIndex;
tunnelParams.bTunneled = tunneled;
tunnelParams.nAudioHwSync = audioHwSync;
err = OMX_SetParameter(mHandle, index, &tunnelParams);
err = OMX_GetParameter(mHandle, index, &tunnelParams);
sidebandHandle = (native_handle_t*)tunnelParams.pSidebandWindow;

If a component supports this configuration, it should allocate a sideband handle to this codec and pass it back through the pSidebandWindow member so that the HWC can identify the associated codec device. If the component does not support this configuration, bTunneled should be set to OMX_FALSE .

codec2

In Android 11 or higher, Codec2 supports tunnel playback. The decoder component should support the following:

• ConfigurationC2PortTunneledModeTuning , which configures tunnel mode and passes in HW_AV_SYNC retrieved from the audio output device or tuner configuration.

• QueryC2_PARAMKEY_OUTPUT_TUNNEL_HANDLE , allocate and retrieve sideband handles for HWC.

• is processed when connecting to C2_PARAMKEY_TUNNEL_HOLD_RENDER C2Work , which instructs the codec to decode and signal the job is complete, but not before 1) the codec is instructed to render it or 2) don't render the output buffer before audio playback starts.

• handlingC2_PARAMKEY_TUNNEL_START_RENDER , which instructs the codec to render frames marked C2_PARAMKEY_TUNNEL_HOLD_RENDER immediately, even if audio playback has not yet started.

• Reserveddebug.stagefright.ccodec_delayed_paramsNot configured (recommended). If you do configure it, set it to false .

  Note:Do not set debug.stagefright.ccodec_delayed_params to true as this will cause delays in < a i=4> sent to codec. C2_PARAMKEY_TUNNEL_START_RENDER

AOSP is implemented throughC2PortTunnelModeTuning configuring tunnel mode inCCodec, as shown in the following code snippet:CCodec a>

if (msg->findInt32("audio-hw-sync", &tunneledPlayback->m.syncId[0])) {
    tunneledPlayback->m.syncType =
            C2PortTunneledModeTuning::Struct::sync_type_t::AUDIO_HW_SYNC;
} else if (msg->findInt32("hw-av-sync-id", &tunneledPlayback->m.syncId[0])) {
    tunneledPlayback->m.syncType =
            C2PortTunneledModeTuning::Struct::sync_type_t::HW_AV_SYNC;
} else {
    tunneledPlayback->m.syncType =
            C2PortTunneledModeTuning::Struct::sync_type_t::REALTIME;
    tunneledPlayback->setFlexCount(0);
}
c2_status_t c2err = comp->config({ tunneledPlayback.get() }, C2_MAY_BLOCK,
        failures);
std::vector<std::unique_ptr<C2Param>> params;
c2err = comp->query({}, {C2PortTunnelHandleTuning::output::PARAM_TYPE},
        C2_DONT_BLOCK, &params);
if (c2err == C2_OK && params.size() == 1u) {
    C2PortTunnelHandleTuning::output *videoTunnelSideband =
            C2PortTunnelHandleTuning::output::From(params[0].get());
    return OK;
}

If a component supports this configuration, it should allocate a sideband handle for this codec and pass it back via C2PortTunnelHandlingTuning so that the HWC can identify the associated codec .

Audio HAL

For on-demand video playback, the audio HAL receives the audio presentation timestamp inline with the audio data in big-endian format, within a header at the beginning of each chunk of audio data written by the application:

struct TunnelModeSyncHeader {
  // The 32-bit data to identify the sync header (0x55550002)
  int32 syncWord;
  // The size of the audio data following the sync header before the next sync
  // header might be found.
  int32 sizeInBytes;
  // The presentation timestamp of the first audio sample following the sync
  // header.
  int64 presentationTimestamp;
  // The number of bytes to skip after the beginning of the sync header to find the
  // first audio sample (20 bytes for compressed audio, or larger for PCM, aligned
  // to the channel count and sample size).
  int32 offset;
}

In order for the HWC to render video frames in sync with the corresponding audio frames, the audio HAL should parse the synchronization header and use the presentation timestamp to resynchronize the playback clock with the audio rendering. To resynchronize when playing compressed audio, the audio HAL may need to parse metadata in the compressed audio data to determine its playback duration.

Suspended support

Android 5 or lower does not include suspend support. You can only pause tunnel playback via A/V starvation, but if the video's internal buffer is large (e.g. one second of data in the OMX component) it can make the pause look unresponsive.

In Android 5.1 or later, AudioFlinger supports pausing and resuming direct (tunneled) audio output. If the HAL implements pauses and resumes, trace pauses and resumes are forwarded to the HAL.

The pause, refresh, resume call sequence is respected by executing HAL calls in the playback thread (same as unloading).

Implementation suggestions

Audio HAL

With Android 11, HW sync IDs from PCR or STC can be used for A/V sync, so video-only streaming is supported.

For Android 10 or lower, devices that support tunneled video playback should have at least one file with and audio_policy.conf Audio output stream configuration file with a> flag. These flags are used to set the system clock from the audio clock. FLAG_HW_AV_SYNCAUDIO_OUTPUT_FLAG_DIRECT

OMX

Device manufacturers should have a separate OMX component for tunneled video playback (manufacturers can have additional OMX components for other types of audio and video playback, such as secure playback). Tunnel components should:

• specifies 0 buffers on its output port ( nBufferCountMin , nBufferCountActual ).

• RealOMX.google.android.index.prepareForAdaptivePlayback setParameterExhibition.

• Specify its capabilities inmedia_codecs.xml the file and declare the tunnel playback capabilities. It should also clarify any restrictions on frame size, alignment, or bitrate. An example looks like this:

  <MediaCodec name="OMX.OEM_NAME.VIDEO.DECODER.AVC.tunneled"
  type="video/avc" >
      <Feature name="adaptive-playback" />
      <Feature name="tunneled-playback" required=”true” />
      <Limit name="size" min="32x32" max="3840x2160" />
      <Limit name="alignment" value="2x2" />
      <Limit name="bitrate" range="1-20000000" />
          ...
  </MediaCodec>
  

If the same OMX component is used to support tunneled and non-tunneled decoding, it should leave the tunnel playback functionality as non-required. Both tunneled and non-tunneled decoders have the same capability limitations. An example looks like this:

<MediaCodec name="OMX._OEM\_NAME_.VIDEO.DECODER.AVC" type="video/avc" >
    <Feature name="adaptive-playback" />
    <Feature name="tunneled-playback" />
    <Limit name="size" min="32x32" max="3840x2160" />
    <Limit name="alignment" value="2x2" />
    <Limit name="bitrate" range="1-20000000" />
        ...
</MediaCodec>

Hardware Composer (HWC)

When there is a tunnel layer (a layer with HWC_SIDEBAND compositionType) on the display, the layer's sidebandStream is the sideband handle assigned by the OMX video component.

HWC synchronizes the decoded video frames (from the tunnel OMX component) to the associated audio track (using the audio-hw-sync ID). When a new video frame becomes current, HWC composites it with the current contents of all layers received during the last prepare or setup call, and displays the resulting image. Prepare or setup is only called when other layers change or when properties of the sideband layer change (such as position or size).

The diagram below represents the HWC working with a hardware (or kernel or driver) synchronizer to combine video frames (7b) with the latest combination (7a) based on audio (7c) to display at the correct time.

Figure 2. HWC hardware (or kernel or driver) synchronizer