VLC source code analysis: the technology behind video playback speed control

Introduction and environment preparation

1.1 Introduction to VLC Player

VLC (full name VideoLAN Client) is a popular open source cross-platform multimedia player with rich features and fast performance. Since its launch in 1996, it has accumulated a large number of users worldwide. What makes VLC unique is its support for various media formats such as video, audio, subtitles, and more. Whether it is local files or network streaming media, VLC can provide users with a good playback experience.

VLC player is written in C, C++ and Qt, under the GNU General Public License. This enables any user to freely modify, distribute and contribute to the code. Due to its open source attributes and cross-platform features, VLC has excellent compatibility in various operating systems, such as Windows, Mac OS, Linux, Android, and iOS.

One of VLC's core strengths is its modular structure, which can be easily extended and customized. Users can customize the functions of the player according to their needs, such as various built-in decoders, renderers, filters, etc. In addition, VLC provides a powerful API, which is convenient for developers to carry out secondary development and realize customized playback functions and user interfaces.

In this blog, we will perform an in-depth analysis of the VLC source code, focusing on the implementation details of playback speed control. By understanding how VLC achieves this function, we can better understand the inner workings of high-performance multimedia players.

1.2 Compile and build VLC

In order to gain an in-depth understanding of the source code of the VLC player and the implementation of playback speed control, we need to obtain the source code of VLC first, and compile and build it in the local environment. A brief step-by-step instruction is provided below to help you get started with VLC source code.

1. Get the source code: visit the VLC official GitHub repository ( https://github.com/videolan/vlc), and clone the source code to the local disk. You can use the following commands:

   git clone https://github.com/videolan/vlc.git
   

2. Install dependencies: Depending on the operating system you are using (Windows, Mac OS, or Linux, etc.), install the dependencies required for VLC builds. The official documentation ( https://wiki.videolan.org/Category:Building_VLC/) provides detailed dependency installation guidelines.
3. Configure the build environment: In the cloned VLC source directory, run bootstrapthe command to generate configurethe script . Then, execute configurethe script and pass the appropriate parameters to set up the build environment. In general, depending on the operating system and hardware architecture, this step may require custom settings.

   cd vlc
   ./bootstrap
   ./configure
   

4. Compile VLC: After successfully configuring the build environment, run makethe command to start compiling VLC. This process may take some time, depending on your hardware configuration.

   make
   

5. Install and run VLC: After compiling, install VLC on the local system and run it. Note that specific installation steps may vary depending on the operating system.

   sudo make install
   vlc
   

So far, we have successfully obtained the source code of VLC and built the VLC player in the local environment. Next, we'll dig into the structure and components of the VLC source code, focusing on how playback speed control is implemented.

Overview of VLC source code structure

2.1 Main modules and functional components

In this section, we will introduce the main modules and functional components in the VLC player source code structure.

The source code structure of VLC player is quite complex, including several powerful modules. Here is an overview of several key modules:

1. Core module (src/):

   The VLC core module is located in the src/ directory, which includes the implementation of basic functions and underlying libraries, such as the main event loop, playback control, playlist, input, output, decoder, etc.

2. Module loader (modules/):

   The modules/ directory contains various plug-ins and function implementations of VLC player, such as audio/video decoder, format demultiplexer, audio/video filter, image/audio output, flow control, etc. These modules can be dynamically loaded at runtime to support different usage scenarios and application requirements.

3. Cross-platform compatibility layer (compat/):

   The compat/ directory contains a series of implementations to provide cross-platform compatibility. They are used to abstract operating system and hardware features so that VLC player runs smoothly on multiple platforms.

4. User Interface (ui/):

   The ui/ directory contains various user interface implementations of VLC players, such as Qt, GTK+, and macOS. Through different user interfaces, VLC can provide a native user experience for various operating systems.

5. Tests and tools (test/, extras/):

   These two directories respectively contain tools such as unit testing, performance testing, and scripts for verifying and assisting VLC player development.

In the next chapters, we will focus on the modules and source code implementations related to playback speed control.

2.2 Play speed control related modules

In this section, we will focus on the modules related to the playback speed control function in the VLC source code.

Playback speed control involves several key components in VLC player. The following modules are closely related to the playback speed control function:

1. Input thread (src/input/):

   The input thread is responsible for receiving data from different input sources (such as files, networks, hardware devices, etc.) and passing these data to appropriate decoders and filters. In src/input/control.c, you can find functions that handle playback speed control related functions (such as input_Control() and ControlSetRate()).

2. Audio output module (modules/audio_output/):

   The audio output module is responsible for passing the decoded audio data to the system audio device or a filter that processes the audio data. In this directory, you can find audio output module implementations for different platforms and audio devices. The playback speed control has effects on the audio output, such as adjusting the audio rate of the audio device.

3. Video output module (modules/video_output/):

   The video output module is responsible for delivering the decoded video data to the system display device or the filter that processes the video data. In this directory, you can find video output module implementations for different platforms and display devices. The playback speed control has an effect on the video output, such as adjusting the video frame rate of the display device.

4. Codec (modules/codec/):

   The decoder module includes implementations of audio and video decoders. Some aspects of the playback speed control need to work in conjunction with the decoder to provide a smoother playback experience, such as the Pitch Shifting function in the audio decoder to adjust the pitch.

By studying these modules, we can gain a deep understanding of how the VLC playback speed control function is implemented, and explore the factors that affect the speed control function. In the following chapters, we will analyze the relevant codes in these modules in detail to understand the implementation details of playback speed control in VLC.

Here I provide you with the first part of the detailed analysis of the playback speed control implementation: "Playback Speed ​​Adjustment Interface and Logic".

Detailed analysis of playback speed control implementation

3.1 Playback rate adjustment interface and logic

In the VLC source code, the playback rate adjustment function is realized through the following key parts:

libvlc interface

libvlc is the core library of the VLC player, providing various functions for the player. To adjust video and audio playback speed, libvlc_media_player_set_rate and libvlc_media_player_get_rate functions can be used. Here is their general usage:

// 设置播放速率（假设已创建libvlc_media_player_t *p_media_player变量）
float speed = 2.0f; // 假设我们要将播放速度加倍
libvlc_media_player_set_rate(p_media_player, speed);

// 获取当前播放速率
float current_speed = libvlc_media_player_get_rate(p_media_player);

Input reader (input_reader) layer

The input reader (input_reader) is responsible for managing the stream of input data so that the decoder can process it. It includes a clock component to control the read speed. The clock component is mainly composed of input_clock_t structure and related functions, such as: input_clock_SetRate function. The input_clock_SetRate function is used to modify the internal clock frequency to achieve different playback rates.

Decoder layer

A decoder is responsible for decoding the raw data stream into audio or video output. During this process, the decoder needs to adjust the decoding speed according to the clock frequency of the input reader. For example, in an audio decoder, the decoder adjusts the playback rate of the sound based on the input clock.

Through the cooperative work of these three key parts, VLC player has successfully realized the control function of playback speed. In the next subsections (3.2 and 3.3), we will dive into the detailed implementation of speed adjustment in the audio and video output modules.

Here, I provide you with the second part of the detailed analysis of the playback speed control implementation: "Speed ​​adjustment in the audio output module".

3.2 Speed ​​adjustment in audio output module

VLC player needs to make sure that when changing the playback speed, the speed of the audio is adjusted accordingly. The following are the key parts in the audio output module, involving the control of playback speed:

Audio Codec and Resampling

When the playback speed changes, the audio decoder needs to ensure that the sample rate during decoding is synchronized with the input clock. In addition, the audio decoder is responsible for the resampling operation to ensure that the audio data is compatible with the sample rate of the target device. Resampling libraries used by VLC include Speex, soxr, etc. In the audio_output/aout_internal.hand audio_output/dec.cfile, we can see the code related to resampling and playback speed adjustment.

audio output interface

The audio output (aout) module implements a unified interface for sending processed audio data to audio devices on different platforms. For example, on the Windows platform, the aout module uses audio_output/mmdevice.cfiles to interoperate with WASAPI (Windows Audio Session API). On the Linux platform, the aout module interacts with audio systems such as ALSA and PulseAudio. In the implementation of the aout module, we can see how to adjust the audio playback rate according to the current playback speed.

Audio Synchronization Mechanism

When VLC player handles the speed adjustment, it needs to ensure the synchronization of audio and video. This is mainly achieved by adjusting the time stamp of the audio output to prevent the out-of-sync phenomenon of audio and video caused by speed changes. In audio_output/aout_internal.hthe file, aout_DecPlayfunctions are used to process the audio decoding output and ensure that the audio is synchronized to the input clock.

Through these key components and steps, VLC player realizes the function of adjusting audio output during variable-speed playback. In the next subsection (3.3), we discuss the detailed implementation of speed adjustment in the video output module.

3.3 Speed ​​adjustment in video output module

In order to ensure that when the playback speed changes, the speed of the video is also adjusted synchronously, the video output module in the VLC player needs to handle the speed adjustment. The following are the key parts related to the playback speed control in the video output module:

Video decoder synchronized with clock

The video decoder needs to adjust the decoding speed according to the input clock frequency. In src/input/decoder.cthe file, DecoderPlayVideofunctions are responsible for processing the decoded video frames. To ensure that the video frames are synchronized with the input clock, the function calculates the display time of each video frame and adjusts it.

Video output interface

The video output (vout) module implements a unified interface for sending processed video data to display devices on different platforms. For example, on Windows, the vout module uses video_output/win32/direct3d11.cfiles to interoperate with Direct3D 11. On the Linux platform, the vout module interacts with display systems such as X11 and Wayland. In the implementation of the vout module, we can see how to adjust the display rate of video frames according to the current playback speed.

Video Synchronization Mechanism

VLC player needs to ensure that the video stays in sync with the audio when changing the playback speed. Video synchronization is achieved by adjusting the display timestamps of video frames to prevent speed changes from causing the picture and audio to be out of sync. In src/video_output/video_output.cthe file, vout_ManageWrapperthe function handles the synchronization of the video output layer with the input clock.

Through the above key components and steps, the VLC player realizes the synchronous adjustment of the playback speed of video and audio. This ensures that users can get a good audio-visual experience when watching videos at any speed.

3.4 Use VLC source code and comments to illustrate playback speed control

We will illustrate the three-dimensional operation of the playback speed control by analyzing the key parts of the VLC source code:

Entry point to control velocity adjustments

VLC player accepts speed adjustment requests via libvlc_media_player_set_rateand libvlc_media_player_get_ratefunctions. The source code file is lib/libvlc_media_player.c. For example, the relevant code to set the speed:

/**
 * Set the media player playback speed
 *
 * \param p_mi media player
 * \param rate the new playback speed as a float
 */
void libvlc_media_player_set_rate( libvlc_media_player_t *p_mi, float rate )
{
    
    
    var_SetFloat( p_mi, "rate", rate );
}

This function will modify the internal variable "rate", and subsequent audio and video decoding will be adjusted according to the value of this variable.

The speed adjustment is passed to the input reader

input_clock_SetRateSynchronously adjust the input reader's input clock rate by calling it when the "rate" variable is changed . The source code file is src/input/clock.c. Here is the relevant part of the source code:

bool input_clock_SetRate( input_clock_t *cl, float rate )
{
    
    
    if( rate <= 0 )
        return false;

    const int i_irate = INPUT_RATE_DEFAULT / rate;

    /*必要时归一化速率*/
    ...
    cl->i_rate = i_irate;
    ...
}

Decoder processing speed adjustment

The audio and video decoder will adjust the decoding speed according to the rate of the input clock. Taking an audio decoder as an example, adjusting the sample rate needs to match the input clock rate change. The relevant files are src/audio_output/dec.c:

static void DecoderProcessSout( aout_owner_t *p_owner, block_t *p_block )
{
    
    
    ...
    if( input_rate != p_owner->sync.rate )
    {
    
    
        if( p_owner->sync.rate != INPUT_RATE_DEFAULT )
        {
    
    
            ...
            module_t *p_module;
            p_owner->p_resampler = module_need( p_owner->p_resampler, "audio resampler", "$resampler", false );
        }
    }
    ...
}

In the source code, we can see that VLC uses an audio resampler to ensure that the audio decoder can handle the speed adjustment correctly. This processing will ensure that the audio does not distort when the playback speed is adjusted.

Similarly, in the video decoder, we can src/input/decoder.cfind the implementation of video decoding speed adjustment in the file:

static void DecoderPlayVideo( decoder_t *p_dec, picture_t *p_picture, float *rates )
{
    
    
    ...
    /*计算原始PTS值*/
    ...
    
    if( i_rate != 0 )
    {
    
    
        if( unlikely( rate != 1.f ) )
        {
    
    
            /*根据播放速率调整PTS值*/
            pts = coords->begin + ( pts - coords->begin ) * rate;
        }
    }

    ...
    /* 实现音视频同步 */
}

Through the above source code examples, we can see how VLC player implements playback speed control at the code level. From the control entrance to the processing of the decoder, the whole process is coordinated to ensure that the playback speed adjustment can smoothly adapt to the audio and video synchronization and playback experience.

3.5 The complete process from input to audio and video rendering

This section will show you in more detail how VLC handles playback speed adjustments. We will trace the process of speed adjustment from the code level, starting from the input, through decoding, and finally reaching the output stage of audio and video rendering.

control input rate

In src/input/var.cthe file we can see how the playback speed value is applied to the input layer:

/**
 * Callback for the "rate" variable
 */
static int RateCallback( vlc_object_t *obj, char const *name,
                         vlc_value_t oldval, vlc_value_t newval, void *data )
{
    
    
    ...
    input_thread_t *input = (input_thread_t *)obj;
    return input_ControlPushHelper( input, INPUT_CONTROL_SET_RATE, &newval );
}

Let the input clock be affected by the rate

The input clock affects the overall processing. In order for it to be affected by playback speed, we want to look at src/input/clock.cthe file.

int input_clock_ConvertTS( input_clock_t *cl, int64_t *pi_ts0, int64_t *pi_ts1, int64_t i_ts_bound, int i_rate )
{
    
    
    ...
    lldiv_t q;
    q = lldiv( CLOCK_FREQ * i_rate * 10LL, INPUT_RATE_DEFAULT );
    i_symbol_duration = q.quot;
    ...
    *pi_ts0 = ( i_date - cl->i_pts_delay ) * cl->i_rate / INPUT_RATE_DEFAULT;
    *pi_ts1 = ( i_pulse - cl->i_pts_delay ) * cl->i_rate / INPUT_RATE_DEFAULT;
    ...
}

Audio decoding and rendering

In src/audio_output/dec.cthe file, we can see the rate adjustment of the audio decoding process. When decoding using the correct resampler based on the playback rate, the resampled audio will be passed to the various platform's audio output implementations.

static void Play( aout_owner_t *p_owner, block_t *p_buffer )
{
    
    
    ...
    if( p_owner->sync.rate != INPUT_RATE_DEFAULT )
    {
    
    
        /*利用重采样器根据播放速率调整音频速度*/
        assert( p_owner->p_resampler != NULL );
        p_buffer->i_nb_samples = resampler->pf_resample( p_owner, &p_sys->p_out_buf, &p_sys->i_out_size, p_buffer->p_buffer, p_buffer->i_buffer / p_sys->psz_frame );
    }
    ...
}

Video decoding and rendering

In src/input/decoder.cthe file we can see the rate handling of the video codec. The following code calculates the rate-adjusted display time and makes the playback speed affect the decoding of video frames.

/* 调整视频帧PTS值 */
static void DecoderPlayVideo( decoder_t *p_dec, picture_t *p_picture, float *rates )
{
    
    
    ...
    if( i_rate != 0 )
    {
    
    
        if( unlikely( rate != 1.f ) )
        {
    
    
            pts = coords->begin + ( pts - coords->begin ) * rate;
        }
    }

    ...
}

In src/video_output/video_output.cthe file, the following code keeps the adjusted video frame time and audio clock in sync.

static void ThreadProcFrame( vout_thread_t *vout, vout_thread_sys_t *sys, bool *p_paused, mtime_t *p_display_date, mtime_t *p_wakeup )
{
    
    
    ...
    if( rate != 1.f )
    {
    
    
        display_gap = lroundf( display_gap / rate );
    }
    ...
}

Through the above sample code, we deeply analyzed the details of VLC's handling of playback speed control at various levels. Key components such as decoders, input clocks, and audio and video rendering are all involved in the adjustment process to ensure that while changing the playback speed, the synchronization and rendering effects of video and audio can meet user expectations.

3.6 Use C++ code to realize playback speed control in VLC source code

In order to use C++ to write a simple program to control the playback speed of VLC, we need to use the libVLC library. The following is a simple C++ code example to implement playback speed control using libVLC:

#include <iostream>
#include <unistd.h>
#include <vlc/vlc.h>

int main(int argc, char** argv) {
    
    
    const char * const argv_new[] = {
    
    
        "-I",
        "dummy",
        "--ignore-config",
        "--no-video-title-show",
        "--no-stats",
        "--verbose=0"
    };
    int argc_new = sizeof(argv_new) / sizeof(*argv_new);
 
    // 初始化libVLC
    libvlc_instance_t *vlc_instance = libvlc_new(argc_new, argv_new);
    if (!vlc_instance) {
    
    
        std::cerr << "Failed to initialize libVLC" << std::endl;
        return 1;
    }

    // 创建媒体播放器
    libvlc_media_player_t *media_player = libvlc_media_player_new(vlc_instance);
    if (!media_player) {
    
    
        std::cerr << "Failed to create media player" << std::endl;
        libvlc_release(vlc_instance);
        return 1;
    }

    // 设定播放媒体源
    libvlc_media_t *media = libvlc_media_new_location(vlc_instance, "file:///path/to/your/video/file.mp4");
    libvlc_media_player_set_media(media_player, media);

    // 开始播放
    libvlc_media_player_play(media_player);
    sleep(10); // 播放 10 秒

    // 改变播放速度
    float rate = 2.0f;
    libvlc_media_player_set_rate(media_player, rate);
    std::cout << "Changing playback speed to: " << rate << "x" << std::endl;
    sleep(10); // 以 2 倍速度播放 10 秒

    // 释放资源
    libvlc_media_player_stop(media_player);
    libvlc_media_release(media);
    libvlc_media_player_release(media_player);
    libvlc_release(vlc_instance);

    return 0;
}

In this example, we first initialize a libVLC instance and create a media player. Then we set a media source and start playing. After 10 seconds of playback, we change the playback speed to 2x. Next, the program plays at 2x speed for 10 seconds, and finally stops playing and releases related resources.

Make sure to link the libVLC library when compiling this example. For example, on a Linux platform, you can use the following command to compile the example code:

g++ vlc_speed_control_example.cpp -o vlc_speed_control_example -lvlc

By using the libVLC library, we can easily implement the playback speed control of the VLC player in the C++ program, which is very suitable for various customized player applications.

3.7 The scheme of playing speed control based on FFmpeg

When implementing the player based on FFmpeg, we can adjust the playback speed according to the design similar to VLC. The following is a simple scheme of how a FFmpeg-based player implements playback speed control:

1. Decoding input: First, decode the audio and video stream by using FFmpeg's AVFormat and AVCodec libraries. Adjusts the rate at which audio and video frames are written to the decode buffer during playback speed control.

   AVFormatContext *pFormatCtx = nullptr;
   avformat_open_input(&pFormatCtx, input_filename, 0, 0);
   avformat_find_stream_info(pFormatCtx, nullptr);
   

2. Audio processing and rate adjustment: implement audio resampling in the audio decoder. When modifying the playback speed, you need to adjust the audio sample speed according to the new rate value. FFmpeg can libswresampleachieve this function through the library.

   SwrContext *swr_ctx;
   swr_ctx = swr_alloc_set_opts(swr_ctx, out_channel_layout, out_sample_fmt, out_sample_rate, in_channel_layout, in_sample_fmt, (int)(in_sample_rate * playback_rate), 0, NULL);
   swr_init(swr_ctx);
   

3. Video processing and rate adjustment: In the video decoder, the presentation time value (PTS, presentation timestamp) of the video frame is adjusted according to the playback speed. For example, when modifying the playback speed, the PTS needs to be adjusted according to the new speed value.

   AVStream *video_stream = pFormatCtx->streams[video_stream_index];
   
   // 解码完一帧数据后，调整frame的PTS
   double playback_rate = 2.0; // 设置当前的播放速度
   frame->pts *= playback_rate; 
   

4. Audio and video synchronization: When playing audio and video, the two with adjusted playback speed are kept in sync. Consider using a synchronized clock to coordinate audio and video playback to ensure that audio and video frames are rendered correctly at the correct playback rate respectively.
5. Render adjusted audio and video: for audio, play according to the new sampling speed; for video, render according to the adjusted PTS value. The rendering of audio and video can be implemented by relying on platform-related libraries.

Through the above steps, the audio and video playback speed control is realized in the FFmpeg-based player. The whole process will involve the adjustment of multiple components to ensure that while changing the playback speed, the audio and video can be rendered synchronously without affecting the playback experience.

Implementation mode and characteristics of VLC playback speed control

Synchronization and Adjustment Policies

In VLC player, playback speed control is achieved by adjusting the rate of audio and video output. In order to ensure that the audio and video are synchronized when the user uses the acceleration or deceleration function, VLC uses the following strategy:

1. Audio and video synchronization: VLC will handle all audio and video decoding, rendering and other tasks internally. When the user adjusts the playback speed, the audio output module and the video output module will independently adapt to this change and maintain the synchronization relationship between audio and video.
2. Speed ​​adjustment sampling: When VLC player adjusts the audio speed, it needs to adjust the sampling rate to the target playback speed. This is achieved by resampling the audio samples. For the video part, VLC utilizes timestamps for frame rate adjustments to keep the video playing smoothly when changing the playback speed.
3. Dynamic speed adjustment: Users can adjust the speed at any time during playback, and VLC will apply these changes in real time. By utilizing threads and an internal event system, VLC is able to respond quickly to these speed changes, enabling user actions to be met without compromising performance.
4. Latency Compensation: When handling playback speed controls, VLC is aware of possible delays in the adjustment process. In order to ensure the synchronization of audio and video and reduce the impact of delay, VLC adopts internal compensation strategies, such as adjusting the buffer size of the decoder according to the delay, to ensure that the adjustment process becomes smoother from the user's perspective.

In VLC, these synchronization and adjustment strategies jointly ensure efficient control of playback speed and meet users' expectations for a balance between performance and performance.

Multi-platform adaptation and performance optimization

In order to ensure that the VLC playback speed control function can work properly on different devices and operating systems, VLC has implemented multi-platform adaptation and performance optimization:

1. Multi-platform adaptation: VLC supports various operating systems, such as Windows, macOS, Linux, Android, etc. In order to use a unified playback speed control function on different platforms, VLC uses a portable programming interface and a general programming method. This cross-platform implementation ensures functional consistency and maintainability.
2. Hardware acceleration: For different platforms and devices, VLC supports hardware acceleration functions, such as using GPU for decoding and rendering operations. In this way, VLC can reduce CPU load, improve performance, and save battery life (especially for mobile devices) when adjusting playback speed.
3. Optimization algorithm: In order to maintain high sound quality and high image quality when adjusting the playback speed, VLC selects some algorithms with excellent performance for resampling and video frame rate adjustment. These algorithms are designed to minimize distortion and loss of data, maintaining audio and video quality.
4. Configurability: VLC allows users to customize settings related to playback speed control, such as speed range and step size. In this way, VLC realizes the adaptation to various usage scenarios and user needs.
5. System resource management: In the process of realizing the playback speed control function, VLC attaches great importance to the management of system resources. By effectively managing resources such as memory, CPU, and GPU, VLC ensures that it does not consume excessive system resources while responding quickly to user operations.

To sum up, VLC pays attention to multi-platform adaptation and performance optimization when implementing the playback speed control function. In this way, VLC ensures an efficient, stable and customizable playback speed control experience on various devices and operating systems.

Summary and Outlook

In this blog, we focus on the source code implementation of the playback speed control function in VLC player. Through in-depth analysis of the source code, we understand the design concept behind this function, the key role of each module and the optimization strategy on different platforms.

From a psychological point of view, parsing and learning the source code of a complex open source project like VLC is of great significance to programmers. Not only does it improve technical proficiency and awareness, but it also helps develop the mental capacity to face technical challenges. Therefore, in-depth analysis of such projects has long-term value for the growth of programmers.

However, it is worth noting that in order to truly master the function of playback speed control in VLC player, more practical operations need to be combined with the actual development environment. With the help of the analysis of this blog, I hope you can establish an overall understanding of this function and provide guidance for your actual development and exploration.

If this blog can inspire and help you, please don't hesitate to like and bookmark it, and we are also looking forward to seeing your comments and feedback. Hope this blog inspires you to make more progress on the dual path of technical and psychological growth. In the future, we will continue to pay attention to the technical analysis of VLC player and other open source projects, and provide more valuable content for programmers to learn and grow.