This article mainly introduces the video development cases of i.MX 8M Mini, including video capture, codec, algorithm processing, display and storage cases based on GStreamer, GigE industrial camera test instructions, H.265 video hardware decoding function demonstration instructions, etc.
Note: The evaluation version of this case uses Chuanglong Technology TLIMX8-EVM, which is a high-performance evaluation board based on the quad-core ARM Cortex-A53 + single-core ARM Cortex-M4 heterogeneous multi-core processor design of NXP i.MX 8M Mini ,
Consists of a core board and an evaluation board. The ARM Cortex-A53 (64-bit) main processing unit has a main frequency of up to 1.6GHz, and the ARM Cortex-M4 real-time processing unit has a main frequency of up to 400MHz. The processor adopts the latest 14nm technology, supports 1080P60 H.264 video hardware codec, 1080P60 H.265 video hardware decoding, and GPU graphics accelerator. The core board has been verified by professional PCB layout and high and low temperature tests. It is stable and reliable, and can meet various industrial application environments. The front appearance is as follows:
3 gige_capture case
In this case, the Gigabit Ethernet port of the evaluation board is connected to the GigE industrial camera to realize the functions of image acquisition, image display and image storage. Camera model: acA720-290gm, brand: Basler, the resolution can be configured up to 720x540, and the frame rate can reach 291fps. For details about the camera, see the official website link: www.baslerweb.com/cn/products/cameras/area-scan-cameras/ace/aca720-290gm .
The measured frame rate in this case is: capture and display as 720x540@138fps, capture and save as 720x540@31fps.
The test case gige_capture is located in the product documentation "4-software documentation\Demo\base-demos\" path. For the convenience of testing, the verified test program executable file is provided in the product documentation, which is located in the bin directory of gige_capture, and the source code is located in the src directory of gige_capture.
3.1 GigE industrial camera configuration
This section uses the pylonSDK development kit officially provided by Basler to configure parameters such as the IP address acquisition method and acquisition frame rate of the GigE industrial camera on the PC Ubuntu.
Copy the pylonSDK-5.1.0.12682-x86_64.tar.gz compressed package in the "gige_capture\pylonSDK\x86_64\" directory to any path in Ubuntu, and enter the directory where the compressed package is located and execute the following command to decompress it to the current directory.
Host# tar -xvf pylonSDK-5.1.0.12682-x86_64.tar.gz -C .
Figure 40
Figure 41
Power on the GigE industrial camera, connect the PC and the GigE industrial camera to the same router through a network cable, and execute the following command to run the IpConfigurator tool to configure the IP address acquisition method.
Host# cd pylon5/bin/
Host# ./IpConfigurator
Figure 42
In the IpConfigurator interface, click Refresh to refresh and identify the GigE industrial camera. It can be seen from the configuration interface that the GigE industrial camera supports two modes: static IP (Static IP) and dynamic IP (DHCP). This time, the dynamic IP mode is used for testing, select the DHCP option, click Save to save, and the IP address obtained in the test is 192.168.0.165.
Figure 43
Close the IpConfigurator tool window, execute the following command to run the PylonViewerApp tool, and configure the acquisition frame rate and other parameters.
Host# ./PylonViewerApp
Figure 44
In the PylonViewerApp interface, double-click to select the GigE network camera device, select Expert for User Level, and click to open the camera configuration column, as shown below.
Figure 45
Click to open the "Configuration Sets" column, and initialize according to the following figure.
Set the "User Set Selector" option and the "Default Startup Set" option under the "Configuration Sets" column to "Default User Set".
Figure 46
Click to open the "Transport Layer" column to set the camera network packet configuration.
Set the "Packet Size" under the "Transport Layer" column to 1500. "Inter-Packet Delay" is set to 4500. The "Inter-Packet Delay" parameter value will affect the capture frame rate, please modify it according to the actual situation.
Click the menu bar button for continuous acquisition, and you can view the images collected by the GigE network camera through the PylonViewerApp.
Figure 47
You can set the "Exposure time" value under the "Acquisition Controls" column to adjust the camera exposure time. The larger the value, the lower the frame rate. The test uses the camera's default value of 3000us, corresponding to a frame rate of 274fps.
Figure 48
The image size captured by the camera can be set in the "AOI Controls" column, and the default size is 720x540.
Figure 49
3.2 Image acquisition test
Connect the RGMII ETH interface of the evaluation board and the GigE industrial camera to the same router through a network cable. The evaluation board supports HDMI display by default. Please connect the HDMI display correctly first.
Remarks: If the GigE industrial camera is configured with a static IP, it is necessary to directly connect the evaluation board and the GigE industrial camera through a network cable, and set the evaluation board and the GigE industrial camera to the same network segment.
Figure 50
Copy the library file compression package libpylon_arm64.tar.gz under the "gige_capture\lib\" directory to any directory of the evaluation board file system, enter the directory where the compression package is located, and execute the following command to decompress it to the "/usr/lib/" directory.
Target# tar -xvf libpylon_arm64.tar.gz -C /usr/lib
Figure 51
Power on the GigE industrial camera and enter the file system of the evaluation board. Please copy the executable file gige_capture in the bin directory to any path in the file system of the evaluation board. Execute the following command in the directory where the gige_capture executable file is located to view the program parameter description.
Target# ./gige_capture --help
Figure 52
Execute the following commands to scan the GigE industrial cameras in the LAN, and test the image acquisition frame rate of the GigE industrial cameras.
Target# ./gige_capture -m scan
Figure 53
And execute the following command to close the Weston interface, and display the image collected by the GigE industrial camera on the display. If the image display is not clear, please turn the knob to adjust the aperture and focal length of the camera lens. The GigE industrial camera will collect in real time at the set resolution of 720x540. There will be certain fluctuations due to the influence of hardware performance and network environment.
Target# systemctl stop [email protected]
Target# ./gige_capture -m display -o /dev/fb
Figure 54
Figure 55
The CPU usage of this test is shown below.
Figure 56
Execute the following command to save the image captured by the GigE industrial camera to a file in the specified path in gray8 format. The number of frames collected and stored will be affected by the reading and writing rate of the device. The actual measurement is 31fps.
Target# ./gige_capture -m save -o /home/root/test.gray8
Figure 57
Copy the saved file to Windows, use the "gige_capture\tools\yuvplayer.exe" program to open the test.gray8 file, click "Size -> Custom", set the image format Width to 720, Height to 540, click "Color -> Y", click the play button to display the image as shown below.
Figure 58
Figure 59
Figure 60
3.3 Case Compilation
Copy the entire directory of gige_capture to any path in Ubuntu. Please ensure that the pylonSDK-5.1.0.12682-arm64.tar.gz in the "pylonSDK\arm64\" directory has been decompressed. Enter the case src directory and execute the following commands to load the SDK environment variables and compile.
Host# source /home/tronlong/SDK/environment-setup-aarch64-poky-linux
Host# make
Figure 61
After the compilation is complete, an executable file will be generated in the current directory.
Figure 62
4 H.265 video hardware decoding function demonstration
The i.MX 8M Mini processor contains a VPU (Video Processing Unit) G2 decoder, which can decode H.265 video files by hardware, and supports up to 1080P60.
This chapter mainly demonstrates that i.MX 8M Mini is based on the Gstreamer framework and uses the VPU G2 decoder to decode H.265 video files, including MKV (HEVC), MP4 (HEVC) video files, and HEVC (H.265 original format) video files.
Figure 63
The i.MX 8 Gstreamer official reference document is the product documentation "6-Development Reference Documents\NXP Official Reference Documents\i.MX 8 GStreamer User Guide.pdf". The document introduces some commonly used plug-ins (Plugins) of Gstreamer, specific plug-ins for i.MX 8, and codec examples.
4.1 Decoding MKV video files
Please connect the HDMI OUT interface of the evaluation board to the HDMI display, and copy the bbb-1920x1080-cfg02.mkv (1080P60 with audio) video file in the product documentation "4-Software Information\Tools\Video\" directory to the evaluation board file system . Enter the path where the video file is located, execute the following command to decode, and the MKV decoded video can be displayed on the HDMI display.
Target# gst-launch-1.0 -v filesrc location=./bbb-1920x1080-cfg02.mkv ! matroskademux name=demux demux.video_0 ! queue ! h265parse ! vpudec ! queue ! waylandsink window-width=1920 window-height=1080
Figure 64
The HDMI display effect is as follows. After the playback ends or press "Ctrl + C" to exit the video playback, the serial debugging terminal will prompt the video output frame rate.
Figure 65
Figure 66
The Pipeline workflow for video decoding in this case is shown below.
Figure 67
The Pipeline is composed of 6 Elements (Plugins), and each Element implements its own function.
(1) filesrc: read files.
(2) matroskademux: parse MKV files and extract data.
(3) queue: cache data.
(4) h265parse: Use the H.265 interpreter.
(5) vpudec: Use the VPU decoder to decode H.265 video files.
(6) waylandsink: select the display device and output it, and set the output resolution to 1920 x 1080.
Figure 68
Figure 69
bbb-1920x1080-cfg02.mkv is a video file with audio. If you want to play video and audio at the same time, please insert the Chuanglong Technology TL3106 audio module into the J10 interface, and connect the LINE OUT interface of the module to headphones or speakers.
Figure 70
Enter the file system of the evaluation board and execute the following commands, and add audio decoding and playback parameters to the commands to play video and audio at the same time.
Target# gst-launch-1.0 -v filesrc location=./bbb-1920x1080-cfg02.mkv ! matroskademux name=demux demux.audio_0 ! queue ! decodebin ! audioconvert ! audioresample ! autoaudiosink demux.video_0 ! queue ! h265parse ! vpudec ! queue ! waylandsink window-width=1920 window-height=1080
Figure 71
Figure 72
The Pipeline workflow for simultaneous decoding of video and audio in this case is shown below.
Figure 73
4.2 Decoding MP4 video files
Please connect the HDMI OUT port of the evaluation board to the HDMI display, and copy the Elecard_about_Tomsk_part1_HEVC_1080p.mp4 (1080P25) video file in the product documentation "4-Software Documentation\Tools\Video\" directory to the evaluation board file system. Enter the path where the video file is located, execute the following command to decode, and the MP4 decoded video can be displayed on the HDMI display.
Target# gst-launch-1.0 -v filesrc location=./Elecard_about_Tomsk_part1_HEVC_1080p.mp4 ! qtdemux name=demux demux.video_0 ! queue ! h265parse ! vpudec ! queue ! waylandsink window-width=1920 window-height=1080
Figure 74
Figure 75
The HDMI display effect is as follows. After the playback ends or press "Ctrl + C" to exit the video playback, the serial debugging terminal will prompt the video output frame rate.
Figure 76
Figure 77
Explanation of command parameters:
(1) filesrc: read files.
(2) qtdemux: parse MP4 files and extract data.
(3) queue: cache data.
(4) h265parse: Use the H.265 interpreter.
(5) vpudec: Use the VPU decoder to decode H.265 video files.
(6) waylandsink: select the display device and output it, and set the output resolution to 1920 x 1080.
Figure 78
4.3 Decoding HEVC video files
Please connect the HDMI OUT interface of the evaluation board to the HDMI display, and copy the sample_1920x1080.hevc (1080P30) video file in the product documentation "4-Software Documentation\Tools\Video\" directory to the evaluation board file system. Enter the path where the video file is located, execute the following command to decode, and the HEVC (H.265 original format) decoded video can be displayed on the HDMI display.
Target# gst-launch-1.0 -v filesrc location=sample_1920x1080.hevc ! queue ! h265parse ! vpudec ! queue ! waylandsink window-width=1920 window-height=1080
Figure 79
The HDMI display effect is as follows. After the playback ends or press "Ctrl + C" to exit the video playback, the serial debugging terminal will prompt the video output frame rate.
Figure 80
Figure 81
Parameter explanation:
(1) filesrc: read files.
(2) queue: cache data.
(3) h265parse: Use the H.265 interpreter.
(4) vpudec: Use the VPU decoder to decode H.265 video files.
(5) waylandsink: select the display device and output it, and set the output resolution to 1920 x 1080.
If you want to know more about embedded applications, welcome to pay attention to Tronlong Chuanglong Technology~