This article is shared from the China Mobile OneOS WeChat public account " The Two-Dimensional World in Wan Couple! ", author: little M brother.
BAD APPLE was originally a song in Touhou games, and then a MAD was added. Because MAD is very stunning and gorgeous, BAD APPLE is loved by everyone.
In the circle of programmers, there is a rumor: "Where there is a screen, there should be a bad apple". As a result, whether it is an oscilloscope, a dot matrix screen, or a liquid crystal display, they all perform videos of brushing bad apples.
Since there is also a screen on the Wancouple development board, then it should satisfy the curse of "where there is a screen, there should be a bad apple".
// Basic knowledge
What is a video?
Video generally refers to various technologies that capture, record, process, store, transmit and reproduce a series of static images in the form of electrical signals.
When the continuous image changes exceed 24 frames per second, according to the principle of persistence of vision, the human eye cannot distinguish a single static image;
It appears to be a smooth continuous visual effect, such a continuous picture is called a video.
What is the principle of persistence of vision?
Persistence of vision is the phenomenon that the vision produced by light on the retina remains in the neural network of the brain for about 0.1-0.4 seconds after the effect of light on the retina.
The reason is the response speed of the optic nerve.
Key point: Play pictures in quick succession. Due to the principle of persistence of vision, it gives the impression that you are playing a video.
Implementation plan
According to the above basic knowledge, it is first necessary to realize the display of pictures on the Wanco development board.
To scan bad apple videos, you will definitely need a lot of pictures, so you need to solve the problem of where the pictures come from and where the pictures exist.
Let's first calculate the data size of a picture. The screen on the Wandu development board is a 1.3-inch 240x240 pixel TFT screen, and each pixel is 16 bits, that is, 2 bytes. Then the data size of a picture is: 240*240*2= 115200, which is 112.5kB.
Play at the speed of 5 pictures per second. In this case, you can feel stuck. It is best to use 10 pictures per second to achieve smoother results. We calculate according to 5 pictures in 1 second, then the storage space required to play a video for 1 second is: 112.5*5 = 562.5kB.
The microprocessor chip on the Wancouple development board is STM32L475VGT6, 128kB RAM/1024kB Flash. In addition to the Flash space occupied by the system itself, there should be six or seven hundred kB Flash space left. According to the storage space requirements calculated above, It can also save video pictures less than 2 seconds. If the picture is stored in the Flash inside the chip, this solution may not work.
The internal Flash cannot be stored, and there is 8MB of external SPI Flash on the Wandu development board. Calculate how long the video can be stored: 8*1024/562.5=14.56 seconds. This ten-second video cannot meet our lofty ideals.
OneOS has a very efficient driver framework, which brings the benefits of stability and ease of use, but does not affect performance at all.
Serial port driver layering
This brings another realization idea. The picture data is placed on the computer, and the picture data is sent to the Wanco development board in real time at a high-speed baud rate of 921600 through the serial port. The Wanco development board receives the data and displays it in real time. As long as the speed of receiving and processing data is fast enough and the speed of displaying pictures is fast enough, the video can be played. Because it is displayed in real time online, no matter how long the video is, it can be played, even if it is a Captain America movie, no problem.
Conversion of video files
The implementation plan has been roughly determined above, and now let's implement the following according to the established ideas.
First, go to the Internet to find bad apple videos, preferably high-definition videos, and download them locally, because we need to convert the videos into pictures first.
B station address: https://www.bilibili.com/video/BV17K411s7pt?from=search&seid=9747428269435705565
After downloading the bad apple video, you also need KMPlayer, the playback software. KMPlayer can continuously take screenshots of the video into multiple pictures. Right-click on the video playback interface, and select "Capture" -> "Screen: Advanced Capture" in the pop-up list box, as shown in the following figure:
The Advanced Capture dialog pops up:
Step 1: Enter the save path
Step 2: The name of the picture, which is automatically named after a number, and the prefix can be input by yourself. Any aaa or 000 is fine. The following digits, if the number of pictures is more than 1,000, is configured as 4 digits, automatically starting from 0000, and then 0001, 0002 are accumulated in this order.
Step 3: Image format, choose "Bitmap (Fast)"
Step 4: Number of captures, choose "Continuous", capture frames, choose "10 frames in 1 second"
Step 5: Capture size, according to the screen Pixels are selected. The screen on our Wanco development board is 240x240 pixels, then we can set it to 480x480, or 240x240.
The last step: first click the KMPlayer play button, and then immediately click the "Start" button of the capture dialog box to set the The entire process of video playback is captured as pictures.
Capture results
So far, we have converted bad apple videos into more than 1500 bmp pictures.
The BMP bitmap has a certain format. If it is directly sent to the Wandu development board for display through the serial port, the Wandu development board is required to decode the BMP image. In order to achieve a smooth playback effect, the need for decoding is not considered first.
BMP image data is divided into four parts:
1. The file header (bmp file header) provides information such as the format and size of the file.
2. Bitmap information header (bitmap information), which provides information such as the size of the image data, the number of bit planes, the compression method, and the color index.
3. Color palette, optional, if an index is used to represent an image, the palette is a mapping table of the index and its corresponding color.
4. Bitmap data is the real image data.
Regarding bitmap data, the most common one is 24-bit map. The so-called 24-bit map means that the color information of a pixel is represented by 24 bits. That is to say, for the three primary colors BRG, each color is represented by bytes (8 ) bits to indicate. In addition to 24-bit images, there are 1-bit (monochrome), 2-bit (4-color, CGA), 4-bit (16-color, VGA), 8-bit (256-color), 16-bit (high color), 24-bit (true color) color) and 32-bit etc.
It should also be noted here that the image data of the 24-bit image is arranged in the order of B (blue), G (green), and R (red), and it starts from the lower left corner, and the image data is arranged line by line until the upper right corner. end.
After studying the format of BMP pictures clearly, it is necessary to consider how to send more than 1,500 pictures to the Wanco development board for real-time display. To send data to the Wanco development board through the computer, the first thing that everyone thinks of is the serial port, yes, the UART.
The serial port is the most basic communication interface in our embedded system. Printing debug logs and completing simple data communication can be realized through the serial port.
OneOS Device Framework
OneOS has a complete set of device frameworks and rich driver frameworks to facilitate the management and use of various peripherals.
Device Management Architecture
The OneOS R&D team has designed and developed various driver frameworks commonly used in embedded systems, and has also adapted to major mainstream MCU chips. For details, please refer to the drivers directory in the OneOS source code. The driver types are shown in the following figure:
Rich driver types
The upper-layer application directly calls standard API interfaces such as os_device_open and os_device_read to operate and read and write to the underlying device. The calling process inside the system is as follows:
Device driver call process
Taking the STM32L475VGT6 chip on our Wanco development board as an example, we will talk about the serial driver implementation process for ST chips inside OneOS in detail. Since ST already has a complete hal driver library, the serial driver of OneOS is designed and implemented based on the official hal library, and a unified serial driver framework has been extracted, which is compatible with all STM32 chips. The detailed process of serial driver reading serial port data is as follows:
OneOS serial device read and write
In addition to the debugging serial port, one serial port is connected to the Wi-Fi module, the other serial port is connected to the mini-PCIe, and there is another serial port on the reserved external interface. By looking at the schematic diagram, it is found that the serial port leading to the external interface is PD5 and PD6 of STM32L475VGT6, which is UART2.
To read serial port data, directly call the standard interface os_device_read of the system:
os_device_read specifies which serial port device to read data from, saves the data in the memory space whose starting address is buffer, and the size of the data expected to be received this time. The os_device_read interface returns the actual received data length.
Of course, you need to configure and open the corresponding serial port before reading the data. os_device_control is used to configure the baud rate, stop bit, callback function, etc., and os_device_open is used to open the serial port device, with various modes of opening the serial port.
swipe bad apple
With the above preparation knowledge, you can start to brush bad apple videos.
First, find the projects/stm32l475-cmcc-oneos project in the OneOS source code directory, and first open the UART2 serial port through the STM32CubeMX tool.
Then use the OneOS-Cube tool for configuration, and change the receive buffer of the serial driver to 1024. In the scenario of high-speed data transmission and reception, the receive buffer must be increased, otherwise data may be lost.
After the configuration is complete, use the scons –ide=mdk5 command to regenerate the project.
Then start the application code development, we create two tasks, one task is responsible for reading data from the serial port, and one task is responsible for displaying the data to the screen.
/**
***********************************************************************************************************************
* Copyright (c) 2020, China Mobile Communications Group Co.,Ltd.
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*
* @file bad apple.c
*
* @brief User application entry
*
* @revision
* Date Author Notes
* 2020-02-20 OneOS Team First Version
***********************************************************************************************************************
*/
#include <board.h>
#include <drv_cfg.h>
static void data_recv_task(void *parameter)
{
int ret;
os_device_t *uart;
os_uint32_t size;
os_size_t rx_cnt;
unsigned int length;
unsigned char step, last_step;
os_mb_init(&urx_mb, "urx_mb", mb_pool, sizeof(mb_pool) / sizeof(mb_pool[0]), OS_IPC_FLAG_FIFO);
uart = os_device_find("uart2");
OS_ASSERT(uart);
struct os_device_cb_info cb_info =
{
.type = OS_DEVICE_CB_TYPE_RX,
.cb = urx_done,
};
os_device_control(uart, IOC_SET_CB, &cb_info);
os_device_open(uart, OS_SERIAL_FLAG_RX_NOPOLL | OS_SERIAL_FLAG_TX_NOPOLL | OS_DEVICE_OFLAG_RDWR);
struct serial_configure config = OS_SERIAL_CONFIG_DEFAULT;
config.baud_rate = BAUD_RATE_921600;
ret = os_device_control(uart, OS_DEVICE_CTRL_CONFIG, &config);
if (ret != 0)
{
os_kprintf("serial control fail %d\r\n", ret);
goto failed;
}
while (1) {
length = 0;
last_step = 0;
do {
rx_cnt = os_device_read(uart, 0, pic_buffer+length, RX_BUFFER_SIZE-length);
length += rx_cnt;
if (rx_cnt == 0) {
os_mb_recv(&urx_mb, &size, OS_IPC_WAITING_FOREVER);
}
}while(length < RX_BUFFER_SIZE);
os_sem_post(&disp_sem);
}
failed:
os_device_close(uart);
}
static void disp_task(void *parameter)
{
os_uint8_t *rgb565;
unsigned int i = 0, j = 0;
unsigned int step = 0;
unsigned char data;
unsigned short *prgb;
/* show CMCC logo */
lcd_show_image(20, 50, 200, 61, gImage_cmcc);
lcd_show_image(20, 150, 200, 52, gImage_oneos);
rgb565 = os_malloc(240*120*2);
OS_ASSERT(rgb565);
while (1) {
prgb = (unsigned short*)rgb565;
os_sem_wait(&disp_sem, OS_IPC_WAITING_FOREVER);
for (i = 0; i < 3600; i++) {
data = pic_buffer[i];
for (j = 0; j < 8; j++) {/* 每一位表示一个像素点 */
if (data & (1 << j)) {
*prgb++ = WHITE;//white
}
else {
*prgb++ = BLACK;
}
}
}
lcd_show_image(0, 0, 240, 120, rgb565);
prgb = (unsigned short*)rgb565;
for (i = 3600; i < RX_BUFFER_SIZE; i++) {
data = pic_buffer[i];
for (j = 0; j < 8; j++) {
if (data & (1 << j)) {
*prgb++ = WHITE;//white
}
else {
*prgb++ = BLACK;
}
}
}
lcd_show_image(0, 120, 240, 120, rgb565);
}
}The task responsible for display is waiting for a semaphore at the beginning. After receiving the semaphore, it directly calls the lcd_show_image interface to display the data in the buffer on the screen.
According to the above introduction, BMP pictures cannot be directly sent to the development board for display. We also made a conversion tool to convert BMP images into data files that can be displayed directly on the development board.
Send a picture first to test whether the Wanco development board can display the picture correctly. If it can be displayed, use the serial port tool sscom5.13.1 to continuously send all the picture data to the Wanco development board for display. In order to achieve a smooth effect, the serial port baud rate is set to 921600 , and the sending file setting selects continuous sending.
Serial tool settings
Now the pictures can be displayed normally, but the continuous pictures are still not as expected, and the freeze is serious.
Improve performance
Since the data of a picture is 112.5kB, many pictures are displayed continuously, it feels like a slideshow, and the freeze is very serious.
We found that the video of bad apple is black and white. According to this feature, we can greatly reduce the effective data of a picture. A pixel is either black or white, and it only needs to be represented by 0 and 1. Before, a pixel needed 2 bytes, and now a pixel only needs 1 bit, and the amount of data is reduced by 16 times.
In fact, this is the principle of binarization. After re-modifying and optimizing our conversion tool, after binarizing the BMP image, each pixel is only represented by 1 bit, and the data of 8 pixels is compressed into 1 byte. The amount of compressed data is very small, and then the compressed data is sent to the Wanco development board through the serial port tool.
Re-optimize the code of the display task. Since the received data is compressed data, the data must be restored before display. After all, the data written to the screen still needs 2 bytes per pixel. Restore one byte of data to 8 pixel data, that is, 16 bytes, and then write the restored data to the screen of the Wanco development board.
Click to jump to the video link
In general, the process of brushing bad apple videos on the Wandu development board is as follows:
OneOS is a lightweight operating system launched by China Mobile for the Internet of Things. It features tailorable, cross-platform, low power consumption, and high security. It supports mainstream CPUs such as ARM Cortex-M/R/A, MIPS, and RISC-V. The architecture is compatible with POSIX, CMSIS and other standard interfaces, supports Micropython language development, and provides graphical development tools, which can effectively improve development efficiency and reduce development costs, helping customers develop stable, reliable, safe and easy-to-use IoT applications. Official website address: https://os.iot.10086.cn/
OneOS software address: http://www.oschina.net/p/cmcc-oneos
OneOS project address: https://gitee.com/cmcc-oneos/ OneOS
OneOS technical exchange group: 158631242