Recent lazy, or overtime write something, or over time again these sort of things lost.
What to write? Write some relevant work with it! Because it engaged in the development of multimedia video-related work, and therefore often involves optimization Writer strategies to enhance things related aspects of card write performance. What say it? As video products tachograph class, recording may last several days, the more back card write speed will become slower, intuitive feeling is when you remove the video file playback, time about the future of video files Caton getting worse .
How to solve it? One solution from the hardware solution, for a good card! But this does not solve the problem once and for all, because the bibliographic record card write speed and fell off. Another solution to the software level, the speed is slower card after the card is formatted, but this program for users but not very friendly (some users may not know that before this is not convenient backup function, or file deletion) . There is also a program, but also to solve the problem from the software level, it is to optimize the write strategy card. Optimization card write strategy, there are some possible solutions, such as pre-allocated file, write buffering data to be written, and encoding the package decoupling, write (DirectIO). The following content describes the contents of the pre-allocated.
1. fallocate Introduction
linux man manual describes:
fallocate that is pre-allocated, English as preallocate. What does that mean? Also did not write data to a file, but the file has been allocated to the full amount of physical space to store data. After you create the file, then call this interface is pre-allocated a certain amount of space, you can follow a previous write data in this file.
Another point to note that support this interface requires a file system. Common TF card recording, and the file system type card is generally fat32, fat32 file system-related need to achieve in order to use this feature.
Again, this is a non-portable dedicated linux system calls for file space is allocated in advance to ensure that, after successful implementation, to ensure that the card write speed, but also guaranteed not to appear because of insufficient disk space to write failure.
2. The interface declaration
| Function prototype |
int fallocate(int fd, int mode, off_t offset, off_t len); |
| fd |
File handle |
| mode |
Creation mode |
| offset |
Offset |
| only |
File size |
Wherein, after creating the file and before writing data, which needs to be called pre-allocated interface, the second mode parameter is generally set to 1, the third parameter is set to 0, the fourth parameter value is pre-assigned a desired fill.
3. Scenarios and objectives
Scenario: sustained write card scene, such as driving recorder, camera movement.
Goal: reduce disk fragmentation, increase card write speed.
Other notes: bottlenecks video equipment is often written card, because any time a video file recorded. And higher sustained write speed card requirements, because video equipment duty cycle may be based day as a unit, not only asked to write a normal video card has just started, and requires a few days to write a work card rate can not fall too much. As the amount of data written per second, depending on the encoder output bit rate video and several paths may be, for single 1080p recording, video bit rate is set to 10Mbps, then the speed of the card to ensure that at least 2MB / s, there is not including the write log and the video camera used.
Although the current time node (end of 2019), the market cards are C10 (10MB / s) and above, but if you write policy is unreasonable or too many cards fragmented files, write speed may be low. A very common example, a copy of the file to the video card speed T, is much larger than the size of the copy of the same source packet. As another example, a card with a newly formatted within the T already exists a number T card file (Brand card, capacity, speed, etc. are the same), the same size copy files, just that card speed formatted faster.
4. The principle
TF card (TransCard) and SSD (SolidStateDisk) as a common storage devices, internal composition is very similar, it is mainly composed of controler and nand flash components. For any storage device, we are most concerned with three parameters: capacity, read / write speed, life.
"Capacity" do not use this parameter, the "speed reading" is not introduced, mainly referring to the next "life" and "write speed" These two parameters below. After introducing these two parameters, introduced again pre-allocated.
4.1 life Related:
Life mainly determined by the storage medium, i.e., such media may nand flash erasing times, nand flash media type has experienced slc, mlc, tlc, qlc (currently on the market is also less) in several stages, the capacity per unit area It is also growing, as reflected in the type of media storage density. Small TF card, this time at the end of 2019 the current node, there has emerged a 512GB capacity TF card, library books store multiple text messages should be no pressure! However, everything both advantages and disadvantages, as the lift capacity, rewritable memory cells within the minimum number of TF less and less.
SLC (SingleLevelCell) appear first, Endurance more than 10 million times; MLC (MultiLevelCell) the subsequent emergence of about 3000-10000 times Endurance, the current mainstream Endurance TLC at 500-1000 times. On a casual look at the East SSD 500GB capacity lexar a section of the following parameters:
Have seen the flash type as TLC, there TBW = 250T this parameter, what is this and how come it?
TBW, i.e. TeraBytesWritten, the amount of data written in units of TB. The value thus calculated: Total Capacity * write times, i.e., 500GB * 500 = 250TB. Where 500 represents the average number of write cycles is 500, according to a flash estimate of the type of TLC. With general enterprise-class sdd, prices are a lot higher civilian, such as compilation / database server, the same capacity TBW value usually PBW (= 1024TBW) as a unit, less the pursuit of read and write speeds, but a very high value life and reliability, after all the data is priceless.
4.2 speed Related:
Write speed is a relatively iffy thing, integrated caused by many factors, such as flash memory type, main control algorithm (firmware wear leveling algorithms), file system write strategy, the degree of fragmentation card, the card's file system type and block size, internal whether with Cache as well as its size, and many other factors.
However, for finalized a card, we need to find some way to improve the write speed of the card. One way is to pre-allocation --fallocate.
Then after the first presentation file storage-related content, again introduced the role of pre-assigned interface.
For fat32 file system, a file storage device, which mainly comprises two parts: part of attribute information metadata (creation / modification time, file name, file size, etc.), the other part is the actual data content. Commonly used fdatasync operation will only force the real content refresh data to the storage device, and will fsync two parts are flushed to the device. For true that part of the data content, there is a linked list to manage SectorId content of each block is located, namely in the form of sector list to complete presentation of data content. Thus, the physical memory address of a file could be a single piece of a continuous region where the sector area, may be distributed in a plurality of discrete physical area.
Fragmentation and memory is fragmented storage device is very similar to that of a file you wish to take advantage of continuous physical storage space as possible to store the data, but because the card is in a highly fragmented state when actually finished write to the file, the file in physical space is "fragmented" in. Even a newly formatted card, when two threads simultaneously writing two different files, respectively, in the physical space (inside the contiguous physical block or sector), these files may be in an intertwined state (interlaced), English as interleave. Audio colleagues have done development may also recall alsa-lib when you turn on the device parameter configuration, data collection for the two-channel pcm, there interleave and non-interleave configuration, this option determines the left and right channels pcm data in a period how the arrangement, a plurality of files similar comparison, stored on the card, the physical block is the mean.
Seek imagine a scenario written document, using normal operating procedures fopen-fwrite-fclose, and just write all the way, when the data prior to each kernel cache in the brush card, you need to go to the site (similar to when writing to disk ) which is available in physical sector, when found in a block in a sector are available, but other sector is occupied by other files, then the next strategy is to copy-modify-write, that is a "write amplification" (WriteAmplification).
Why is there this situation, you need to understand the basic components of the flash memory: page page (also known as sector, size of 4KB) -> block block (typically 64 or 128 page consisting of a block) -> surface plane (multiple block composition) -> die (Plane is a die) -> Flash sheet (composed of a plurality of die) -> SSD or TF (sheet composed of multiple pieces flash).
Write the amplification process described below: first complete copy of the entire data in the block to ddr, then the data in a sector is modified to write desired data in the erased block ssd content, and then the whole of ddr has the edited data is written to the block position in ssd. Why do this? Because writing is carried out by block as the basic unit. So write single data, involving a number of basic operations, not only slow down the write speed, and reduced life expectancy. However, after a pre-assigned, in advance of a file is divided into "spheres of influence", the calibration is already occupied certain positions, you can reduce write amplification and subsequent process to find available space.
4.3 Pre-allocation principles:
Following the presentation of the content file storage structure for pre-allocation functionality we have rough guess! fallocate this interface, the purpose of which is to be achieved, the content data is not yet written to the device before, well in advance for a number of file allocation size space, and the space that is physically continuous as possible, thus reducing the writing after amplification frequency of occurrence, and do not need to look for available space in the writing process, but when writing data is insufficient disk space problem does not arise!
5. other problems
Use a pre-assigned biggest problem is - disk space utilization is not high! How does this start? File you just created yet written data, we have first to set the size of the file and the file takes up physical space of fixed size, but generally may not be written so large amount of data on the size fclose the file, then the file inside unused space can not be written to take advantage of the other documents. File a pre-allocated 100MB, even if only write 1MB closed, then there is 99MB of wasted space. However, the use of pre-assigned to the tachograph products is a superior choice, because the file switching timing of the switching is, if the encoder output bit rate is relatively stable, the final file size can be estimated, the size of pre-allocated to stay some margin on it.