OS large file transfer

Blocking IO:

• Kernel buffer (PageCache) is copied to the user buffer

Asynchronous I/O does not use PageCache

• Direct I/O, no PageCache
• Cached I/O, using PageCache
• For disk, asynchronous I/O can only use direct I/O

For transferring large files, use asynchronous I/O + direct I/O instead of zero copy

Direct I/O application scenarios:

• The application has implemented disk caching, such as: MySQL enables direct I/O, default: not enabled
• When transferring large files, it is difficult for large files to hit the PageCache, which also leads to the inability to use the cache for hot data and increases performance overhead. Therefore, direct I/O should be used at this time.

Direct I/O bypasses PageCache optimizations:

• Cache multiple I/O requests in PageCache, combine them into large I/O requests, and send them to disk to reduce disk addressing
• Read-ahead subsequent I/O requests are placed in PageCache, reducing disk operations

According to file size, transfer file method:

• Transfer large files, using asynchronous I/O + direct I/O
• Transfer small files, using zero-copy technology