RTMP protocol video platform EasyDSS realizes super large file copy function through Golang. File backup method

In the development process of EasyDSS, because some users have mistakenly deleted during use, we need to back up the videos uploaded by users to prevent users from deleting by mistake. This process requires copying files.

Commonly used file copy functions are directly implemented using the io.Copy() method provided by the official go library.

nBytes, err := io.Copy(destination, source)

The above method is suitable for the copying of small files. For very large files, such as 500MB or even more than 1GB files, there will be problems.

The basic logic of the io.Copy() method is to read all the source files into the memory at one time, and then write the data from the memory to the new file. For larger files, such as reading a 2GB video file, 2GB of memory will be consumed at one time. If the computer memory itself is small, the program will provide insufficient memory. Therefore, this method should not be called directly for larger files.

For copy operations of larger files, small data, such as 1024 bytes, should be read every time and written to a new file. Then read the file again and write to the new file until all the data is written to the new file.

func Copy(src, dst string) (int64, error) {
   sourceFileStat, err := os.Stat(src)
   if err != nil {
      return 0, err
   }

   if !sourceFileStat.Mode().IsRegular() {
      return 0, fmt.Errorf("%s is not a regular file", src)
   }

   source, err := os.Open(src)
   if err != nil {
      return 0, err
   }
   defer source.Close()

   destination, err := os.Create(dst)
   if err != nil {
      return 0, err
   }
   defer destination.Close()
   //nBytes, err := io.Copy(destination, source)

   nBytes := int64(0)
   buf := make([]byte, 4096)
   for {
      n, err := source.Read(buf)
      if err != nil && err != io.EOF {
         return 0, err
      }
      if n == 0 {
         break
      }

      if _, err := destination.Write(buf[:n]); err != nil {
         return 0, err
      }

      // 更新写入的数量
      nBytes = nBytes + int64(n)
   }

   return nBytes, err
}

When reading a file, the EOF mark is encountered, which means that the file has been read. The core code in the above method is as follows.

nBytes := int64(0)
buf := make([]byte, 4096)
for {
   n, err := source.Read(buf)
   if err != nil && err != io.EOF {
      return 0, err
   }
   if n == 0 {
      break
   }

   if _, err := destination.Write(buf[:n]); err != nil {
      return 0, err
   }

   // 更新写入的数量
   nBytes = nBytes + int64(n)
}

EasyDSS is an Internet video live/on-demand platform developed by Tsingsee Video. It can connect to RTMP protocol cameras and equipment, and generate a push stream address for video monitoring. Users who use EasyDSS know that we provide a wealth of API interfaces, developers can freely carry out secondary development, API interface details: http://demo.easydss.com:10080/apidoc.