Comparison of Java memory problems to solve large file read and file stream

Comparison of Java memory problems to solve large file read and file stream

The traditional way

Reading of papers generally read from memory, the official several ways, such as the BufferedReader, InputStream series and, also packaged as Apache commons IO Guava and provides the read files fast manner

Files.readLines(new File(path), Charsets.UTF_8); 
FileUtils.readLines(new File(path));

Which are utilized to achieve the BufferedReader or a subclass LineNumberReader to read additional Scanner is a scanning manner, the efficiency is very slow. In addition there are the problems, if it is a large file, which can not be a one-time memory storage, and does not require a one-time need to use all the data files.

Iterative read mode

General scenario is that we need to read a row of data into memory, then processed separately processed throwing it away, does not need to be placed in memory in its entirety, in this way like the iterator, a similar process may Scanner scene, line by line just read ,, the advantage is that you can read the data in a specific format, easy to deal with, but it is very slow.

inputStream = new FileInputStream(path); 
 sc = new Scanner(inputStream, UTF-8); 
 while (sc.hasNextLine()) {
  String line = sc.nextLine(); 
  // System.out.println(line); 
  } 

Another way is to use Apache Commons IO stream that there is a utility class FileUtils, inferior treatment on the Format Scanner, but the efficiency and BufferedReader almost, it is generally recommended this one.

LineIterator it = FileUtils.lineIterator(file, UTF-8); 
try {
 while (it.hasNext()) {
 String line = it.nextLine(); 
 // do something with line 
  } 
} finally {
 LineIterator.closeQuietly(it);
}

reference

https://www.breakyizhan.com/java/4018.html
https://www.jianshu.com/p/7a81f603fe1d
https://www.cnblogs.com/lovebread/archive/2009/11/23/1609122.html