I流
principle
Classification
byte stream and character stream
Node Stream vs Wrapper Stream
- Java IO Detailed Explanation (5)------Packaging Flow- YSOcean - Blog Garden (cnblogs.com)
- JAVA I/O stream Character stream and byte stream, node stream and processing stream (wrapper stream, filter stream), buffer stream_filter stream and buffer stream, relationship between byte stream_X-Dragon Misty Rain Ren Pingsheng's Blog-CSDN blog
character stream
import org.junit.jupiter.api.Test;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
public class CopyBytes {
public static void main(String[] args) throws IOException {
}
//单个字节的读取
@Test
public void test1() throws IOException{
FileInputStream in = null;
FileOutputStream out = null;
String path="src\\single\\in.txt";
String dpath="src\\single\\out.txt";
try {
in = new FileInputStream(path);
//String,boolean true追加；false覆盖，默认
out = new FileOutputStream(dpath,true);
int datacode;
//read()一次读取一个字节,返回读取字节的ASCII码，返回-1，表示读取完毕
while ((datacode = in.read()) != -1) {
System.out.println(datacode);//输出ASCII码 例如 a 是97 ；一个汉字是3个字节
System.out.println((char)datacode);
out.write(datacode);
}
} finally {
if (in != null) {
in.close();
}
if (out != null) {
out.close();
}
}
}
@Test
public void test2() throws IOException{
FileInputStream in = null;
FileOutputStream out = null;
String path="src\\single\\in.txt";
String dpath="src\\single\\out.txt";
try {
in = new FileInputStream(path);
out = new FileOutputStream(dpath);
//一次读取多个字节，返回读取字节的长度
byte[] buf=new byte[3];
int datalen=0;//实际读取的长度，读取完毕返回-1
while((datalen=in.read(buf))!=-1){
System.out.println(datalen);
System.out.println(new String(buf,0,datalen));
//System.out.println(new String(buf));//这样会导致读取异常，因为当最后一次读取不够时，上一次的读取并没有清空
out.write(buf,0,datalen);
}
} finally {
if (in != null) {
in.close();
}
if (out != null) {
out.close();
}
}
}
}
byte stream
package file;
import org.junit.jupiter.api.Test;
import java.io.*;
public class FileReader_ {
public static void main(String[] args) {
}
//单个字符
@Test
public void readFile01() {
String filePath = "src\\file\\in.txt";
String dPath="src\\file\\out.txt";
FileReader fileReader = null;
FileWriter fileWriter=null;
int datacode = 0;
//1. 创建 FileReader 对象
try {
fileReader = new FileReader(filePath);
fileWriter=new FileWriter(dPath);
//循环读取 使用 read, 单个字符读取
while ((datacode = fileReader.read()) != -1) {
System.out.print(datacode);
System.out.println((char) datacode);
fileWriter.write(datacode);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (fileReader != null) {
fileReader.close();
}
if(fileWriter!=null){
//这里一定要关闭，才会写入 或者flush
//close 等价于 flush + close
//底层还是字节流
fileWriter.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
@Test
public void readFile02() {
String filePath = "src\\file\\in.txt";
String dPath="src\\file\\out.txt";
FileReader fileReader = null;
FileWriter fileWriter=null;
int readLen = 0;
char[] buf = new char[8];
//1. 创建 FileReader 对象
try {
fileReader = new FileReader(filePath);
fileWriter=new FileWriter(dPath,true);
//循环读取 使用 read(buf), 返回的是实际读取到的字符数
//如果返回-1, 说明到文件结束
while ((readLen = fileReader.read(buf)) != -1) {
System.out.print(new String(buf, 0, readLen));
fileWriter.write(buf,0,readLen);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (fileReader != null) {
fileReader.close();
}
if(fileWriter!=null){
fileWriter.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
buffered stream
package file;
import java.io.*;
/**
* @author 韩顺平
* @version 1.0
* 演示 bufferedReader 使用
*/
public class BufferedReader_ {
public static void main(String[] args) {
String filePath = "src\\file\\in.txt";
String dPath="src\\file\\out.txt";
//创建 bufferedReader
BufferedReader bufferedReader =null;
BufferedWriter bufferedWriter=null;
try {
bufferedReader=new BufferedReader(new FileReader(filePath));//也可以传入其他的Reader
bufferedWriter=new BufferedWriter(new FileWriter(dPath));
//读取
String line; //按行读取, 效率高
//说明
//1. bufferedReader.readLine() 是按行读取文件
//2. 当返回 null 时，表示文件读取完毕
while ((line = bufferedReader.readLine()) != null) {
System.out.println(line);
bufferedWriter.write(line);
bufferedWriter.newLine();
}
} catch (Exception e) {
e.printStackTrace();
}finally {
//关闭流, 这里注意，只需要关闭 BufferedReader ，因为底层会自动的去关闭 节点流
try {
if(bufferedReader!=null) {
bufferedReader.close();
}
if(bufferedWriter!=null){
bufferedWriter.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
benefit
- It is more convenient to use and shields the underlying details
- Use buffered reads, more efficient
- more powerful
large file
- Before the front-end uploads the file, first query the data and compare the md5 value; if it exists, it will prompt the second transmission
- If it does not exist, the front-end uploads in fragments. Before uploading, first check whether the fragment exists. If it exists, it does not need to upload to realize the breakpoint resume
- Backend storage sharding
- After the file is uploaded, merge the fragments into a complete file, and save the file record to the database
distributed file system
A computer cannot store a large number of files. Several computers are organized through the network to store a large number of files and receive requests from a large number of users. These organized computers communicate through the network.
benefit:
1. The file system processing capability of one computer can be extended to multiple computers for simultaneous processing.
2. If one computer is down, another copy computer will provide data.
3. Each computer can be placed in a different region, so that users can visit nearby and improve the access speed.
smaller
Official website: https://min.io
Chinese: https://www.minio.org.cn/, http://docs.minio.org.cn/docs/
- Lightweight, Simple, Powerful
- It is suitable for storing large-capacity unstructured data, such as pictures, videos, log files, backup data, and container/virtual machine images. Support size from several k to 5T
- Good read and write performance
- The MinIO cluster adopts a decentralized shared architecture, and each node is in a peer-to-peer relationship. MinIO can be load-balanced and accessed through Nginx.
- Redundant storage, high security
- What are the benefits of decentralization?
In the field of big data, the usual design concepts are centerless and distributed. The Minio distributed mode can help you build a highly available object storage service, and you can use these storage devices regardless of their real physical location.
It combines multiple hard disks distributed on different servers into an object storage service. Since hard disks are distributed on different nodes, distributed Minio avoids a single point of failure. As shown below:
Minio uses erasure code technology to protect data. It is a mathematical algorithm for recovering lost and damaged data. It stores data blocks redundantly on the disks of each node. All available disks form a set. The figure consists of 8 hard drives. When a file is uploaded, the file will be stored in blocks through the erasure code algorithm calculation. In addition to dividing the file itself into 4 data blocks, 4 check blocks will be generated. The data block and The check blocks will be stored scattered on these 8 hard disks.
The advantage of using erasure codes is that even if half of the hard drives (N/2) are lost, data can still be recovered. For example, if there are less than 4 hard disks damaged in the above collection, data recovery can still be guaranteed without affecting uploading and downloading. If more than half of the hard disks are damaged, it cannot be recovered.