[Python] numpy read and write files

official website

https://numpy.org/doc/stable/reference/routines.io.html

text file

read text file

illustrate

1. np.loadtxt()

• File format: plain text file, each row of data can be of different data types.
• Parameter setting: common parameters include delimiterspecifying the delimiter, dtypespecifying the data type, skiprowsspecifying the number of skipped rows, etc.
• Applicable scenarios: suitable for reading simple plain text data, such as a set of experimental data.

2. np.genfromtxt()

• File format: plain text file, can handle missing data and irregular data.
• Parameter settings: common parameters include delimiterspecifying the delimiter, dtypespecifying the data type, skip_headerspecifying the number of skipped header lines, missing_valuesspecifying missing values, etc.
• Applicable scenarios: suitable for reading irregular plain text data, such as data containing missing values ​​or blank rows.

3. np.recfromtxt()

• File format: plain text file, can handle missing data and irregular data.
• Parameter settings: Common parameters are np.genfromtxt()similar to .
• Applicable scenarios: suitable for reading structured data with column names, such as experimental data tables.

4. np.recfromcsv()

• File format: CSV file, each row of data is usually the same data type.
• Parameter settings: common parameters and delimiterspecified delimiter, dtypespecified data type, skip_headerspecified number of skipped header lines, etc.
• Applicable scenarios: suitable for reading CSV data with column names, such as experimental data tables.

the difference

• np.loadtxt(): Used to read data from a text file and store it as a NumPy array. By default, this function assumes that the data is numeric, separated by spaces, and has no header row. Additionally, you can specify the data type, skip specific rows or columns, and other options.
• np.genfromtxt(): loadtxt()Similar , but more flexible. It can handle missing values, different delimiters, different data types, and different text file formats (such as CSV, TSV, etc.). Additionally, it can handle data files with header and comment lines.
• np.recfromtxt(): genfromtxt()Similar to , but creates a structured array where each column can have a different data type and each column can be identified by a header.
• np.recfromcsv(): recfromtxt()Similar to , but it only handles the CSV file format and uses commas as delimiters by default.

write to text file

np.savetxt()

• File format: plain text file, each row of data can be of different data types.
• Parameter setting: common parameters include delimiterspecifying separator, fmtspecifying output format, etc.
• Applicable scenarios: suitable for saving a set of experimental data into plain text files.

binary file

read binary file

illustrate

1. np.load()

• File format: NumPy binary.
• Parameter settings: Common parameters include allow_picklespecifying whether to allow loading of pickled objects, mmap_modespecifying the memory mapping mode, etc.
• Applicable scenarios: suitable for loading binary files that save NumPy arrays.

2. np.fromfile()

• File Format: Any binary file.
• Parameter setting: common parameters include dtypespecifying the data type, countspecifying the number of elements to be read, etc.
• Applicable scenarios: suitable for reading large binary files, such as image data.

the difference

• np.load(): Used to load data from NumPy-specific binaries (.npy, .npz). These files np.save(), np.savez()created by the or functions, can save NumPy arrays on disk along with their metadata (such as data type, shape, etc.). np.load()Functions can easily read these files and restore them to raw NumPy arrays.
• np.fromfile(): Used to load data from a binary file and store it as a NumPy array. This function assumes that the data in the file is contiguous and encoded according to the specified data type. Therefore, you need to specify the data type and the shape of the array in order to correctly interpret the read data as a NumPy array.

In summary, np.load()both np.fromfile()and can be used to load data from files, but np.load()is specifically for loading NumPy binary files, while np.fromfile()for loading binary files, you need to manually specify the data type and shape.

Write NumPy binary files

illustrate

1. np.save()

• File format: NumPy binary.
• Parameter settings: Common parameters include allow_picklespecifying whether to allow pickle objects, fix_importsspecifying whether to fix differences between Python 2 and Python 3, etc.
• Applicable scenarios: Suitable for saving NumPy arrays into binary files.

2. np.savez()

• File format: Compressed NumPy binaries.
• Parameter setting: common parameters include allow_pickle, fix_importsand and **arraysso on .
• Applicable scenarios: Suitable for saving multiple NumPy arrays into a compressed binary file.

3. np.savez_compressed()

• File format: Compressed NumPy binaries.
• Parameter settings: Common parameters are np.savez()similar to .
• Applicable scenarios: It is suitable for saving multiple NumPy arrays into a highly compressed binary file to save storage space.

the difference

• np.save(): Save a single NumPy array to a binary file (.npy) on disk. This file only contains the raw data of the array and does not contain any metadata (such as data type, shape, etc.). Therefore, when you load this file, you need to know the metadata of the original array in order to interpret the data correctly.
• np.savez(): Save multiple NumPy arrays to a compressed binary file (.npz) on disk. This file contains arrays and their metadata. This file can access each array by index, just like a dictionary. The argument to this function is a series of NumPy arrays, each of which needs to be assigned a name so that it can be referenced when loading the file.
• np.savez_compressed(): np.savez()Similar to , but saves with a higher compression ratio, resulting in a smaller file size. This function takes the np.savez()same , and the resulting filename also ends with the .npz file extension.