Table of contents
report error
numpy.core._exceptions.MemoryError: Unable to allocate 1.45 GiB for an array with shape (13935, 13935) and data type float64
reason
The most important thing is that the computer memory is insufficient, because the amount of data to be processed is too large, the performance of the GPU is not enough, and there is a phenomenon of memory overflow
But in fact it is not a model file, but a parameter file. In the model file, the complete model is stored, while in the state file, only the parameters are stored. So collections.OrderedDict is just the value of the model.
solution
1. Modify float precision
In the code I am using the flaot 64 type. But in fact, such a large precision may not be required. At this time, float32, float16, etc. in numpy can be used, which can reduce a lot of memory requirements.
In short, choose according to your own needs, as long as it is enough, this alone can save several times the memory.
X = np.array(Xc, dtype=np.float16)
source code:
X = np.array(Xc)
2. The python library, Pandas and Numpy libraries are updated to 64-bit
The original data type of python is 32 bits, but it can only use a maximum of 2G memory, and a MemoryError will be reported if it exceeds 2G.
If your Python is 32-bit, then your pandas and Numpy can only be 32-bit, then when your memory usage exceeds 2G, the memory will be automatically terminated. And 64bit python has no such limitation, so it is recommended to use 64bit python.
The solution is: check how many bits of python you have installed, enter python in CMD, if it is 32 bits, then reinstall the 64-bit Python.
If your python is originally installed in 64-bit, you can use the following method!
3. Expand virtual memory
In the process of running the code, I found that the memory is actually only less than half used, but why is there a Memory error?
After entering Baidu, I found that the memory is limited, so I considered using the method of expanding the virtual memory.
Ways to expand virtual memory:
- Open the Control Panel ;
- Click on the System item;
- Click on Advanced System Settings ;
- Click the Settings button of the Performance module ;
- Select the Advanced panel , click Change Virtual Memory Module;
- Remember not to select "Automatically manage the paging file size of all drives" , then select a drive , that is, a disk, select Custom Size , and manually enter the initial size and maximum value, but not too large!
- After everything is set, remember to click " Settings " and then confirm, otherwise it will be invalid, and finally restart the computer .
5. Use python's gc module
The garbage collection mechanism of python is relatively lazy. Sometimes the variables in a for loop are not used up and zo will not be recycled. The next time it is re-initialized, the space will be re-opened. At this time, you can manually delthis variable, then manually import gc, and then manually gc.collect().
————————————————
Reference link: https://blog.csdn.net/muye_IT/article/details/125067419
[1]