Problem Description
Just after training a batch, the following situation occurs:
solution
The most likely situation where nan occurs is that the learning rate is set too large. Set lr to 0.0014, which is of the order of 1e-3. The following situation occurred:
Hey, it does have an effect, but nan still appears. Then continue to reduce it to 0.00014, and then increase the batch size to 128 (increasing the batch size can enable the network to better learn the distribution of the data. Of course, this also It needs to be adjusted according to the size of the video memory), and nan does not appear again.