one epoch = one forward pass and one backward pass of all the training examples
batch size = the number of training examples in one forward/backward pass. The higher the batch size, the more memory space you’ll need.
number of iterations = number of passes, each pass using [batch size] number of examples. To be clear, one pass = one forward pass + one backward pass (we do not count the forward pass and backward pass as two different passes).
Example: if you have 1000 training examples, and your batch size is 500, then it will take 2 iterations to complete 1 epoch.
重点是:
如果你有1000个训练集,batch_size=500,那么需要两次迭代才能完成一次epoch
In batch processing, the gradient is evaluated for several different input/output values, with each observation yielding a different vector. We then average together the gradient vectors over all observations in the batch and take a single step in the resulting direction.
The benefit of this is that the gradient estimate using a single point only may be very noisy. By averaging together the gradient over many samples we obtain a less noisy estimate.
I’ll also point out that having too large of a batch can also be a bad thing. Sometimes the noise in the gradient direction can kick you out of local minima and help you get to a better solution. This is problem specific, though, so when training a NN, you should experiment with many different batch sizes. This paper goes into further detail about this
batch_size
The documentation for Keras about batch size can be found under the fit function in the Models (functional API) page
这段话的重点是:
从反向传播的误差角度进行评估,如果batch_size很小的话,那么Loss的计算就很容易受到噪声的影响,因此需要手动设置.
batch_size: Integer or None. Number of samples per gradient update.
If unspecified, batch_size will default to 32.
If you have a small dataset, it would be best to make the batch size equal to the size of the training data. First try with a small batch then increase to save time. As itdxer mentioned, there’s a tradeoff between accuracy and speed.
重点是:
在你的数据集很小的时候,最好和你的数据集设置的差不多大小.
当你不指定的时候,batch_size默认是32位
转载自:
https://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network
https://stats.stackexchange.com/questions/303022/understanding-batch-size-in-neural-networks