Darknet - When should I stop training -? When should I stop training?

Darknet - When should I stop training -? When should I stop training?

https://github.com/AlexeyAB/darknet

Usually sufficient 2000 iterations for each class ( object), but not less than 4000 iterations in total But for a more precise definition when you should stop training, use the following manual:.
Typically, each class (object) requires 2,000 iterations , but the total number of not less than 4000 times. But for a more precise definition of when to stop training, use the following manual:

1. training

Training During, you VARYING by Will See Indicators of error, and the when you Should STOP NO longer Decreases 0.XXXXXXX avg:
During training, you will see all kinds of wrong instructions, and when not reduce 0.XXXXXXX avg, you should stop training.

Region Avg IOU: 0.798363, Class: 0.893232, Obj: 0.700808, No Obj: 0.004567, Avg Recall: 1.000000, count: 8 Region Avg IOU: 0.800677, Class: 0.892181, Obj: 0.701590, No Obj: 0.004574, Avg Recall: 1.000000, count: 8

9002: 0.211667, 0.60730 avg, 0.001000 rate, 3.868000 seconds, 576128 images Loaded: 0.000000 seconds

9002 - iteration number (number of batch)
0.60730 avg - average loss (error) - the lower, the better

The when you See that Average Loss 0.xxxxxx avgNO longer Decreases AT MANY the Iterations the then you Should STOP Training. At The Final avgerage Loss CAN BE from 0.05(for A Small Model and the Easy the DataSet) to 3.0(for A Big Model and A 'apologetic' the DataSet).
When you find that the average loss 0.xxxxxx avgis no longer reduced many iterations, you should stop training. The final average loss from 0.05(and simple model for a small data set) to 3.0(difficulty for large models and data sets).

2. mAP (mean average precision)

Training IS stopped Once, you Should some of the Take Last .weights-filesfrom darknet\build\darknet\x64\backupand the Choose at The Best of Them:
After the train stopped, you should start darknet\build\darknet\x64\backupto get some of the last .weightsfile, and choose the best file:

The For Example, you stopped Training the After 9000 the Iterations, But at The Best the Result CAN give One of Previous weights (7000, 8000, 9000) It CAN happen Due to overfitting.. Overfitting - IS Case the when you CAN Detect Objects ON ImagesRF Royalty Free from Training-the DataSet , but not Detect Objects cAN oN the any Others ImagesRF Royalty Free. by You Should GET weights from Early stopping Point :
For example, you stop training after 9000 iterations, but the best results can be given one of the weights before (7000, 8000, 9000). It may be due to over-fitting occurs. Over-fitting - this situation is that you can detect an object image from the centralized training data, but can not detect objects on other images. You should get weight from stopping point in advance:

To get weights from Early Stopping Point:

At first, in your file obj.data you must specify the path to the validation dataset valid = valid.txt (format of valid.txt as in train.txt), and if you haven’t validation images, just copy data\train.txt to data\valid.txt.

If training is stopped after 9000 iterations, to validate some of previous weights use this commands:
(If you use another GitHub repository, then use darknet.exe detector recall... instead of darknet.exe detector map...)

darknet.exe detector map data/obj.data yolo-obj.cfg backup\yolo-obj_7000.weights
darknet.exe detector map data/obj.data yolo-obj.cfg backup\yolo-obj_8000.weights
darknet.exe detector map data/obj.data yolo-obj.cfg backup\yolo-obj_9000.weights

./darknet detector map ./cfg/yolov3-tiny.data ./cfg/yolov3-tiny.cfg /media/famu/DISK_DATA/yongqiang/yolov3-tiny_best.weights -i 1

: And compare last output lines for each weights (7000, 8000, 9000)
the last line of output and compare each weight (7000,8000,9000) of:

Choose weights-file with the highest mAP (mean average precision) or IoU (intersect over union)

For example, bigger mAP gives weights yolo-obj_8000.weights - then use this weights for detection.

Or just train with -map flag:

darknet.exe detector train data/obj.data yolo-obj.cfg darknet53.conv.74 -map

./darknet detector train ./train_cfg/yolov3-tiny.data ./train_cfg/yolov3-tiny.cfg -gpus 0,1,2,3 -map

So you will see mAP-chart (red-line) in the Loss-chart Window. mAP will be calculated for each 4 Epochs using valid=valid.txt file that is specified in obj.data file (1 Epoch = images_in_train_txt / batch iterations)
(to change the max x-axis value - change max_batches= parameter to 2000*classes, f.e. max_batches=6000 for 3 classes)

Example of custom object detection: darknet.exe detector test data/obj.data yolo-obj.cfg yolo-obj_8000.weights

• IoU (intersect over union)- average instersect over union of objects and detections for a certain threshold = 0.24.
  A time threshold = 0.24, objects and detections and the average ratio of the cross.

• mAP (mean average precision) - mean value of average precisions for each class, where average precision is average value of 11 points on PR-curve for each possible threshold (each probability of detection) for the same class (Precision-Recall in terms of PascalVOC, where Precision=TP/(TP+FP) and Recall=TP/(TP+FN)).
  The PASCAL Visual Object Classes (VOC) Challenge
  http://homepages.inf.ed.ac.uk/ckiw/postscript/ijcv_voc09.pdf

mAP is default metric of precision in the PascalVOC competition, this is the same as AP50 metric in the MS COCO competition. In terms of Wiki, indicators Precision and Recall have a slightly different meaning than in the PascalVOC competition, but IoU always has the same meaning.
mAP PascalVOC race is the default accuracy index, the same as the MS COCO competitions AP50 index. In terms of Wiki, meaning indicators Precision and Recall meaning PascalVOC race are slightly different, but IoU always have the same meaning.