Road dust is one of the common environmental pollution problems in the urban environment. In some previous projects, we have actually encountered similar detection and identification requirements. The main purpose of this paper is to compare the actual performance of different magnitude parameter models in the bare soil data scenario from the perspective of experimental analysis.
First look at the overall effect:
Next, take a brief look at the dataset:
The example annotation data is as follows:
0 0.248611 0.515278 0.133333 0.119444
0 0.9625 0.688194 0.075 0.068056
0 0.799306 0.636111 0.106944 0.063889
0 0.658333 0.5875 0.119444 0.061111
Here, five models of n--->x with different parameter levels are used in turn to develop and realize the final target detection. Next, let's look at the details of the results in turn:
【n series model】
【s series model】
【m series model】
【l series model】
【X series model】
Next, let’s compare and analyze the results of the key indicators as a whole:
[Precision Curve]
The Precision-Recall Curve is a visualization tool used to evaluate the precision performance of the binary classification model under different thresholds. It helps us understand how the model performs at different thresholds by plotting the relationship between precision and recall at different thresholds.
Precision refers to the ratio of the number of samples correctly predicted as positive to all the samples predicted as positive. The recall rate (Recall) refers to the proportion of the number of samples that are correctly predicted as positive examples to the number of samples that are actually positive examples.
The steps to plot the precision curve are as follows:
Convert the predicted probabilities to binary class labels using different thresholds. Usually, when the predicted probability is greater than a threshold, the sample is classified as a positive example, otherwise it is classified as a negative example.
For each threshold, calculate the corresponding precision and recall.
Plot the precision and recall at each threshold on the same graph to form a precision curve.
According to the shape and changing trend of the accuracy rate curve, an appropriate threshold can be selected to achieve the required performance requirements.
By observing the precision rate curve, we can determine the optimal threshold according to the needs to balance the precision rate and recall rate. Higher precision means fewer false positives, while higher recall means fewer false negatives. Depending on specific business needs and cost trade-offs, an appropriate operating point or threshold can be chosen on the curve.
[Recall Curve]
The Recall Curve is a visualization tool for evaluating the recall performance of a binary classification model under different thresholds. It helps us understand the performance of the model at different thresholds by plotting the relationship between the recall rate and the corresponding precision rate at different thresholds.
The recall rate (Recall) refers to the proportion of the number of samples that are correctly predicted as positive examples to the number of samples that are actually positive examples. The recall rate is also called the sensitivity (Sensitivity) or the true positive rate (True Positive Rate).
The steps to plot the recall curve are as follows:
Convert the predicted probabilities to binary class labels using different thresholds. Usually, when the predicted probability is greater than a threshold, the sample is classified as a positive example, otherwise it is classified as a negative example.
For each threshold, the corresponding recall and corresponding precision are calculated.
Plot the recall and precision at each threshold on the same graph to form a recall curve.
According to the shape and changing trend of the recall rate curve, an appropriate threshold can be selected to achieve the desired performance requirements.
By observing the recall curve, we can determine the optimal threshold according to the needs to balance the recall and precision. Higher recall means fewer false negatives, while higher precision means fewer false positives. Depending on specific business needs and cost trade-offs, an appropriate operating point or threshold can be chosen on the curve.
[F1 value curve]
F1 value curve is a visualization tool for evaluating the performance of binary classification models under different thresholds. It helps us understand the overall performance of the model by plotting the relationship between Precision, Recall and F1 scores at different thresholds.
The F1 score is the harmonic mean of precision and recall, which takes into account both performance metrics. The F1 value curve can help us determine a balance point between different precision rates and recall rates to choose the best threshold.
The steps to plot the F1 value curve are as follows:
Convert the predicted probabilities to binary class labels using different thresholds. Usually, when the predicted probability is greater than a threshold, the sample is classified as a positive example, otherwise it is classified as a negative example.
For each threshold, the corresponding precision, recall and F1 score are calculated.
The precision rate, recall rate and F1 score under each threshold are plotted on the same graph to form an F1 value curve.
According to the shape and changing trend of the F1 value curve, an appropriate threshold can be selected to achieve the required performance requirements.
F1-value curves are often used together with receiver operating characteristic curves (ROC curves) to help evaluate and compare the performance of different models. They provide a more comprehensive classifier performance analysis, and can select appropriate models and threshold settings according to specific application scenarios.
There is also the final Loss comparison curve, as shown below:
Judging from the actual experimental results: when the parameter magnitude exceeds a certain threshold, the result improvement brought about by continuing to increase the parameter amount is minimal, so it is not necessary to use the largest model. In fact, the l-series models are already in a certain degree of performance. Thanks for the x-series models too. However, this is also a choice based on your actual business needs.