Article directory
⭐⭐⭐Content correction
I previously mentioned in the article on April 5, 2023 : "Documents submitted to the official website need to includeTraining set results after testingand the test set results after testing", this conclusion was found to be wrong after testing.
An individual re-tested the content submission for the MOT Challenge under the reminder in the comment area on 2023/12/8, and found that the files submitted to the official website did not need to be tested on the test set, butDirectly convert the GT file of the training setJust merge the test results of the test set with your tracker and submit them to the official website. Therefore, correct the content to say that there is no need to test the test set .
Previously, because the number of submissions was precious, I did not conduct too many tests. In addition, I had a wrong understanding and came to the wrong conclusion. I would like to apologize.
The following are the test results:
Submit your own results after testing the test set
and directly submit the results of the training set GT file.
You can find that the performance of both is the same, so the submission content only needs the training set GT file + your own test result file on the test set .
Preface
Recently I am working on a project related to multi-target tracking. Anyone who has done multiple target tracking knows that the MOT Challenge benchmark data set contains multiple data sets such as MOT15 and MOT16. Each data set can be divided into two categories: test and train.
However, the officially provided test data set does not contain gt.txt (that is, the annotation information of the real tracking frame), so we cannot use motmetrics to obtain performance results such as MOTA and IDF1. Because the official wants to prevent some people from fitting an algorithm with seemingly high performance based on this GT information, which is similar to taking an exam with the answers, thus affecting the actual performance of the algorithm.
Therefore, we need to upload the gt.txt result we obtained locally (submit) to the official website and let them return the performance index evaluation results to us:
Uploading gt requires you to register an official website account, and the process of registering an account and uploading a data set is easy to encounter pitfalls, so I will record my implementation process to help latecomers.
1. Account registration
Enter the official website registration page: https://motchallenge.net/register/
AffiliationFill in the full name of the school,SupervisorEnter the tutor’s English name.
The main problems that tend to occur are the mailbox:
1. Do not use QQ or 163 or gmail email
These are several commonly used email addresses in China. Once filled in, the registration page will display the following information:
Use the school’s email addresseduRegister with your email address (basically every university has a free channel to register your own edu.cn email address. You can go to the official website to check it out). The instructor’s email address also needs to have the suffix:edumailbox. Just like paper submission, emails sent using QQ mailbox are easy to be put into the trash can.eduThe email address is more formal.
2.Normal registration process
useeduAfter registering by email, the registration page will remind you that "your account is under review." Then just wait for the official email to be sent to your email.
The first email tells you that the review has been approved and asks you to go to the designated link to fill in your application.passwordand other information. After filling in and submitting, you will receive a second email.
The second email asks you to click on the specified link to proceedAccount activation。
Once activated, you're done.
There are also cases where some people useeduThere has been no response after registering your email (like me). At this time, you usually need to write a short composition to the official email address.[email protected]. Explain to the administrator which school you are from, what you want to use MOT for, non-commercial use, who is the tutor, what is the tutor's Google Scholar website, etc.
I received a reply from the other party on the third day after I sent the email, and then the registration was successful.
2. The process of uploading test results
The official writing is actually very clear. First create different methods, select different data sets, and finally package and upload all your prediction files.
(Note that the packaged files are not only the files you obtained during test inference, but also include the GT files under the train folder. That is, the content of the submitted package file is: gt.txt under n train/ folders. File + your n test results.txt for the test set . Remember to rename the gt.txt in train to the format required on the page.
For example, if you want to submit the results of MOT16 for testing, your package file should contain 14 files from MOT16-01.txt to MOT16-14.txt)
There are two main problems here:
1. The results cannot be uploaded frequently for evaluation (once every 72 hours, and a MOT data set can only upload results 4 times! The number of uploads is precious). If you upload too frequently, you may be banned. This is probably why Tao Baoshang does not provide testing services because the number of times is limited.
2. The official data set actually uses its own detector (DPM v5, ACF) to obtain the detection frame. These detection frame information also exist in the data set.thefolder, if you are using your own detector, such as YOLOv5, the official will recommend that you send the detector to them, but in fact it will not affect if this is just for your own private testing.
Of course, if you want to officially publish a paper and improve the rankings, you should read the pedestrian detection frame data in the Det file under the data set as the input of the tracking algorithm, and you cannot use your own detector. (Personal understanding: If the tracking algorithm is replaced with a good detector, such as changing from YOLOv5 to YOLOv8, the accuracy will directly increase a lot. It is too cheating. In this way, you are not comparing whose algorithm is better but whose detector is better. to control variables)
1.Usage steps
Create a Method (usually a MOT method, which depends on the data set you use. If you don’t know, create all the methods and see what data sets each method contains.):
Click Create new method, because I use the MOT series data set. I create it directly under the MOT module and fill in the information as you wish:
After creating the method, click on the rightSubmit
Choose your own data set to be used for testing (for example, choose MOT15 here), fill in your own hardware data, inference time, and training data set range. You can fill in the hardware data and inference time as you like.
After filling in, you can see that there is an additional line of MOT15 Challenge under your method. Click Submit Results:
Place your inferred txt in order according to the official website requirements, and package it into a compressed file.zipJust upload the format:
Someone here may encounter Wrong number of result files when uploading! 0 results found in archive, but XXX needed. Please make sure sure that all required txt files are in the root folder of the zip archive , This problem is because you compressed an entire folder, which is actually wrong. This will cause your compressed package to still have a folder (matryoshka) after decompression. The correct approach isctrl+ASelect all txt files, for example, there are 14 txt files, and compress them after selecting them all. In this way, the compressed package will directly get 14 txt files after decompression, without intermediate folders being generated, and there will be no errors in uploading. .
The information in each txt file should be placed in order:
among them, the first column is the frame number, the second column is the character ID, and the next four are (x coordinate of the upper left corner of the pedestrian frame, y coordinate of the upper left corner of the pedestrian frame, frame width, height of the box), confidence, and some other definitions. The txt files you want to infer should be placed based on the definitions in the data set.
(But I saw someone on the Internet saying that as long as the format of each column is correct, the row order in txt will not affect the performance results even if it is not sorted according to the frame order? But it is recommended to use python to handle the order yourself)
For more specific information, please see the description in the data set paper, such as the format mentioned in the MOT Challenge2020 paper below:
After uploading, you can see that the Submit Results will no longer be uploaded.Not allowed(Refresh after 72 hours). Click MOT15 in Challenge to see the data you uploaded.
Here I take MOT16 as an example. After clicking, you can see the performance indicators:
Summarize
The above is how to use MOT Challenge.