[YOLO] YOLOv5+Deep Sort realizes MOT evaluation (open source data set + custom data set)

introduction

YOLOv5+Deep Sort realizes target tracking, and uses MOTChallengeEvalKit to realize the evaluation of multi-target tracking results.
YOLOv5+Deep Sort to achieve target tracking can refer to the author [YOLOv5] yolov5 target recognition + DeepSort target tracking

Implementation steps

1 Install MATLAB

Install MATLAB
MATLAB is a commercial mathematical software, which is used for algorithm development, data visualization, data analysis and numerical calculation of advanced technical computing language and interactive environment. It mainly includes MATLAB and Simulink, which can perform matrix operations, drawing functions and Data, implementing algorithms, creating user interfaces, connecting programs in other programming languages, etc., are mainly used in engineering calculations, control design, signal processing and communication, image processing, signal detection, financial modeling design and analysis and other fields.
Directly attach the reference link for installation: https://mp.weixin.qq.com/s/LAatgqNf55zpxzOYlpn8fg

2 Necessary open source

1. MOT evaluation tool: dendorferpatrick-MOTChallengeEvalKit
2. Open source evaluation database: such as MOT 17, other links are as follows
   
3. Deep Sort target tracking: mikel-brostrom-Yolov5_DeepSort_Pytorch
4. YOLOv5 target recognition: ultralytics-yolov5

下载上述开源压缩包，进行解压，文件的部署如下：

-----------------------------------------------------------------------------------------------------------
# 1.准备数据集（树形框架）
|-MOTChallengeEvalKit(评估工具源码)# 源码1
	|-MOT  # 多目标评估
		|-matlab_devkit  
		|-evalMOT.py  # 评估主代码1
		|-Evaluator.py  # 评估主代码2
		|-Metrics.py
		|-MOT_metrics.py
		|-MOTVisualization.py 
		|-Visualize.py   
|-Yolov5_DeepSort_Pytorch（目标跟踪） # 源码2
	|-deep_sort  # 跟踪源码
	|-yolov5  # 源码3：该文件夹是空的，需要把YOLOv5的源码项目放进去
|-Evaluation-MOTdata（开源数据库） # 笔者自定义的名称(用于存放开源数据库)
	|-MOT16  # 开源数据集
	|-MOT17  
	|-MOT20
-----------------------------------------------------------------------------------------------------------

3 Environment deployment

First, use Anaconda to create a new environment to give priority to the deployment of YOLOv5 environment, especially the python version cannot be lower than the lower limit required by YOLOv5.

conda create -n yolov5-6 python=3.7
conda activate yolov5-6  # 激活环境

3.1 Compile with MATLAB

The author's version of MATLAB is: MATLAB R2021a
the terminal enters the installation path of MATLAB: D:\Program Files\Polyspace\R2021a\extern\engines\pythoncustom installation path

cd D:\Program Files\Polyspace\R2021a\extern\engines\python
# 执行如下指令
python setup.py build --build-base="builddir" install
python setup.py install --prefix="installdir"
python setup.py build --build-base="builddir" install --prefix="installdir" 
python setup.py install --user

After executing the above instructions, the new files under the installation path of MATLAB are as follows:

Open MATLAB to complete the compilation
Open the source code project file of the evaluation tool, and copy the following files to MOTthe folder. The copy operation of this step is very important, otherwise, even if the compilation is successful, Evaluation also fails.

After copying, the files under the MOT folder are as follows:

Complete the compilation according to the screenshot

. If the above files are not copied, the evaluation will have the following errors:

3.2 DeepSort +YOLOv5 environment construction

Prioritize the environment deployment of YOLOv5. DeepSort has no special requirements, and what is missing at runtime is what to make up for.
Reference link [YOLOv5] 6.0 environment construction (updated from time to time)

3.3 MOT benchmark dataset format

Introduce the data format in the benchmark data set, taking MOT20 as an example:

including test and train, of which only train contains the true value of tracking, which can be used for evaluation
1. seqinfo.ini
has this in each subfolder, mainly It is used to describe some information of this file, such as length, frame rate, length and width of the picture, and the suffix name of the picture.

2. The det.txt
file stores the information of the detection frame of the picture (the detected information file), and MOT evaluation cannot be performed because there is no id information (the author could not understand the -1 in the second column and the self-generated one at the beginning. Assess the relationship between the formats. Later, I will know that det.txt only saves the detection information and has nothing to do with the mot evaluation. I will introduce how to generate the results for MOT evaluation later.) Part of the content is shown as follows:

从左到右分别代表：
• frame: 第几帧图片
• id: 这个检测框分配的 id，在这里都是-1 代表没有 id 信息（因为检测是没有跟踪信息id的）
• bbox(四位): 分别是左上角坐标和长宽
• conf：这个 bbox 包含物体的置信度，可以看到并不是传统意义的 0-1，分数越高代表置信度越高• MOT3D(x,y,z): 是在 MOT3D 中使用到的内容，这里关心的是 MOT2D，所以都设置为-1可以看出以上内容主要提供的和目标检测的信息没有区别，所以也在一定程度上可以用于检测器的训练。

3.
The file gt.txt is only found in the subfolder of train, not in test. The format of its content is somewhat similar to that of det.txt, and part of the content is
as follows:

从左到右分别是：
• frame: 第几帧图片
• ID: 也就是轨迹的 ID，可以看出 gt 里边是按照轨迹的 ID 号进行排序的
• bbox: 分别是左上角坐标和长宽
• 是否忽略：0 代表忽略（即不需要跟踪的目标）
• classes: 目标的类别索引，从1开始！！！！！！！！
• 最后一个代表目标运动时被其他目标包含、覆盖、边缘裁剪的情况，1代表完全被遮挡（如果是自定义基准的话，这部分一般为0，或者可以写入其他信息）

4 Assessment Test

Test whether the environment built above can be evaluated in the open source MOT17

4.1 Download ready-made tracking results

Ready-made tracking results The author's reference link: ( python version MOTChallengeEvalKit evaluation tool running, run MOT evaluation tool )
link: https://pan.baidu.com/s/1CkPse7rcx53qOEy8YlU6qw
Extraction code: rlwj
Put the downloaded file into the open source dataset :

4.2 Operational evaluation

(1) Open the evaluation code MOTChallengeEvalKit-master\MOT\evalMOT.py
and modify the configuration as follows:

if __name__ == "__main__":
	eval = MOT_evaluator()

	benchmark_name = "MOT17"
	gt_dir = r"E:\Alian\yolov5\MOTChallengeEvalKit-master\MOT17"  # GT
	res_dir = r"E:\Alian\yolov5\MOTChallengeEvalKit-master\MOT17\results\test_track_MOT17_fuc_v0"  # 跟踪结果
	eval_mode = "train"  # MOT17下的train模式，因为只有train才有GT
	seqmaps_dir = r'E:\Alian\yolov5\MOTChallengeEvalKit-master\seqmaps'
	eval.run(
		benchmark_name = benchmark_name,
		gt_dir = gt_dir,
		res_dir = res_dir,
		eval_mode = eval_mode,
		seqmaps_dir=seqmaps_dir
	)

(2) Open MOTChallengeEvalKit-master\seqmaps\MOT17-train.txt
Only keep the following content, delete the redundant ones (because there are only the tracking results of the following files), check whether the benchmarks and tracking results correspond one-to

-

one

. .txt):

The terminal executes the command, and the printed results are as follows

python evalMOT.py

5 Realize tracking evaluation under customized data

5.1 Modification of evaluation code

./MOT/evalMOT.py is modified as follows

"""
2022.3.12
author:alian
实现跟踪结果的评估
"""
import sys, os
sys.path.append(os.path.abspath(os.getcwd()))
import math
from collections import defaultdict
from MOT_metrics import MOTMetrics
from Evaluator import Evaluator, run_metrics,My_Evaluator
import multiprocessing as mp
import pandas as pd
# 2、自定义数据集评估
class My_MOTevaluator(My_Evaluator):  # 继承父类
	def __init__(self):
		super().__init__()
		pass

	def my_eval(self):  # 重写评估函数
		arguments = []  # 传入参数: 项目名称，跟踪结果文件列表，基准文件列表
		for seq, res, gt in zip(self.sequences, self.tsfiles, self.gtfiles):  # 评估路段的名称，跟踪结果文件，基准文件
			arguments.append({
    
    "metricObject": MOTMetrics(seq), "args": {
    
    
			"gtDataDir":  os.path.join(os.path.dirname(gt),seq),
			"sequence": str(seq),
			"pred_file":res,
			"gt_file": gt,
			"benchmark_name": self.benchmark_name}})
		try:
			if self.MULTIPROCESSING:
				p = mp.Pool(self.NR_CORES)  # 实例化多进程并行池
				print("Evaluating on {} cpu cores".format(self.NR_CORES))
				# 计算每个序列的指标
				processes = [p.apply_async(run_metrics, kwds=inp) for inp in arguments]  # 得到多进程处理结果
				self.results = [p.get() for p in processes]  # 评估结果
				p.close()
				p.join()
			else:
				self.results = [run_metrics(**inp) for inp in arguments]
			self.failed = False  # 成功
		except:
			self.failed = True
			raise Exception("<exc> MATLAB evalutation failed <!exc>")
		self.Overall_Results = MOTMetrics("OVERALL")  # 定义评估指标矩阵的名称
		return self.results,self.Overall_Results

./MOT/Evaluator.py

"""
2022.3.12
author:alian
"""
import sys, os
sys.path.append(os.getcwd())
import argparse
import traceback
import time
import pickle
import pandas as pd
import glob
from os import path
import numpy as np
class My_Evaluator(object):  # 多目标跟踪评估器类
	""" 评估器类运行每个序列的评估并计算基准的整体性能"""
	def __init__(self):
		pass

	def run(self, benchmark_name=None,  gt_dir=None, mot_dir=None,save_csv = None):
		self.benchmark_name = benchmark_name  # 项目名称
		start_time = time.time()
		self.gtfiles = glob.glob('%s/*_gt.txt'%gt_dir)  # 基准文件gt.txt全路径
		self.tsfiles = glob.glob('%s/*_mot.txt'%mot_dir)  # 评估结果全路径
		self.sequences = [os.path.split(x)[-1].split('_')[0] for x in self.gtfiles]
		print('Found {} ground truth files and {} test files.'.format(len(self.gtfiles), len(self.tsfiles)))
		# 设置多核处理
		self.MULTIPROCESSING = True  # 多处理
		MAX_NR_CORES = 10  # 最大的处理量
		if self.MULTIPROCESSING: self.NR_CORES = np.minimum(MAX_NR_CORES, len(self.tsfiles))
		try:
			""" 执行评估 """
			self.results,self.Overall_Results = self.my_eval()
			# 计算整体结果
			results_attributes = self.Overall_Results.metrics.keys()  # 评估指标（名称）
			for attr in results_attributes:
				""" 在所有序列上累积评估值 """
				try:
					self.Overall_Results.__dict__[attr] = sum(obj.__dict__[attr] for obj in self.results)
				except:
					pass
			cache_attributes = self.Overall_Results.cache_dict.keys()  # 缓存属性
			for attr in cache_attributes:
				""" accumulate cache values over all sequences """
				try:
					self.Overall_Results.__dict__[attr] = self.Overall_Results.cache_dict[attr]['func']([obj.__dict__[attr] for obj in self.results])
				except:
					pass
			print("evaluation successful")
			# 计算clearmot指标
			for res in self.results:
				res.compute_clearmot()
			self.Overall_Results.compute_clearmot()
			summary = self.accumulate_df(type="mail")
			# 将评估果写入csv文件
			summary.to_csv(save_csv)
		except:pass
		end_time=time.time()
		self.duration = (end_time - start_time)/60.

		# 收集评估错误
		print("Evaluation Finished")
		print("Your Results")
		print(self.render_summary())
		print("Successfully save results")
		return self.Overall_Results, self.results

	def my_eval(self):  # 实际使用时重写
		raise NotImplementedError

	def accumulate_df(self, type = None):
		""" create accumulated dataframe with all sequences """
		summary = None
		for k, res in enumerate(self.results):
			res.to_dataframe(display_name = True, type=type)
			if k == 0: summary = res.df
			else:summary = pd.concat([summary,res.df])
		summary = summary.sort_index()
		self.Overall_Results.to_dataframe(display_name=True, type=type)
		self.summary = pd.concat([summary,self.Overall_Results.df])
		return self.summary


	def render_summary( self, buf = None):  # 终端打印评估指标
		"""
		Params:
		summary : pd.DataFrame
		Kwargs：
		buf : StringIO-like, optional 写入缓存区
		formatters : dict, optional(为单个指标定义自定义格式化程序的字典){'mota': '{:.2%}'.format}.MetricsHost.formatters
		namemap : dict, optional {'num_false_positives': 'FP'}
		Returns：
		string
		"""
		output = self.summary.to_string(
			buf=buf,
			formatters=self.Overall_Results.formatters,
			justify="left"
		)
		return output
def run_metrics( metricObject, args ):
	""" Runs metric for individual sequences
	Params:
	-----
	metricObject: metricObject that has computer_compute_metrics_per_sequence function
	args: dictionary with args for evaluation function
	"""
	metricObject.compute_metrics_per_sequence(**args)  # 计算每个序列的指标
	return metricObject
if __name__ == "__main__":
	Evaluator()

5.2 Benchmarking and Assessment Results Preparation

(1) Benchmark file: name_gt.txt

从左到右分别是：
 1. frame: 第几帧图片
 2. ID:也就是轨迹的ID，可以看出gt里边是按照轨迹的ID号进行排序的
 3. bbox: 分别是左上角坐标和长宽
 4. bbox: 分别是左上角坐标和长宽
 5. bbox: 分别是左上角坐标和长宽
 6. bbox: 分别是左上角坐标和长宽
 7. bool类型：代表是否忽略：0代表忽略，1代表该目标被选
 8. classes:目标的类别个数（开源的MOT数据集中：驾驶场景包括12个类别，7代表的是静止的人， 第8个类代表错检，9-11代表被遮挡的类别，如下图），自定数据集就根据自己的目标类别数
 9. 最后一个代表目标运动时被其他目标包含、覆盖、边缘裁剪的情况，这里笔者将目标的置信度写入该列的值

(2) Tracking result: name_mot.txt

从左到右分别是：
 1. frame: 第几帧图片
 2. ID:也就是轨迹的ID，可以看出gt里边是按照轨迹的ID号进行排序的
 3. bbox: 分别是左上角坐标和长宽
 4. bbox: 分别是左上角坐标和长宽
 5. bbox: 分别是左上角坐标和长宽
 6. bbox: 分别是左上角坐标和长宽
 7. conf：目标置信度
 8. MOT3D(x,y,z): 是在MOT3D（三维跟踪）中使用到的内容，这里关心的是MOT二维，所以都设置为-1

5.3 Implement MOT evaluation of custom data sets

Open ./MOT/evalMOT.py and modify the configuration as follows

if __name__ == "__main__":
	#  自定义数据集评估
	my_eval = My_MOTevaluator()
	benchmark_name = "MOT北京地铁9号线"
	gt_dir = r"E:\Alian\yolov5\Yolov5_DeepSort_alian\results"  # GT
	res_dir = r"E:\Alian\yolov5\Yolov5_DeepSort_alian\results"  # 跟踪结果
	my_eval.run(
		benchmark_name=benchmark_name,
		gt_dir=gt_dir,
		mot_dir=res_dir,
		save_csv='MOT北京地铁9号线.csv'
	)

6 Complete Code Summary

6.1 evalMOT.py

"""
2022.3.12
author:alian
实现跟踪结果的评估
"""
import sys, os
sys.path.append(os.path.abspath(os.getcwd()))
import math
from collections import defaultdict
from MOT_metrics import MOTMetrics
from Evaluator import Evaluator, run_metrics,My_Evaluator
import multiprocessing as mp
import pandas as pd


# 1、开源跟踪评估
class MOT_evaluator(Evaluator):  # 继承父类
	def __init__(self):
		super().__init__()
		self.type = "MOT"

	def eval(self):  # 重写评估函数
		print("Check prediction files")  # 检查评估文件
		error_message = ""
		for pred_file in self.tsfiles:  # 遍历跟踪结果
			print(pred_file)
			df = pd.read_csv(pred_file, header=None, sep=",")  # 默认用逗号隔开
			if len(df.columns) == 1:  # 若只有一列
				f = open(pred_file, "r")
				error_message+= "<exc>Submission %s not in correct form. Values in file must be comma separated.<br>Current form:<br>%s<br>%s<br>.........<br><!exc>" % (pred_file.split("/")[-1], f.readline(),  f.readline())
				raise Exception(error_message)

			df.groupby([0,1]).size().head()  # 根据帧序，ID对数据进行分组(取前5行)
			count = df.groupby([0,1]).size().reset_index(name='count')  # 将分组索引（帧序，ID）设置为列，则此时的列有：帧序，ID，计数

			# 检查ID的独一性
			if any(count["count"]>1):
				doubleIDs  = count.loc[count["count"]>1][[0,1]].values
				error_message+= "<exc> Found duplicate ID/Frame pairs in sequence %s." % pred_file.split("/")[-1]
				for id in doubleIDs:
					double_values = df[((df[0] == id[0]) & (df[1] == id[1]))]
					for row in double_values.values:
						error_message += "<br> %s" % row
				error_message+="<br> <!exc> "
		if error_message != "":
			raise Exception(error_message)
		print("Files are ok!")  # 文件检查完毕！
		arguments = []  # 传入参数:
		for seq, res, gt in zip(self.sequences, self.tsfiles, self.gtfiles):  # 基准名称，跟踪结果文件，基准文件
			arguments.append({
    
    "metricObject": MOTMetrics(seq), "args": {
    
    
			"gtDataDir":  os.path.join(self.datadir,seq),
			"sequence": str(seq),
			"pred_file":res,
			"gt_file": gt,
			"benchmark_name": self.benchmark_name}})
		try:
			if self.MULTIPROCESSING:
				p = mp.Pool(self.NR_CORES)  # 实例化多进程并行池
				print("Evaluating on {} cpu cores".format(self.NR_CORES))
				# 计算每个序列的指标
				processes = [p.apply_async(run_metrics, kwds=inp) for inp in arguments]  # 得到多进程处理结果
				self.results = [p.get() for p in processes]  # 评估结果
				p.close()
				p.join()
			else:
				self.results = [run_metrics(**inp) for inp in arguments]
			self.failed = False  # 成功
		except:
			self.failed = True
			raise Exception("<exc> MATLAB evalutation failed <!exc>")
		self.Overall_Results = MOTMetrics("OVERALL")  # 定义评估指标矩阵的名称
		return self.results,self.Overall_Results


# 2、自定义数据集评估
class My_MOTevaluator(My_Evaluator):  # 继承父类
	def __init__(self):
		super().__init__()
		pass

	def my_eval(self):  # 重写评估函数
		arguments = []  # 传入参数: 项目名称，跟踪结果文件列表，基准文件列表
		for seq, res, gt in zip(self.sequences, self.tsfiles, self.gtfiles):  # 评估路段的名称，跟踪结果文件，基准文件
			arguments.append({
    
    "metricObject": MOTMetrics(seq), "args": {
    
    
			"gtDataDir":  os.path.join(os.path.dirname(gt),seq),
			"sequence": str(seq),
			"pred_file":res,
			"gt_file": gt,
			"benchmark_name": self.benchmark_name}})
		try:
			if self.MULTIPROCESSING:
				p = mp.Pool(self.NR_CORES)  # 实例化多进程并行池
				print("Evaluating on {} cpu cores".format(self.NR_CORES))
				# 计算每个序列的指标
				processes = [p.apply_async(run_metrics, kwds=inp) for inp in arguments]  # 得到多进程处理结果
				self.results = [p.get() for p in processes]  # 评估结果
				p.close()
				p.join()
			else:
				self.results = [run_metrics(**inp) for inp in arguments]
			self.failed = False  # 成功
		except:
			self.failed = True
			raise Exception("<exc> MATLAB evalutation failed <!exc>")
		self.Overall_Results = MOTMetrics("OVERALL")  # 定义评估指标矩阵的名称
		return self.results,self.Overall_Results

if __name__ == "__main__":
	# 1 开源数据集跟踪评估
	# benchmark_name = "MOT17"
	# gt_dir = r"E:\Alian\yolov5\MOTChallengeEvalKit-master\MOT17"  # GT
	# res_dir = r"E:\Alian\yolov5\MOTChallengeEvalKit-master\MOT17\results\test_track_MOT17_fuc_v0"  # 跟踪结果
	# eval_mode = "train"  # MOT17下的train模式，因为只有train才有GT
	# seqmaps_dir = r'E:\Alian\yolov5\MOTChallengeEvalKit-master\seqmaps'
	# eval.run(
	# 	benchmark_name = benchmark_name,
	# 	gt_dir=gt_dir,
	# 	res_dir=res_dir,
	# 	eval_mode=eval_mode,
	# 	seqmaps_dir=seqmaps_dir
	# )
	# 2 自定义数据集评估
	my_eval = My_MOTevaluator()
	benchmark_name = "MOT北京地铁9号线"
	gt_dir = r"E:\Alian\yolov5\Yolov5_DeepSort_alian\results"  # GT
	res_dir = r"E:\Alian\yolov5\Yolov5_DeepSort_alian\results"  # 跟踪结果
	my_eval.run(
		benchmark_name=benchmark_name,
		gt_dir=gt_dir,
		mot_dir=res_dir,
		save_csv='MOT北京地铁9号线.csv'
	)

6.2 Evaluator.py

"""
2022.3.12
author:alian
"""
import sys, os
sys.path.append(os.getcwd())
import argparse
import traceback
import time
import pickle
import pandas as pd
import glob
from os import path
import numpy as np


class Evaluator(object):  # 多目标跟踪评估器 类
	""" 评估器类运行每个序列的评估并计算基准的整体性能"""
	def __init__(self):
		pass

	def run(self, benchmark_name=None,  gt_dir=None, res_dir=None, save_pkl=None, eval_mode="train", seqmaps_dir="seqmaps"):
		"""
		Params
		benchmark_name: 基准名称，如 MOT17
		gt_dir: directory of folders with gt data, including the c-files with sequences
		res_dir: directory with result files
			<seq1>.txt
			<seq2>.txt
			...
			<seq3>.txt
		eval_mode:
		seqmaps_dir:
		seq_file: File name of file containing sequences, e.g. 'c10-train.txt'
		save_pkl: path to output directory for final results
		"""
		start_time = time.time()
		self.benchmark_gt_dir = gt_dir  # 基准文件夹MOT17  'E:\\Alian\\yolov5\\Evaluation-MOTdata\\MOT17'
		self.seq_file = "{}-{}.txt".format(benchmark_name, eval_mode)  # MOT17-train.txt（序列）
		res_dir = res_dir  # 评估结果文件夹 'E:\\Alian\\yolov5\\Evaluation-MOTdata\\MOT17\\results\\test_track_MOT17_fuc_v0'
		self.benchmark_name = benchmark_name  # 基准名称'MOT17'
		self.seqmaps_dir = seqmaps_dir  # 序列文件夹（包含各数据集的序列）E:\Alian\yolov5\MOTChallengeEvalKit-master\seqmaps
		self.mode = eval_mode  # 'train'
		self.datadir = os.path.join(gt_dir, self.mode)  # E:\Alian\yolov5\Evaluation-MOTdata\MOT17\train
		error_traceback = ""  # 汇总错误信息
		assert self.mode in ["train", "test", "all"], "mode: %s not valid " %self.mode
		print("Evaluating Benchmark: %s" % self.benchmark_name)
		# 处理评估 ======================================================
		# 加载所有序列的列表 'E:\\Alian\\yolov5\\MOTChallengeEvalKit-master\\seqmaps\\MOT17-train.txt'
		self.sequences = np.genfromtxt(os.path.join(self.seqmaps_dir, self.seq_file), dtype='str', skip_header=True)
		self.gtfiles = []  # 基准文件gt.txt全路径
		self.tsfiles = []  # 评估结果全路径
		for seq in self.sequences:
			gtf = os.path.join(self.benchmark_gt_dir, self.mode ,seq, 'gt/gt.txt')  # gt.txt
			if path.exists(gtf): self.gtfiles.append(gtf)
			else: raise Exception("Ground Truth %s missing" % gtf)  # 判断文件是否存在
			tsf = os.path.join(res_dir, "%s.txt" % seq)
			if path.exists(gtf): self.tsfiles.append(tsf)
			else: raise Exception("Result file %s missing" % tsf)
		print('Found {} ground truth files and {} test files.'.format(len(self.gtfiles), len(self.tsfiles)))
		print(self.tsfiles)  # 打印全部的评估结果文件路径
		# 设置多核处理
		self.MULTIPROCESSING = True  # 多处理
		MAX_NR_CORES = 10  # 最大的处理量
		if self.MULTIPROCESSING: self.NR_CORES = np.minimum(MAX_NR_CORES, len(self.tsfiles))
		try:
			""" 执行评估 """
			self.results,self.Overall_Results = self.eval()
			# 计算整体结果
			results_attributes = self.Overall_Results.metrics.keys()  # 评估指标（名称）

			for attr in results_attributes:
				""" 在多进程上累积评估值 """
				try:
					self.Overall_Results.__dict__[attr] = sum(obj.__dict__[attr] for obj in self.results)
				except:
					pass
			cache_attributes = self.Overall_Results.cache_dict.keys()  # 缓存属性
			for attr in cache_attributes:
				""" accumulate cache values over all sequences """
				try:
					self.Overall_Results.__dict__[attr] = self.Overall_Results.cache_dict[attr]['func']([obj.__dict__[attr] for obj in self.results])
				except:
					pass
			print("evaluation successful")
			# 计算clearmot指标
			for res in self.results:
				res.compute_clearmot()
			self.Overall_Results.compute_clearmot()
			summary = self.accumulate_df(type="mail")
			# 将评估果写入csv文件
			self.summary.to_csv('summary.csv')
			self.failed = False

		except Exception as e:
			print(str(traceback.format_exc()))
			print ("<br> Evaluation failed! <br>")
			error_traceback+= str(traceback.format_exc())
			self.failed = True
			self.summary = None
		end_time=time.time()
		self.duration = (end_time - start_time)/60.

		# 收集评估错误
		if self.failed:
			startExc = error_traceback.split("<exc>")
			error_traceback = [m.split("<!exc>")[0] for m in startExc[1:]]
			error = ""
			for err in error_traceback:
				error += "Error: %s" % err
			print( "Error Message", error)
			self.error = error
			print("ERROR %s" % error)
		print ("Evaluation Finished")
		print("Your Results")
		print(self.render_summary())
		if save_pkl:  # 保存评估结果
			self.Overall_Results.save_dict(os.path.join( save_pkl, "%s-%s-overall.pkl" % (self.benchmark_name, self.mode)))
			for res in self.results:
				res.save_dict(os.path.join( save_pkl, "%s-%s-%s.pkl" % (self.benchmark_name, self.mode, res.seqName)))
			print("Successfully save results")
		return self.Overall_Results, self.results

	def eval(self):  # 实际使用时重写
		raise NotImplementedError


	def accumulate_df(self, type = None):
		""" create accumulated dataframe with all sequences """
		summary = None
		for k, res in enumerate(self.results):
			res.to_dataframe(display_name = True, type=type)
			if k == 0: summary = res.df
			else: summary = pd.concat([summary,res.df])
		summary = summary.sort_index()
		self.Overall_Results.to_dataframe(display_name=True, type=type)
		self.summary = pd.concat([summary,self.Overall_Results.df])
		return self.summary


	def render_summary( self, buf = None):  # 终端打印评估指标
		"""
		Params:
		summary : pd.DataFrame
		Kwargs：
		buf : StringIO-like, optional 写入缓存区
		formatters : dict, optional(为单个指标定义自定义格式化程序的字典){'mota': '{:.2%}'.format}.MetricsHost.formatters
		namemap : dict, optional {'num_false_positives': 'FP'}
		Returns：
		string
		"""
		output = self.summary.to_string(
			buf=buf,
			formatters=self.Overall_Results.formatters,
			justify="left"
		)
		return output

class My_Evaluator(object):  # 多目标跟踪评估器类
	""" 评估器类运行每个序列的评估并计算基准的整体性能"""
	def __init__(self):
		pass

	def run(self, benchmark_name=None,  gt_dir=None, mot_dir=None,save_csv = None):
		self.benchmark_name = benchmark_name  # 项目名称
		start_time = time.time()
		self.gtfiles = glob.glob('%s/*_gt.txt'%gt_dir)  # 基准文件gt.txt全路径
		self.tsfiles = glob.glob('%s/*_mot.txt'%mot_dir)  # 评估结果全路径
		self.sequences = [os.path.split(x)[-1].split('_')[0] for x in self.gtfiles]
		print('Found {} ground truth files and {} test files.'.format(len(self.gtfiles), len(self.tsfiles)))
		# 设置多核处理
		self.MULTIPROCESSING = True  # 多处理
		MAX_NR_CORES = 10  # 最大的处理量
		if self.MULTIPROCESSING: self.NR_CORES = np.minimum(MAX_NR_CORES, len(self.tsfiles))
		try:
			""" 执行评估 """
			self.results,self.Overall_Results = self.my_eval()
			# 计算整体结果
			results_attributes = self.Overall_Results.metrics.keys()  # 评估指标（名称）
			for attr in results_attributes:
				""" 在所有序列上累积评估值 """
				try:
					self.Overall_Results.__dict__[attr] = sum(obj.__dict__[attr] for obj in self.results)
				except:
					pass
			cache_attributes = self.Overall_Results.cache_dict.keys()  # 缓存属性
			for attr in cache_attributes:
				""" accumulate cache values over all sequences """
				try:
					self.Overall_Results.__dict__[attr] = self.Overall_Results.cache_dict[attr]['func']([obj.__dict__[attr] for obj in self.results])
				except:
					pass
			print("evaluation successful")
			# 计算clearmot指标
			for res in self.results:
				res.compute_clearmot()
			self.Overall_Results.compute_clearmot()
			summary = self.accumulate_df(type="mail")
			# 将评估果写入csv文件
			summary.to_csv(save_csv)
		except:pass
		end_time=time.time()
		self.duration = (end_time - start_time)/60.

		# 收集评估错误
		print("Evaluation Finished")
		print("Your Results")
		print(self.render_summary())
		print("Successfully save results")
		return self.Overall_Results, self.results

	def my_eval(self):  # 实际使用时重写
		raise NotImplementedError

	def accumulate_df(self, type = None):
		""" create accumulated dataframe with all sequences """
		summary = None
		for k, res in enumerate(self.results):
			res.to_dataframe(display_name = True, type=type)
			if k == 0: summary = res.df
			else:summary = pd.concat([summary,res.df])
		summary = summary.sort_index()
		self.Overall_Results.to_dataframe(display_name=True, type=type)
		self.summary = pd.concat([summary,self.Overall_Results.df])
		return self.summary


	def render_summary( self, buf = None):  # 终端打印评估指标
		"""
		Params:
		summary : pd.DataFrame
		Kwargs：
		buf : StringIO-like, optional 写入缓存区
		formatters : dict, optional(为单个指标定义自定义格式化程序的字典){'mota': '{:.2%}'.format}.MetricsHost.formatters
		namemap : dict, optional {'num_false_positives': 'FP'}
		Returns：
		string
		"""
		output = self.summary.to_string(
			buf=buf,
			formatters=self.Overall_Results.formatters,
			justify="left"
		)
		return output
def run_metrics( metricObject, args ):
	""" Runs metric for individual sequences
	Params:
	-----
	metricObject: metricObject that has computer_compute_metrics_per_sequence function
	args: dictionary with args for evaluation function
	"""
	metricObject.compute_metrics_per_sequence(**args)  # 计算每个序列的指标
	return metricObject

if __name__ == "__main__":
	Evaluator()