[Dataset Production] VOC2007 format data set production and processing tutorial (Faster-RCNN model standard input)

illustrate

This case is a tutorial for making a VOC2007 data set for target detection. This data set processing can be used for standard input of target detection models in Faster-RCNN, YOLOv3 and other networks.

• VOC2007 data set structure

#VOC2007数据集结构如下：
VOC2007
	|-Annotations#里面存放的是每一张图片对应的标注结果，为XML文件，
				 #标注完成后JPEGImages每张图片在此都有一一对应的xml文件
	|-ImageSets#存放的是每一种类型对应的图像数据，为txt文件
		|-Layout#存放的是具有人体部位的数据(如人的head、hand、feet等等)
		|-Main#存放的是图像物体识别的数据
		|-Segmentation#存放的是可用于分割的数据
	|-JPEGImages#存放的是原图片
	|-SegmentationClass#按类别进行图像分割，同一类别的物体会被标注为相同颜色
	|-SegmentationObject#按对象进行图像分割，即使是同一类别的物体会被标注为不同的颜色。

This data set can be used for target detection and image segmentation. This experimental case is to explain the standard data set input for target detection of Faster-RCNN, YOLOv3 and other models. Segmentation is not mentioned, so all the above data sets are not used. Only one of them is discussed. Production of target detection part of the data set.

• The structure of the VOC2007 data set used in this experiment is as follows:

#VOC2007数据集结构如下：
VOC2007
	|-Annotations#里面存放的是每一张图片对应的标注结果，为XML文件，
				 #标注完成后JPEGImages每张图片在此都有一一对应的xml文件
	|-ImageSets#存放的是每一种类型对应的图像数据，为txt文件
		|-Main#存放的是图像物体识别的数据
	|-JPEGImages#存放的是原图片

1. Installation tools

• Installation Environment

This data annotation uses the labelImg tool. I installed it with pip in the pycharm terminal. There are many installation methods, so I won’t list them one by one. You also need to install the lxml library and pytq. The installation steps are as follows:

python pip3 install labelImg

pip install lxml

pip install pytq

• Running screenshot:
  
• Enter the environment
  and enter the labeling tool. You only need to enter LabelImg at the pycharm terminal, and the LabelImg tool will automatically pop up and open, as follows:

LabelImg

• Running screenshot:
  

2. Data set preparation

Create these three folders under the VOC2007 folder:

• The Annotations folder stores tag files in xml format. Each xml file corresponds to a picture in the JPEGImages folder.
• JPEGImages folder: This folder stores data set images (original images), including training and test images. Put the original image data in this folder

• ImageSets folder There are three files stored in this folder, namely Layout, Main, and Segmentation. Here we only use the Main file that stores image data

Create the Main folder under ImageSets , and create the following 4 txt files in this folder:

• test.txt : test set
• train.txt : training set
• val.txt : validation set
• trainval.txt : training and validation sets
  

Rename data

• Processing code:

"""文件重命名为000000、000001、000002...nnnnn"""
import os
path = r"C:\Users\xxx\Desktop\VOC2007\JPEGImages"#JPEGImages文件夹所在路径
filelist = os.listdir(path) #该文件夹下所有的文件（包括文件夹）
count=0
for file in filelist:
    print(file)
for file in filelist:   #遍历所有文件
    Olddir=os.path.join(path,file)   #原来的文件路径
    if os.path.isdir(Olddir):   #如果是文件夹则跳过
        continue
    filename=os.path.splitext(file)[0]   #文件名
    filetype='.jpg'   #文件扩展名
    Newdir=os.path.join(path,str(count).zfill(6)+filetype)  #用字符串函数zfill 以0补全所需位数
    os.rename(Olddir,Newdir)#重命名
    count+=1

• Running screenshot:
  
• Processing result display:
  

4. Data annotation

• Add labeled dataset

After installing the tool, enter LabelImge to open the labeling tool, and then add the data folder that needs to be labeled. The original data image is placed in the JPEGImages' folder. The operation is as follows:

• Select the labeling method.
  Select Create rectBox for rectangular labeling, or select the shortcut key w key for quick labeling:

• Labeling and label settings
  Use a frame to mark the target. After the labeling is completed, the label value box will pop up. Enter the label to be added and click OK:
  
• Save the annotation results.
  The default way to save the annotation results is to select the save folder every time. Use the shortcut key to select the default save folder. Use the shortcut key Ctrl+r to select the default save folder as Annotation:

If there is only one tag value marked, you can set the default tag value in the upper right corner of the annotation tool, as shown below:

After the first annotation is set, the next step is quick annotation. Shortcut keys:
w key to select annotation, Ctrl+s key to save, d key to switch to the next picture until the annotation is completed.

• Display of annotation results
  After annotation, the annotation results are saved in the Annotation file under VOC2007. The content is displayed as follows:

5. Data processing

• Perform data processing on the annotation result xml folder

import os
import random

trainval_percent = 0.9#验证集和训练集占的百分比
train_percent = 0.7#训练集占的百分比
xmlfilepath = r'C:\Users\xxx\Desktop\VOC2007\Annotations'#Annotation文件夹所在位置
txtsavepath = r'C:\Users\xxx\Desktop\VOC2007\ImageSets\Main'#ImageSets文件下的Main文件夹所在位置
total_xml = os.listdir(xmlfilepath)

num = len(total_xml)
list = range(num)
tv = int(num * trainval_percent)
tr = int(tv * train_percent)
trainval = random.sample(list, tv)
train = random.sample(trainval, tr)
#Main文件夹下所对应的四个txt文件夹路径
ftrainval = open(r'C:\Users\xxx\Desktop\VOC2007\ImageSets\Main\trainval.txt', 'w')
ftest = open(r'C:\Users\xxx\Desktop\VOC2007\ImageSets\Main\test.txt', 'w')
ftrain = open(r'C:\Users\xxx\Desktop\VOC2007\ImageSets\Main\train.txt', 'w')
fval = open(r'C:\Users\xxx\Desktop\VOC2007\ImageSets\Main\val.txt', 'w')

for i in list:
    name = total_xml[i][:-4] + '\n'
    if i in trainval:
        ftrainval.write(name)
        if i in train:
            ftrain.write(name)
        else:
            fval.write(name)
    else:
        ftest.write(name)

ftrainval.close()
ftrain.close()
fval.close()
ftest.close()

• Running screenshot:
  
  Now open the VOC2007/ImagesSets/Main folder, which contains the following four files:

Open the four files, the contents of which are as follows:

At this point, the VOC2007 format data set is completed.