Table of contents
1 Label the image data set and save it as a txt file
2 Randomly divide the image label data set in the txt file into a training set and a test set
3 Load the image label dataset in the txt file
0 Preface
It's the fourth day of being banned, and I can only stay in the dormitory and can't go out. I remember the last time I was like this was a year ago when I was about to graduate in my senior year...
There is nothing to do in the dormitory these days, and the experiment can’t be done for the time being. After processing some of the data, let’s take notes on the previous content. This is not new knowledge. Simply record it for future convenience can be viewed.
1 Label the image data set and save it as a txt file
Since what I am doing here is regression prediction using deep learning, my labels are saved in (.csv) files. At this time, I need to match the pictures and labels one by one, and divide them into folders. The following is my division. Folder ( images saves pictures, label.csv saves corresponding labels , here you can change the file name according to your personal data set):
The following is the code for labeling the image dataset and saving it as a txt file (the path of the file needs to be changed according to the location of the file):
import os
import numpy as np
import pandas as pd
label = pd.read_csv('../dataset_1/label_1.csv')
label = np.array(label)
label = label.tolist()
target = ''
# for i in range(len(label)):
# for j in range(len(label[i])):
# target += str(label[i][j]) + ' '
# print(target)
# target = ''
def generate(dir):
files = os.listdir(dir) #os.listdir() 方法用于返回指定的文件夹包含的文件或文件夹的名字的列表。
# files.sort() #对文件或文件夹进行排序
files.sort(key=lambda x: int(x.replace("frame", "").split('.')[0]))
print('****************')
print('input :', dir)
print('start...')
target = ''
i = 0
listText = open('H:/代码练习/Deeplearning/data_txt_path/all_data_list_1.txt', 'a+') #创建并打开一个txt文件，a+表示打开一个文件并追加内容
listText.truncate(0)#清空txt文件里的内容
for file in files: #遍历文件夹中的文件
fileType = os.path.split(file) #os.path.split（）返回文件的路径和文件名，【0】为路径，【1】为文件名
if fileType[1] == '.txt': #若文件名的后缀为txt,则继续遍历循环，否则退出循环
continue
name = outer_path + folder + '/' +file #name 为文件路径和文件名+空格+label+换行
for j in range(len(label[i])):
target += str(label[i][j]) + ' '
name = name + ' ' + target + '\n'
# print(name)
# listText.write(name) # 在创建的txt文件中写入name
target = ''
i += 1
listText.write(name) #在创建的txt文件中写入name
listText.close() #关闭txt文件
print('down!')
print('****************')
outer_path = 'H:/代码练习/Deeplearning/dataset_1/' # 这里是你的图片路径
if __name__ == '__main__': #主函数
folderlist = os.listdir(outer_path)# 列举文件夹
for folder in folderlist: #遍历文件夹中的文件夹(若engagement文件夹中存在txt或py文件，则后面会报错）
generate(os.path.join(outer_path, folder))#调用generate函数，函数中的参数为：（图片路径+文件夹名，标签号）
The result after the code is run is as follows: serial number 1 is the path of the image, serial number 2 is the corresponding label, because I have a picture corresponding to 3 labels, so there are the following 3 values.
2 Randomly divide the image label data set in the txt file into a training set and a test set
After completing the first step, you need to randomly divide the image label data set in the txt file into a training set and a test set. After the division, two txt files, the training set and the test set, are generated. You can change the training set and test set according to your own needs. set proportions. The following is the code for dividing the data set (the path of the file needs to be changed according to the location of the file):
import os
import random
# 划分比例，训练集 : 验证集 = 8 : 2
split_rate = 0.2
class SplitFiles():
"""按行分割文件"""
def __init__(self, file_name):
"""初始化要分割的源文件名和分割后的文件行数"""
self.file_name = file_name
# def get_random(self):
# """生成随机数组，随机划分 （0，190001）txt标签行数， 7600测试集标签行数"""
# random_num = random.sample(range(0, 19001), 108)
#
# return random_num
def split_file(self):
if self.file_name and os.path.exists(self.file_name):
try:
with open(self.file_name) as f: # 使用with读文件
# temp_count = 1
file = f.readlines()
count = len(file)
eval_index = random.sample(file, k=int(count * split_rate)) # 从images列表中随机抽取 k 个图像名称
for index,image_path in enumerate(file):
if image_path in eval_index:
self.write_file('test', image_path)
else:
self.write_file('train', image_path)
# temp_count += 1
except IOError as err:
print(err)
else:
print("%s is not a validate file" % self.file_name)
def get_part_file_name(self, part_name):
""""获取分割后的文件名称：在源文件相同目录下建立临时文件夹temp_part_file，然后将分割后的文件放到该路径下"""
temp_path = os.path.dirname(self.file_name) # 获取文件的路径（不含文件名）
file_folder = temp_path
if not os.path.exists(file_folder): # 如果临时目录不存在则创建
os.makedirs(file_folder)
part_file_name = file_folder + "/" + str(part_name) + "_list_1.txt"
return part_file_name
def write_file(self, part_num, line):
"""将按行分割后的内容写入相应的分割文件中"""
part_file_name = self.get_part_file_name(part_num)
try:
with open(part_file_name, "a") as part_file:
part_file.writelines(line)
except IOError as err:
print(err)
if __name__ == "__main__":
file = SplitFiles(r'H:/代码练习/Deeplearning/data_txt_path/all_data_list_1.txt')
file.split_file()
Here I store the total data files and the divided data sets in a folder for later management (originally there were only 3 txt files, but I made two data sets, so there are 6 files).
3 Load the image label dataset in the txt file
After completing steps 1 and 2, the last step is to load the data. The code for loading the data is as follows, and you can call this class function to read the data later:
import os
import numpy as np
import torch
from torchvision import transforms
from PIL import Image
from torch.utils.data import DataLoader
from torch.utils.data import Dataset
# 我们读取图片的根目录， 在根目录下有所有图片的txt文件， 拿到txt文件后， 先读取txt文件， 之后遍历txt文件中的每一行， 首先去除掉尾部的换行符， 在以空格切分，前半部分是图片名称， 后半部分是图片标签， 当图片名称和根目录结合，就得到了我们的图片路径
class MyDataset(Dataset):
def __init__(self, img_path, transform=None):
super(MyDataset, self).__init__()
self.root = img_path
# self.txt_root = self.root + 'all_list.txt'
f = open(self.root, 'r')
data = f.readlines()
imgs = []
labels = []
# label_1,label_2,label_3 = [],[],[]
for line in data:
line = line.rstrip()
word = line.split()
imgs.append(os.path.join(self.root, word[1],word[2],word[3],word[0]))
# labels.append([float(word[1]),float(word[2]),float(word[3])])
labels.append([word[1],word[2],word[3]])
# label_1,label_2,label_3 = word[1],word[2],word[3]
# labels.append([[label_1],[label_2],[label_3]])
self.img = imgs
self.label = labels
self.transform = transform
# print(self.img)
# print(self.label)
def __len__(self):
return len(self.label)
return len(self.img)
def __getitem__(self, item):
img = self.img[item]
label = self.label[item]
# print(img)
img = Image.open(img).convert('RGB')
# 此时img是PIL.Image类型 label是str类型
if transforms is not None:
img = self.transform(img)
# print(img.max())
label = np.array(label).astype(np.float32)
label = torch.from_numpy(label)
return img, label
Note: The code above/below is that I loaded three labels at the same time, because I have three labels corresponding to one picture at the same time. If one picture corresponds to one label, it can be changed in the function of the following picture:
Because my label is a floating-point number, I will change it to a floating-point number type here. If it is an integer, you can change it in the position of the above code and the figure below.
After executing the code file in step 1 , label the image data set and save it as a txt file; execute the code file in step 2 to randomly divide the image label data set in the txt file into a training set and a test set; finally write the step 3 Load the image tag dataset code in the txt file, and then you can load your own dataset. The following is the deep learning training, call the above class to load the data to load the data, you can also write according to your own code, you can refer to the following example:
root_train = r'H:/代码练习/Deeplearning/data_txt_path/train_list_1.txt'
root_test = r'H:/代码练习/Deeplearning/data_txt_path/test_list_1.txt'
#将图像的像素值归一化到[-1,1]之间
normalize = transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
train_transform = transforms.Compose([
transforms.Resize((224,224)),
# transforms.RandomVerticalFlip(),
transforms.ToTensor(),
normalize])
val_transform = transforms.Compose([
transforms.Resize((224,224)),
# transforms.RandomVerticalFlip(),
transforms.ToTensor(),
normalize])
train_dataset = MyDataset(root_train,transform=train_transform)
val_dataset = MyDataset(root_test,transform=val_transform)
train_dataloader = DataLoader(dataset=train_dataset,batch_size=16,shuffle=True)
val_dataloader = DataLoader(dataset=val_dataset,batch_size=16,shuffle=True)
device = 'cuda' if torch.cuda.is_available() else 'cpu'
for batch, (x, y) in enumerate(data_loader):
image, y= x.to(device), y.to(device)
Reference source: Making Datasets (2) -- Labeling the Image Dataset and saving it as a txt file
Python divides the data set file (txt label file is randomly divided in proportion)