Table of contents
CT image enhancement method: windowing method
Tumor extraction in tumor CT images
5. Construction of UNet neural network model
1. Semantic Segmentation
The third generation of image segmentation: semantic segmentation
Image segmentation (Image Segmentation) is an important basic technology in the field of computer vision. Image segmentation is the process of subdividing a digital image into multiple image sub-regions, making the image easier to understand by simplifying or changing the representation of the image. More simply, image segmentation is to attach a label to each pixel in a digital image, so that pixels with the same label have some common visual characteristics.
The modalities (formats) of medical images are more diverse, such as X-ray, CT, MRI, and ultrasound, etc., and of course some common RGB images (such as fundus retinal images) are also included. The information emphasis of different modal image responses is different. For example, X-ray observation of bones is more clear, CT can reflect tissue and organ bleeding, and MRI is suitable for observation of soft tissues. Moreover, the imaging results obtained by different types of imaging devices are somewhat different.
The first picture is the visualization result of a liver CT image slice, which has been converted into a grayscale image after preprocessing, and the dividing line between tissues and organs is blurred.
The second picture shows the liver CT images of different individuals, and the difference is huge, which brings great difficulties to the extraction of liver tissue.
2. Dataset
3D-IRCADb (3D Image Reconstruction for Comparison of Algorithm Database, 3D Image Reconstruction for Algorithm Database Comparison), the database consists of CT scans of 10 female and 10 male patients with liver tumors in 75% of the cases, and each folder corresponds to Depending on the patient, information about the image was provided, such as liver size (width, depth, height) or tumor location according to Couninurd segmentation. It also shows that liver segmentation can encounter significant difficulties due to contact with neighboring organs, atypical shape or density of the liver, or even artifacts in the images.
Link: https://pan.baidu.com/s/1P76AF-wvrFjElc6tR82tRA
Extraction code: 5el7
3. Data Augmentation
Image data processing steps
1. Data loading
2. CT image enhancement
3. Enhanced histogram equalization
4. Obtain the CT image corresponding to the tumor and the mask image of the liver tumor
5. Save the image
DICOM data reading
Use the pydicom library to read the file, the pydicom.dcmread function reads the file, sorts the file, the pixel_array attribute extracts the image pixel information, and displays the image:
batch data read
# 批量读取数据
img_slices = [pydicom.dcmread(os.path.join(data_path, file_name)) for file_name in os.listdir(data_path)]
os.listdir(data_path)
# 排序，避免CT图乱序
img_slices.sort(key=lambda x:x.InstanceNumber) # 顺序属性
img_array = np.array([i.pixel_array for i in img_slices]) # 提取像素值
CT image enhancement method: windowing method
The large range of CT images leads to poor contrast and needs to be processed for specific organs.
The physical meaning of the CT value is the degree of attenuation of the radiation intensity when the CT rays irradiate your body and the radiation passes through your body.
The characteristic of CT is that it can distinguish slight differences in the density of human tissue, and the standard adopted is determined according to the linear absorption coefficient ( μ value) of various tissues to x -rays .
Filter the pictures of the parts we want according to the Hu ( CT ) value, and all other parts are blacked out or whitened, in order to increase the contrast of the image. Use the windowing method. Observed CT value range: window width. The observed central CT value is the window level, and then the tumor part is binarized.
def windowing(img, window_width, window_center):
# params:需要增强的图片, 窗口宽度, 窗中心 通过窗口最小值来线性移动窗口增强
min_windows = float(window_center)-0.5*float(window_width)
new_img = (img-min_windows)/float(window_width)
new_img[new_img<0] = 0 # 二值化处理 抹白
new_img[new_img>1] = 1 # 抹黑
return (new_img * 255).astype('uint8') # 把数据整理成标准图像格式
img_ct = windowing(img_array, 500, 150)
Histogram equalization
Histogram equalization function: Divide the entire image into many small blocks (for example, 10*10 as a small block), and equalize each small block. It is more practical for images whose image histogram is not so single (for example, there are multi-peaks). In 0pencv, this method is: cv2.createCLAHE()
def clahe_equalized(imgs):
# 输入imgs的形状必须是3维
assert (len(imgs.shape) == 3)
# 定义均衡化函数
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))
# 新数组存放均衡化后的数据
img_res = np.zeros_like(imgs)
for i in range(len(imgs)):
img_res[i,:,:] = clahe.apply(np.array(imgs[i,:,:], dtype=np.uint8))
return img_res/255
plt.hist(img_array.reshape(-1,),bins=50) # reduce to one dimension, divided into 50 points
Get mask image depth
In order to speed up the training speed, the custom function realizes the extraction of the corresponding CT image (after processing) and the corresponding mask image of the liver tumor, and saves them in different folders, respectively, as the input and output of the model. Read tumor CT image:
tumor_slice = [pydicom.dcmread(os.path.join(data_path_mask, file_name)) for file_name in os.listdir(data_path_mask)]
#避免CT图乱序,顺序属性
tumor_slice.sort(key=lambda x: x.InstanceNumber)
#提取像素值
tumor_array = np.array([i.pixel_array for i in tumor_slice])
print(tumor_array.shape) # (129, 512, 512)
The white part is the tumor mask map, and the pixel array corresponding to the black part is all 0.
Tumor extraction in tumor CT images
index = [i.sum() > 0 for i in tumor_array] # 提取含肿瘤部分
# print(len(index)) # = tumor_array.shape[0] 129
# 提取掩模图的肿瘤部分
img_tumor = tumor_array[index]
# 对增强后的CT图提取肿瘤部分
img_patient = img_clahe[index]
save tumor data
# 设置保存文件路径
patient_save_path = r'E:/datasets/liver/tmp/patient/'
tumor_save_path = r'E:/datasets/liver/tmp/tumor/'
for path in [patient_save_path, tumor_save_path]:
if os.path.exists(path): # 判断文件夹是否存在
shutil.rmtree(path) # 清空
os.mkdir(path)
# 保留一个肿瘤的数据
# plt.imsave(os.path.join(patient_save_path, '0.jpg'), img_patient[0],cmap='gray')
for i in range(len(img_patient)):
plt.imsave(os.path.join(patient_save_path, f'{i}.jpg'), img_patient[i], cmap='gray')
plt.imsave(os.path.join(tumor_save_path, f'{i}.jpg'), img_tumor[i], cmap='gray')4. Data loading
data batch processing
Take the data in the 3dircadb1 folder as the experimental object, take the data of the first 1-10 patients as training samples, and take the data of 10-20 patients as test samples.
- 1. Read CT image fragments
- 2. Extract pixel value
- 3. CT image enhancement and equalization
- 4. Processing Tumor Mask Map
- 5. Sort the tumor map of each patient and extract the pixel value of the tumor segment
- 6. Extract the tumor part pixel number
- 7. Find the corresponding position in the CT image
- 8. Save all tumor data
def processImage(start, end):
for num in range(start, end):
print('正在处理第%d号病人' % num)
data_path = fr'G:/dataPack/基于深度学习的肝脏肿瘤图像分割/3dircadb1/3dircadb1.{num}/PATIENT_DICOM'
# 读取CT图图像片段
image_slices = [pydicom.dcmread(os.path.join(data_path, file_name)) for file_name in os.listdir(data_path)]
os.listdir(data_path)
image_slices.sort(key=lambda x: x.InstanceNumber) # 排序
# 提取像素值
image_array = np.array([i.pixel_array for i in image_slices])
# CT图增强-windowing
img_ct = windowing(image_array, 250, 0)
# 直方图均衡化
img_clahe = clahe_equalized(img_ct)
# 肿瘤掩模图处理
livertumor_path = fr'G:/dataPack/基于深度学习的肝脏肿瘤图像分割/3dircadb1/3dircadb1.{num}/MASKS_DICOM'
tumor_paths = [os.path.join(livertumor_path, i) for i in os.listdir(livertumor_path) if 'livertumor' in i]
# 重新排序
tumor_paths.sort()
# 提取所有肿瘤数据
j = 0
for tumor_path in tumor_paths:
print("正在处理第%d个肿瘤" % j)
tumor_slices = [pydicom.dcmread(os.path.join(tumor_path, file_name)) for file_name in
os.listdir(tumor_path)]
# 重新对肿瘤片段图排序
tumor_slices.sort(key=lambda x: x.InstanceNumber)
# 提取像素值
tumor_array = np.array([i.pixel_array for i in tumor_slices])
# 没有肿瘤的掩模图全为黑色，对应像素全为0
index = [i.sum() > 0 for i in tumor_array] # 提取肿瘤部分编号
img_tumor = tumor_array[index]
# 对增强后的CT图提取肿瘤
img_patient = img_clahe[index]
# 保存所有肿瘤数据
for i in range(len(img_patient)):
plt.imsave(os.path.join(patient_save_path, f'{num}_{j}_{i}.jpg'), img_patient[i], cmap='gray') # 保存CT图
plt.imsave(os.path.join(tumor_save_path, f'{num}_{j}_{i}.jpg'), img_tumor[i], cmap='gray') # 保存肿瘤掩模图
j += 1
return img_patient, img_tumorEnhanced CT image and tumor mask image saved after processing
Dataset loading
Define the Dataset data loader, further process each picture in the sample, convert it into an np array and convert the dimensions to facilitate UNet network training, and set the size of each batch of data to 2 by defining batch_size in Dataloader (to improve training efficiency from a computing perspective)
transform = transforms.Compose([
transforms.Resize((512, 512)),
transforms.ToTensor()
])First read the local file, set the file path of the training set and test set, and convert the tumor map into a single-channel array through torch.squeeze(anno_tensor).type(torch.long)
And binarize the tumor map
# 读取之前保存的处理后的病人CT图片与肿瘤图片
train_img_patient, train_img_tumor = processImage(1, 5)
test_img_patient, test_img_tumor = processImage(5, 7)
patient_images = glob.glob(r'E:\\datasets\liver\tmp\patient\*.jpg')
tumor_images = glob.glob(r'E:\\datasets\liver\tmp\tumor\*.jpg')
train_images = [p for p in patient_images if '1_' in p]
train_labels = [p for p in tumor_images if '1_' in p]
test_images = [p for p in patient_images if '2_' in p]
test_labels = [p for p in tumor_images if '2_' in p]
train_images = np.array(train_images)
train_labels = np.array(train_labels)
test_images = np.array(test_images)
test_labels = np.array(test_labels)
# img = Image.open(train_images[1])
# plt.imshow(img)
# plt.show()
class Portrait_dataset(data.Dataset):
def __init__(self, img_paths, anno_paths):
self.imgs = img_paths
self.annos = anno_paths
def __getitem__(self, index):
img = self.imgs[index]
anno = self.annos[index]
pil_img = Image.open(img)
img_tensor = transform(pil_img)
pil_anno = Image.open(anno)
anno_tensor = transform(pil_anno)
# 由于蒙版图都是黑白图，会产生channel为1的维度。经过转换后，256x256x1，这个1并不是我们需要的。
anno_tensor = torch.squeeze(anno_tensor).type(torch.long)
anno_tensor[anno_tensor > 0] = 1 # 语义分割。二分类。
return img_tensor, anno_tensor
def __len__(self):
return len(self.imgs)
BATCH_SIZE = 2
train_set = Portrait_dataset(train_images, train_labels)
test_set = Portrait_dataset(test_images, test_labels)
train_dataloader = data.DataLoader(train_set, batch_size=BATCH_SIZE, shuffle=True)
test_dataloader = data.DataLoader(test_set, batch_size=BATCH_SIZE)
img_batch, anno_batch = next(iter(train_dataloader))Read the data set through DataSet, and set the number of batch processing pictures = 2 through DataLoader (to improve computing efficiency)
Randomly select the patient's CT image and tumor image to display:
5. Construction of UNet neural network model
The Unet network structure is symmetrical, the blue/white box represents the feature map; the blue arrow represents 3x3 convolution, which is used for feature extraction; the gray arrow represents skip-connection, which is used for feature fusion; the red arrow represents pooling, which is used to reduce Dimension; the green arrow indicates upsampling upsample, which is used to restore the dimension; the cyan arrow indicates 1x1 convolution, which is used to output the result .
class downSample(nn.Module):
def __init__(self, in_channels, out_channels):
super(downSample, self).__init__()
# 两层*（卷积+激活）
self.conv_relu = nn.Sequential(
# padding=1，希望图像经过卷积之后大小不变
nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1),
nn.ReLU(inplace=True),
nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1),
nn.ReLU(inplace=True)
)
# 下采样（池化）
self.pool = nn.MaxPool2d(kernel_size=2)
def forward(self, x, is_pool=True):
if is_pool:
x = self.pool(x)
print("downSample forward x.shape",x.shape)
x = self.conv_relu(x)
print("downSample forward after conv_relu(x) x.shape",x.shape)
return x
# 上采样模型。卷积、卷积、上采样（反卷积实现上采样）
class upSample(nn.Module):
def __init__(self, channels):
# 两层*（卷积层+激活层）
super(upSample, self).__init__()
self.conv_relu = nn.Sequential(
nn.Conv2d(2 * channels, channels, kernel_size=3, padding=1),
nn.ReLU(inplace=True),
nn.Conv2d(channels, channels, kernel_size=3, padding=1),
nn.ReLU(inplace=True)
)
# 上采样激活层（ConvTransposed）将输出层的channel变成原来的一半
self.upConv_relu = nn.Sequential(
nn.ConvTranspose2d(channels, channels // 2,
kernel_size=3, stride=2,
padding=1, output_padding=1),
nn.ReLU(inplace=True)
)
def forward(self, x):
print("upSample - forward x.shape",x.shape)
x = self.conv_relu(x)
x = self.upConv_relu(x)
return x
# 创建Unet。要初始化上、下采样层，还有其他的一些层
class Unet(nn.Module):
def __init__(self):
super(Unet, self).__init__()
self.down1 = downSample(3, 64)
self.down2 = downSample(64, 128)
self.down3 = downSample(128, 256)
self.down4 = downSample(256, 512)
self.down5 = downSample(512, 1024)
self.up = nn.Sequential(
nn.ConvTranspose2d(1024, 512,
kernel_size=3,
stride=2,
padding=1,
output_padding=1),
nn.ReLU(inplace=True)
)
self.up1 = upSample(512)
self.up2 = upSample(256)
self.up3 = upSample(128)
self.conv_2 = downSample(128, 64) # 最后两层卷积
self.last = nn.Conv2d(64, 2, kernel_size=1) # 输出层为2分类
def forward(self, x):
x1 = self.down1(x, is_pool=False)
x2 = self.down2(x1)
x3 = self.down3(x2)
x4 = self.down4(x3) # ([512, 64, 64])
print("x4.shape",x4.shape) # x4.shape torch.Size([512, 64, 64])
x5 = self.down5(x4)
print("x5.shape",x5.shape) # x5.shape torch.Size([1024, 32, 32])
x6 = self.up(x5)
print("x6.shape",x6.shape) # x6.shape torch.Size([512, 64, 64])
# 将下采用过程x4的输出与上采样过程x5的输出做一个合并
x6 = torch.cat([x4, x6], dim=0) # dim=0
print("x6.shape",x6.shape) # x6.shape torch.Size([512, 128, 64])
x7 = self.up1(x6)
x7 = torch.cat([x3, x7], dim=0)
x8 = self.up2(x7)
x8 = torch.cat([x2, x8], dim=0)
x9 = self.up3(x8)
x9 = torch.cat([x1, x9], dim=0)
x10 = self.conv_2(x9, is_pool=False)
result = self.last(x10)
return resultImage shape change graph in neural network:
Single Image Prediction Map
