一、下载

首先下载2.1版本的源码和数据集，其中“mask_rcnn_balloon”是训练好的权重，如果是自己训练就不用下载。

2.1版本链接：https://github.com/matterport/Mask_RCNN/releases

还要下载2.0版本中的“mask_rcnn_coco.h5”文件，该文件是coco数据集的预训练权重，用于模型初始化

2.0版本链接：https://github.com/matterport/Mask_RCNN/releases/tag/v2.0

下载后，我将“balloon”文件夹和“mask_rcnn_coco.h5”预训练权重移入到“Mask_RCNN-2.1”这个工程中（注意：绝对路径中不要包含中文）

将mask_rcnn_coco.h5复制一份到如下路径中

二、安装环境

使用anaconda创建一个python为3.5的虚拟环境

conda create -n MASK_RCNN python=3.5

安装一些包

pip install pip install TensorFlow==1.3 -i http://pypi.douban.com/simple --trusted-host pypi.douban.com
pip install Keras==2.1.6 -i http://pypi.douban.com/simple --trusted-host pypi.douban.com
pip install Pillow -i http://pypi.douban.com/simple --trusted-host pypi.douban.com
pip install scikit-image -i http://pypi.douban.com/simple --trusted-host pypi.douban.com
pip install h5py -i http://pypi.douban.com/simple --trusted-host pypi.douban.com
pip install scipy==1.2.1 -i http://pypi.douban.com/simple --trusted-host pypi.douban.com
pip install matplotlib -i http://pypi.douban.com/simple --trusted-host pypi.douban.com

三、运行demo.py

工程中只有demo.ipynb：

需要使用如下命令将demo.ipynb转为demo.py

jupyter nbconvert --to script demo.ipynb

此时运行demo.py会有如下错误：

安装pycocotools

python -m pip install pycocotools

将demo.py中的第25行注释

安装IPython和parso：

conda install IPython
conda install parso==0.3.1

运行结果：

四、安装Labelme

conda install pyqt
pip install labelme
pip install Pillow==5.3.0

如果安装过程中出现“ImportError: cannot import name 'FormatControl'”错误

可以做如下操作：

（1）执行命令： python -m ensurepip --default-pip

（2）下载 get-pip.py 文件（https://bootstrap.pypa.io/get-pip.py），复制网页上全部代码，粘贴到新建文件get-pip.py

（3）cd到 get-pip.py的路径下，执行命令： python get-pip.py

五、使用Labelme标注图片

生成xxx.json文件的过程

打开后的界面如下：

将待标注的图片存放在文件夹中

点击“Open Dir”来打开存放有数据的文件夹

打开数据后，在图片上点击鼠标右键，选择“Create Polygons”，然后将待分割的目标包围起来

可以标注多个类别的目标：

标注完成后，Ctrl+S保存标注结果（注意保存json的名字要和标注的图片名字一致）

2. 生成xxx_json文件夹的过程

在储存有json的文件夹的路径下执行如下命令：

labelme_json_to_dataset 1.json

会生成一个“1_json”的文件夹：

该文件夹中内容如下，可以看到缺少“info.yaml”文件

找到json_to_dataset.py文件，我的在“C:\ProgramData\Anaconda3\envs\MASK_RCNN\Lib\site-packages\labelme\cli”路径下

打开该文件，添加如下代码：（注意添加的位置）

logger.warning('info.yaml is being replaced by label_names.txt')
    info = dict(label_names=label_names)
    with open(osp.join(out_dir, 'info.yaml'), 'w') as f:
        yaml.safe_dump(info, f, default_flow_style=False)

    logger.info("Saved to: {}".format(out_dir))

然后别忘了在开头导入yaml

编辑后保存修改后的代码，我们将“1_json”删除，然后重新使用“labelme_json_to_dataset 1.json”命令来生成“1_json”，发现此时就有了“info.yaml”

如果觉得每次都要使用“labelme_json_to_dataset xxx.json”命令来生成对应的json文件夹太麻烦，可以使用如下代码实现批量转换：（注意修改代码中路径）

记得每标完一张就要保存一次，保存后会打上√

这里我标了10张图片，生成的json和json文件夹如下：

3. 整理目录

新建4个文件夹，分别命名为“pic”、“json”、“labelme_json”和“cv2_mask”

其中，“pic”文件夹存放原始图片

“json”文件夹存放json文件：

“labelme_json”文件夹存放“xxx_json”文件夹

“cv2_mask”文件夹存放每个labelme_json文件下的label文件，需要更改名称为xxx.png

我们可以使用如下代码来完成批量复制任务：（注意修改路径）

import os
import shutil
filename = "D:/czc/DiZaiDetect/Project/mydata/labelme_json/"  # 存放json转化得到的文件夹名称，需要保证没有隐藏的文件夹
fileList = os.listdir(filename)

"""
抽取json转化得到的5个文件中的label.png，并重新命名
"""
for i in range(len(fileList)):
    path = filename + fileList[i]
    # 如果不是文件夹，跳过
    if os.path.isfile(path):
        continue
    no = fileList[i][:-5]   # 从文件夹上取出文件名
    mask_source = path + "/label.png"
    mask_target = "D:/czc/DiZaiDetect/Project/mydata/cv2_mask/{}.png".format(no)  # 命名为“文件名.png”
    shutil.copy(mask_source, mask_target)  # 利用shutil直接copy过去

六、训练模型

工程如下：

运行“train_test.py”

代码如下：

# -*- coding: utf-8 -*-

import os
import sys
import random
import math
import re
import time
import numpy as np
import cv2
import matplotlib
import matplotlib.pyplot as plt
import tensorflow as tf
from mrcnn.config import Config
#import utils
from mrcnn import model as modellib,utils
from mrcnn import visualize
import yaml
from mrcnn.model import log
from PIL import Image
"""
加入自己类别名称
更改类别个数
"""

#os.environ["CUDA_VISIBLE_DEVICES"] = "0"
# Root directory of the project
ROOT_DIR = os.getcwd()

#ROOT_DIR = os.path.abspath("../")
# Directory to save logs and trained model
MODEL_DIR = os.path.join(ROOT_DIR, "logs")

iter_num=0

# Local path to trained weights file
COCO_MODEL_PATH = os.path.join(ROOT_DIR, "mask_rcnn_coco.h5")
# Download COCO trained weights from Releases if needed
if not os.path.exists(COCO_MODEL_PATH):
    utils.download_trained_weights(COCO_MODEL_PATH)


class ShapesConfig(Config):
    """Configuration for training on the toy shapes dataset.
    Derives from the base Config class and overrides values specific
    to the toy shapes dataset.
    """
    # Give the configuration a recognizable name
    NAME = "shapes"

    # Train on 1 GPU and 8 images per GPU. We can put multiple images on each
    # GPU because the images are small. Batch size is 8 (GPUs * images/GPU).
    GPU_COUNT = 1
    IMAGES_PER_GPU = 1

    # Number of classes (including background)
    NUM_CLASSES = 1 + 2  # background + 3 shapes

    # Use small images for faster training. Set the limits of the small side
    # the large side, and that determines the image shape.
    IMAGE_MIN_DIM = 320
    IMAGE_MAX_DIM = 384

    # Use smaller anchors because our image and objects are small
    RPN_ANCHOR_SCALES = (8 * 6, 16 * 6, 32 * 6, 64 * 6, 128 * 6)  # anchor side in pixels

    # Reduce training ROIs per image because the images are small and have
    # few objects. Aim to allow ROI sampling to pick 33% positive ROIs.
    TRAIN_ROIS_PER_IMAGE = 100

    # Use a small epoch since the data is simple
    STEPS_PER_EPOCH = 100

    # use small validation steps since the epoch is small
    VALIDATION_STEPS = 50


config = ShapesConfig()
config.display()

class DrugDataset(utils.Dataset):
    # 得到该图中有多少个实例（物体）
    def get_obj_index(self, image):
        n = np.max(image)
        return n

    # 解析labelme中得到的yaml文件，从而得到mask每一层对应的实例标签
    def from_yaml_get_class(self, image_id):
        info = self.image_info[image_id]
        with open(info['yaml_path']) as f:
            temp = yaml.load(f.read(), Loader=yaml.FullLoader)
            labels = temp['label_names']
            del labels[0]
        return labels

    # 重新写draw_mask
    def draw_mask(self, num_obj, mask, image,image_id):
        info = self.image_info[image_id]
        for index in range(num_obj):
            for i in range(info['width']):
                for j in range(info['height']):
                    at_pixel = image.getpixel((i, j))
                    if at_pixel == index + 1:
                        mask[j, i, index] = 1
        return mask

    # 重新写load_shapes，里面包含自己的自己的类别
    # 并在self.image_info信息中添加了path、mask_path 、yaml_path
    # yaml_pathdataset_root_path = "/tongue_dateset/"
    # img_floder = dataset_root_path + "rgb"
    # mask_floder = dataset_root_path + "mask"
    # dataset_root_path = "/tongue_dateset/"
    def load_shapes(self, count, img_floder, mask_floder, imglist, dataset_root_path):
        """Generate the requested number of synthetic images.
        count: number of images to generate.
        height, width: the size of the generated images.
        """
        # Add classes
        #self.add_class("shapes", 1, "tank") # 黑色素瘤
        #
        self.add_class("shapes", 1, "category1") 
        self.add_class("shapes", 2, "category2")  

        for i in range(count):
            # 获取图片宽和高

            filestr = imglist[i].split(".")[0]
            
            mask_path = mask_floder + "/" + filestr + ".png"
            yaml_path = dataset_root_path + "labelme_json/" + filestr + "_json/info.yaml"
            print(dataset_root_path + "labelme_json/" + filestr + "_json/img.png")
            cv_img = cv2.imread(dataset_root_path + "labelme_json/" + filestr + "_json/img.png")

            self.add_image("shapes", image_id=i, path=img_floder + "/" + imglist[i],
                           width=cv_img.shape[1], height=cv_img.shape[0], mask_path=mask_path, yaml_path=yaml_path)

    # 重写load_mask
    def load_mask(self, image_id):
        """Generate instance masks for shapes of the given image ID.
        """
        global iter_num
        print("image_id",image_id)
        info = self.image_info[image_id]
        count = 1  # number of object
        img = Image.open(info['mask_path'])
        num_obj = self.get_obj_index(img)
        mask = np.zeros([info['height'], info['width'], num_obj], dtype=np.uint8)
        mask = self.draw_mask(num_obj, mask, img,image_id)
        occlusion = np.logical_not(mask[:, :, -1]).astype(np.uint8)
        for i in range(count - 2, -1, -1):
            mask[:, :, i] = mask[:, :, i] * occlusion

            occlusion = np.logical_and(occlusion, np.logical_not(mask[:, :, i]))
        labels = []
        labels = self.from_yaml_get_class(image_id)
        labels_form = []
        for i in range(len(labels)):
            if labels[i].find("category1") != -1:
                labels_form.append("category1")
            if labels[i].find("category2") != -1:
                labels_form.append("category2")
        class_ids = np.array([self.class_names.index(s) for s in labels_form])
        return mask, class_ids.astype(np.int32)

def get_ax(rows=1, cols=1, size=8):
    """Return a Matplotlib Axes array to be used in
    all visualizations in the notebook. Provide a
    central point to control graph sizes.

    Change the default size attribute to control the size
    of rendered images
    """
    _, ax = plt.subplots(rows, cols, figsize=(size * cols, size * rows))
    return ax

#基础设置
dataset_root_path="mydata/"
img_floder = dataset_root_path + "pic"
mask_floder = dataset_root_path + "cv2_mask"
imglist = os.listdir(img_floder)
count = len(imglist)

#train与val数据集准备
dataset_train = DrugDataset()
dataset_train.load_shapes(count, img_floder, mask_floder, imglist,dataset_root_path)
dataset_train.prepare()

print("dataset_train-->",dataset_train._image_ids)

dataset_val = DrugDataset()
dataset_val.load_shapes(7, img_floder, mask_floder, imglist,dataset_root_path)
dataset_val.prepare()

print("dataset_val-->",dataset_val._image_ids)

# Load and display random samples
#image_ids = np.random.choice(dataset_train.image_ids, 4)
#for image_id in image_ids:
#    image = dataset_train.load_image(image_id)
#    mask, class_ids = dataset_train.load_mask(image_id)
#    visualize.display_top_masks(image, mask, class_ids, dataset_train.class_names)

# Create model in training mode
model = modellib.MaskRCNN(mode="training", config=config,
                          model_dir=MODEL_DIR)

# Which weights to start with?
init_with = "coco"  # imagenet, coco, or last

if init_with == "imagenet":
    model.load_weights(model.get_imagenet_weights(), by_name=True)
elif init_with == "coco":
    # Load weights trained on MS COCO, but skip layers that
    # are different due to the different number of classes
    # See README for instructions to download the COCO weights
    model.load_weights(COCO_MODEL_PATH, by_name=True,
                       exclude=["mrcnn_class_logits", "mrcnn_bbox_fc",
                                "mrcnn_bbox", "mrcnn_mask"])
elif init_with == "last":
    # Load the last model you trained and continue training
    model.load_weights(model.find_last()[1], by_name=True)

# Train the head branches
# Passing layers="heads" freezes all layers except the head
# layers. You can also pass a regular expression to select
# which layers to train by name pattern.
model.train(dataset_train, dataset_val,
            learning_rate=config.LEARNING_RATE,
            epochs=10,
            layers='heads')



# Fine tune all layers
# Passing layers="all" trains all layers. You can also
# pass a regular expression to select which layers to
# train by name pattern.
model.train(dataset_train, dataset_val,
            learning_rate=config.LEARNING_RATE / 10,
            epochs=10,
            layers="all")

这里需要修改的地方可能有：

（1）在代码的第37行，表示需要使用预训练的 mask_rcnn_coco.h5 这个训练好的模型作为参数的初始化。

（2）在代码的第57行，表示背景 + 训练数据的类别数

（3）在代码的第177行，需要修改为我们制作数据源的路径。

（4）在代码的第121~122行和158~161行，改为你的类别

运行成功后可以看到如下打印信息：

七、使用模型

运行“test_model.py”

代码如下：

# -*- coding: utf-8 -*-


import os
import sys
import random
import math
import numpy as np
import skimage.io
import matplotlib
import matplotlib.pyplot as plt
import cv2
import time
from mrcnn.config import Config
from datetime import datetime 
# Root directory of the project
ROOT_DIR = os.getcwd()

# Import Mask RCNN
sys.path.append(ROOT_DIR)  # To find local version of the library
from mrcnn import utils
import mrcnn.model as modellib
from mrcnn import visualize
# Import COCO config
sys.path.append(os.path.join(ROOT_DIR, "samples/coco/"))  # To find local version
from samples.coco import coco


# Directory to save logs and trained model
MODEL_DIR = os.path.join(ROOT_DIR, "logs")
MODEL_WEIGHT = './logs/shapes20191104T0033/mask_rcnn_shapes_0010.h5'
"""
# Local path to trained weights file
COCO_MODEL_PATH = os.path.join(MODEL_DIR ,"mask_rcnn_coco.h5")
# Download COCO trained weights from Releases if needed
if not os.path.exists(COCO_MODEL_PATH):
    utils.download_trained_weights(COCO_MODEL_PATH)
    print("cuiwei***********************")
"""
# Directory of images to run detection on
IMAGE_DIR = os.path.join(ROOT_DIR, "images")

class ShapesConfig(Config):
    """Configuration for training on the toy shapes dataset.
    Derives from the base Config class and overrides values specific
    to the toy shapes dataset.
    """
    # Give the configuration a recognizable name
    NAME = "shapes"

    # Train on 1 GPU and 8 images per GPU. We can put multiple images on each
    # GPU because the images are small. Batch size is 8 (GPUs * images/GPU).
    GPU_COUNT = 1
    IMAGES_PER_GPU = 1

    # Number of classes (including background)
    NUM_CLASSES = 1 + 2  # background + 3 shapes

    # Use small images for faster training. Set the limits of the small side
    # the large side, and that determines the image shape.
    IMAGE_MIN_DIM = 320
    IMAGE_MAX_DIM = 384

    # Use smaller anchors because our image and objects are small
    RPN_ANCHOR_SCALES = (8 * 6, 16 * 6, 32 * 6, 64 * 6, 128 * 6)  # anchor side in pixels

    # Reduce training ROIs per image because the images are small and have
    # few objects. Aim to allow ROI sampling to pick 33% positive ROIs.
    TRAIN_ROIS_PER_IMAGE =100

    # Use a small epoch since the data is simple
    STEPS_PER_EPOCH = 100

    # use small validation steps since the epoch is small
    VALIDATION_STEPS = 50

#import train_tongue
#class InferenceConfig(coco.CocoConfig):
class InferenceConfig(ShapesConfig):
    # Set batch size to 1 since we'll be running inference on
    # one image at a time. Batch size = GPU_COUNT * IMAGES_PER_GPU
    GPU_COUNT = 1
    IMAGES_PER_GPU = 1

config = InferenceConfig()

# Create model object in inference mode.
model = modellib.MaskRCNN(mode="inference", model_dir=MODEL_DIR, config=config)

# Load weights trained on MS-COCO
model.load_weights(MODEL_WEIGHT, by_name=True)

# COCO Class names
# Index of the class in the list is its ID. For example, to get ID of
# the teddy bear class, use: class_names.index('teddy bear')
class_names = ['BG', 'category1', 'category2']
# Load a random image from the images folder
file_names = next(os.walk(IMAGE_DIR))[2]
image = skimage.io.imread(os.path.join(IMAGE_DIR, random.choice(file_names)))

a=datetime.now() 
# Run detection
results = model.detect([image], verbose=1)
b=datetime.now() 
# Visualize results
print("time:",(b-a).seconds)
r = results[0]
print (r)
visualize.display_instances(image, r['rois'], r['masks'], r['class_ids'], 
                            class_names, r['scores'])

这里需要修改的地方可能有：

（1）第31行，替换为你的模型的路径

（2）第41行，替换为你的待识别图片文件夹的路径

（2）第57行，替换为你的类别数

（3）第96行，替换为你的类别名称

我标了10张图片的训练结果如下，效果很差，大家可以试试增加数据量

跑通CPU版 Mask_RCNN