The SAM semantic segmentation model is open source. In the era of AIGC, the image matting tools are unified by the big model? (Down)

Hello everyone, I am Spirited Away, and I am very happy to share my learning experience on ChatGPT with you again today!

This time is "SAM Semantic Segmentation Model Open Source, AIGC Era, Image Matting Tools Are Unified by Large Models? "The final version of the series.

In the previous two sections, we introduced the Segment Anything model for segmenting everything, and the Segment-and-Track Anything model for segmenting video. It can only be said that the SAM segmentation model has improved too fast!

Don't tell me the name of the project brought today, if you were given an image segmentation algorithm model for matting, how would you optimize it?

1. Optimizing the application scenarios of the model

First of all, the easiest thing to think of is to change the application scenario from image to video, and then the Segment-and-Track Anything model of the novel was born.

Optimizing from the application scenario is a point of optimization, so what other innovative points of optimization are there?

2. Optimize the structure of the model itself

Of course the model itself! Then it is today's project. With the continuous popularization of 5G technology, we also have high requirements for computing speed, so today's project is to compress and lighten the model.

1. The birth of the MobileSAM model

I believe that everyone still remembers the large model where we used SAM to split everything. The following is the home page of the Demo deployment website.

Second, the structure of the MobileSAM model

The original SAM model mainly consists of two parts, image encoder and prompt-guided mask decoder.

Among them, the image encoder uses vit, the model parameters are very large, the model is relatively large, and the vit-h model has 632M;

The second part is very lightweight, including mask decoder and prompt encoder, with fewer model parameters, no more than 4M.

3. Deployment of the MobileSAM model

The difference between the segmentation model of MobileSAM and SAM is that mobileSAM can be deployed on CPU;

Due to the large size of the model, the SAM model can only be deployed on the GPU server in the cloud. We deploy the MobileSAM model on a local PC.

1. Conda creates a new environment named "mobileSAM", Python 3.8 version

（还包含PyTorch 1.7及以上版本和Torchvision 0.8及以上版本）

conda create -n mobileSAM python=3.8

2.激活创建的模型环境

conda activate mobileSAM

3.安装完成后，我们检查你的环境中的Python，PyTorch和Torchvision的版本，你可以使用以下命令：

python --version
python -c "import torch; print(torch.__version__)"
python -c "import torchvision; print(torchvision.__version__)"

4.安装Mobile Segment Anything的依赖:

pip install git+https://github.com/ChaoningZhang/MobileSAM.git

5.本地运行mobileSAM模型

cd app
python app.py

6.安装最新版本的gradio与timm安装包

pip3 install gradio timm

7.修改部分代码，进行服务链接部署

运行项目是部署于Google colab的云端实验室，使用gradio安装包进行快速web UI应用的搭建，创建部署调用的界面。

    def clear():
        return None, None

    def clear_text():
        return None, None, None

    # clear_btn_e.click(clear, outputs=[cond_img_e, segm_img_e])
    clear_btn_p.click(clear, outputs=[cond_img_p, segm_img_p])

demo.queue()
#设置share=True，将部署的demo网页链接设置为公网访问
demo.launch(share=True)

终端输出日志效果：