Source｜Heart of the Machine
What's next for CV researchers?
"The CV really doesn't exist now. <Quick Run>" This is Zhihu netizens' evaluation of a new Meta paper.
As the title says, this paper does only one thing: (zero-shot) splitting everything. Similar to "answer everything" that GPT-4 has already done.
According to Meta, this is the first base model dedicated to image segmentation. Since then, CV has also embarked on the road of "making an all-round model that unifies a certain (some? All?) tasks".
Prior to this, segmentation, as a core task of computer vision, has been widely used. However, creating an accurate segmentation model for a specific task usually requires highly specialized work by technical experts. In addition, this task also requires a large amount of domain-labeled data, and various factors limit the further development of image segmentation.
The new model released by Meta in the paper is called Segment Anything Model (SAM). "SAM has learned general concepts about objects, and it can generate masks for any object in any image or video, even objects and image types not encountered during training," they blogged. General enough to cover a wide range of use cases, and can be used out-of-the-box on new image "domains" without additional training." In the field of deep learning, this ability is often referred to as zero-shot transfer, which is also GPT- 4 One of the reasons for shocking the world.
Paper address:
https://arxiv.org/abs/2304.02643
Project address:
https://github.com/facebookresearch/segment-anything
Demo address:
https://segment-anything.com/
In addition to the model, Meta released an image annotation dataset Segment Anything 1-Billion (SA-1B), which it claims is the largest segmentation dataset ever created. This dataset is available for research purposes and the Segment Anything Model is available under an open license (Apache 2.0).
Let's look at the effect first. As shown in the animation below, SAM can automatically segment everything in the image very well:
SAM can also perform image segmentation based on prompt words. For example, enter the prompt word Cat, and SAM will draw boxes around several cats in the photo and achieve segmentation:
SAM can also prompt with interactive dots and boxes:
Additionally, SAM can generate several effective masks for ambiguous hints:
Jim Fan, an artificial intelligence scientist at Nvidia, said: "For this study of Meta, I think it is one of the GPT-3 moments in the field of computer vision. It has understood the general concept of objects, even for unknown objects, unfamiliar scenes (such as Underwater images) and ambiguous situations can also perform good image segmentation. Most importantly, the model and data are open source. IMHO, Segment-Anything has done everything (segmentation) very well gone."
Some netizens said that the Prompt paradigm in the NLP field has begun to extend to the CV field. It can be expected that this type of paradigm will usher in an explosion in the academic world this year.
Some netizens even said that the clam can't live anymore. Once SAM comes out, the CV really doesn't exist. Be careful when submitting to ICCV.
Others, however, said the model wasn't ideal for testing in a production environment. Maybe it will take time to solve this long-standing problem?
method introduction
There are roughly two methods to solve the segmentation problem before. The first is interactive segmentation, which allows segmentation of objects of any class but requires a human to guide the method by iteratively refining the mask. The second, automatic segmentation, allows segmentation of specific object categories defined in advance (e.g., cats or chairs), but requires a large number of manually annotated objects for training (e.g., thousands or even tens of thousands of examples of segmented cats). Neither of these approaches provides a general, fully automated approach to segmentation.
SAM nicely summarizes both approaches. It is a single model that can easily perform interactive and automatic segmentation. The promptable interface of this model allows users to use it in a flexible way, and a wide range of segmentation tasks can be accomplished by simply designing the correct prompts (clicks, boxes, text, etc.) for the model.
Taken together, these features enable SAMs to generalize to new tasks and domains. This flexibility is the first of its kind in the field of image segmentation.
Meta says they are inspired by cues in language models, so their trained SAMs can return valid segmentation masks for any cues, where cues can be foreground, background points, thick boxes or masks, free-form text, or Any information that indicates what to segment in the image. And the requirement of a valid mask simply means that even if the hint is ambiguous and might refer to multiple objects (e.g. a dot on a shirt might indicate either the shirt or the person wearing it), the output should be a reasonable mask (as in The above animation "SAM can also generate multiple effective masks for ambiguous prompts"). This task is used to pre-train models and solve general downstream segmentation tasks with hints.
As shown in the figure below, the Image Encoder generates one-shot embeddings for images, while the Lightweight Encoder converts hints into embedding vectors in real-time. These two sources of information are then combined in a lightweight decoder that predicts segmentation masks. After computing an image embedding, SAM can generate a segmentation within 50 milliseconds from any prompt in a web browser.
11 million images, 1B+ masks
The dataset was collected using SAM. The annotators use the SAM to interactively annotate the image, and then the newly annotated data in turn updates the SAM, which is mutually reinforcing.
Using this method, it only takes about 14 seconds to interactively annotate a mask. Compared with previous large-scale segmentation data collection work, Meta's method is 6.5 times faster than COCO fully manual polygon-based mask annotation, and 2 times faster than the largest previous data annotation work, due to the results assisted by the SAM model .
The final dataset exceeds 1.1 billion segmentation masks, collected on approximately 11 million licensed and privacy-preserving images. SA-1B has 400 times more masks than any existing segmentation dataset, and human evaluation studies confirm that these masks are of high quality and diversity, in some cases even comparable in quality to previous smaller, Masks are comparable to fully manually annotated datasets.
The SA-1B has images from photo contributors across multiple countries across different geographic regions and income levels, with more imagery and a better overall representation of all regions. A Meta-analysis of potential biases in their models in terms of perceived gender representation, perceived skin color, and age range found that SAM performed similarly across groups.
SA-1B can help other researchers train basic models for image segmentation. Meta further hopes that this data will form the basis for new datasets with additional annotations, such as textual descriptions associated with each mask.
future outlook
Through research and dataset sharing, Meta hopes to further accelerate research into image segmentation and more general image and video understanding. A hintable segmentation model can act as a component in a larger system to perform segmentation tasks. Composition is a powerful tool that allows a single model to be used in a scalable manner, with the potential to accomplish tasks unknown at the time the model was designed.
Meta anticipates that composable system designs based on techniques such as prompt engineering will support a wider range of applications than systems trained specifically for a fixed set of tasks. SAMs can be powerful components for AR, VR, content creation, scientific domains, and more general AI systems. For example, SAM can identify everyday objects through AR glasses and provide users with tips.
SAM also has the potential to help farmers in agriculture or assist biologists in their research.
In the future, we will see a tighter coupling between pixel-level image understanding and higher-level semantic understanding of visual content, unlocking more powerful AI systems.
Reply keywords in the background [ join the group ]
Join the NLP, CV, search promotion and job hunting discussion group
[1]https://ai.facebook.com/blog/segment-anything-foundation-model-image-segmentation/
[2]https://www.zhihu.com/question/593914819
