In the object detection problem discussed, we have always used square bounding boxes to annotate and predict objects in images. This section will explore the problem of semantic segmentation , which focuses on how to segment images into regions belonging to different semantic categories. Different from object detection, semantic segmentation can identify and understand the content of each pixel in the image: the annotation and prediction of its semantic regions are at the pixel level. The figure below shows the labels for dog, cat and background images in semantic segmentation. Compared with target detection, the pixel-level bounding boxes annotated by semantic segmentation are obviously more refined.
Image segmentation and instance segmentation
There are two other important issues in the field of computer vision that are similar to semantic segmentation, namely image segmentation and instance segmentation. Here we briefly distinguish them from semantic segmentation.
-
Image segmentation divides an image into component regions, and methods for this type of problem usually exploit the correlation between pixels in the image. It does not require label information about image pixels during training, and there is no guarantee that the segmented areas have the semantics we want during prediction. Taking the image above as input, image segmentation might divide the dog into two regions: one covering the predominantly black mouth and eyes, and the other covering the rest of the body, which is predominantly yellow.
-
Instance segmentation is also called simultaneous detection and segmentation . It studies how to identify pixel-level areas of each target instance in an image. Different from semantic segmentation, instance segmentation not only needs to distinguish semantics, but also distinguish different target instances. For example, if there are two dogs in an image, instance segmentation requires distinguishing which of the two dogs a pixel belongs to.