Use of the COCO dataset

• Compiling Pycocotools for Windows 10

• Introduction to COCO Database

The COCO database released by Microsoft provides object detection, segmentation and semantic textual description information for images in addition to images.
The URL of the COCO database is:

• MS COCO API - http://mscoco.org/
• Github URL - https://github.com/pdollar/coco
• More details about the API at the website: http://mscoco.org/dataset/#download

The database provides API interfaces of Matlab, Python and Lua. The API interfaces of matlab and python can provide complete image label data loading, parsing and visualization. In addition, the website also provides data-related articles, tutorials, etc.

When using the API and demo provided by COCO database, you need to download COCO's image and label data first.

• Install:
  1. First unzip the data file:
     • The image data is downloaded into coco/images/the folder
     • The tag data is downloaded into coco/the folder.
  2. matlab, add in the default path of matlabcoco/MatlabApi
  3. Python. Open a terminal, switch the path to coco/PythonAPI, and entermake
• Annotation information of COCO dataset

COCO's data annotation information includes:

• category flag
• number of categories
• pixel-level segmentation

import sys
sys.path.append('E:/xinlib')
from data import cocox
import zipfile

coco/images/View the data under the folder:

image_names = cocox.get_image_names()
image_names

['E:/Data/coco/images/test2017.zip',
 'E:/Data/coco/images/train2017.zip',
 'E:/Data/coco/images/unlabeled2017.zip',
 'E:/Data/coco/images/val2017.zip']

View the coco/files in the folder:

import os
dataDir = cocox.root

os.listdir(dataDir)

['annotations',
 'annotations_trainval2017.zip',
 'cocoapi',
 'images',
 'image_info_test2017.zip',
 'image_info_unlabeled2017.zip',
 'stuff_annotations_trainval2017.zip']

We only need to get the information of annotations (here all .zipend with ):

annDir = [z_name for z_name in os.listdir(dataDir) if z_name.endswith('.zip')]
annDir

['annotations_trainval2017.zip',
 'image_info_test2017.zip',
 'image_info_unlabeled2017.zip',
 'stuff_annotations_trainval2017.zip']

Unzip the annotations file:

for ann_name in annDir:
    z = zipfile.ZipFile(dataDir + '/' + ann_name)
    # 全部解压
    z.extractall(dataDir)

# 封装为函数
cocox.unzip_annotations()

# 删除标签的压缩文件
cocox.del_annotations()

Since the image data is relatively large, I will not decompress it, but you MXNet + zipfilecan .

Get image data

I take test2017.zipas an example :

image_names

['E:/Data/coco/images/test2017.zip',
 'E:/Data/coco/images/train2017.zip',
 'E:/Data/coco/images/unlabeled2017.zip',
 'E:/Data/coco/images/val2017.zip']

z = zipfile.ZipFile(image_names[0])

# 测试集的图片名称列表
z.namelist()

['test2017/',
 'test2017/000000259564.jpg',
 'test2017/000000344475.jpg',
 ...]

We can see that the first one is the directory name, and the next one is the image. Let's take a look at the first image:

from mxnet import image

r = z.read(z.namelist()[1])    # bytes
data = image.imdecode(r)       # 转换为 NDArray 数组，可以做数值运算
data

[[[ 87  94  78]
  [ 85  94  77]
  [ 87  96  79]
  ..., 
  [108  63  44]
  [252 244 233]
  [253 253 253]]

 [[ 86  95  76]
  [ 88  97  78]
  [ 85  94  75]
  ..., 
  [ 55  14   0]
  [150  94  81]
  [252 245 216]]

 [[ 90  99  78]
  [ 89  98  77]
  [ 89  98  77]
  ..., 
  [ 63  37  12]
  [ 90  30   6]
  [149  83  61]]

 ..., 
 [[ 86 104  82]
  [ 89 102  82]
  [ 84 102  80]
  ..., 
  [ 50  62  40]
  [ 50  61  45]
  [ 51  58  50]]

 [[ 89 101  77]
  [ 87  96  75]
  [ 89 104  83]
  ..., 
  [ 54  63  42]
  [ 49  53  39]
  [ 53  54  48]]

 [[ 96 100  77]
  [ 94  97  76]
  [ 88 103  82]
  ..., 
  [ 44  58  32]
  [ 45  57  37]
  [ 49  57  42]]]
<NDArray 480x640x3 @cpu(0)>

x = data.asnumpy()   # 转换为 array

# 显示图片
%pylab inline 
plt.imshow(x)

To do this, we can wrap it as an iterator:cocox.data_iter(dataType)

Get tag information (using the official given tutorial)

• Install the python API:

  pip install -U pycocotools

There are many pits under Windows: Windows 10 Compiling Pycocotools Stepping on the pits

%pylab inline
from pycocotools.coco import COCO
import numpy as np
import skimage.io as io
import matplotlib.pyplot as plt
import pylab
pylab.rcParams['figure.figsize'] = (8.0, 10.0)

There is a pit (caused by PIL) import skimage.io as iothat may report an error under Windows. My solution is:

• Uninstall Pillow first, then reinstall it.

• Interlude: PIL (Python Imaging Library) is a powerful and convenient image processing library for Python, and it is also famous. Pillow is a fork of PIL, but has now grown into a more dynamic image processing library than PIL itself.

dataDir = cocox.root
dataType = 'val2017'
annFile = '{}/annotations/instances_{}.json'.format(dataDir, dataType)

# initialize COCO api for instance annotations
coco=COCO(annFile)

loading annotations into memory...
Done (t=0.93s)
creating index...
index created!

COCO??

COCOis a class:

Constructor of Microsoft COCO helper class for reading and visualizing annotations.
:param annotation_file (str): location of annotation file
:param image_folder (str): location to the folder that hosts images.

display COCO categories and supercategories

cats = coco.loadCats(coco.getCatIds())
nms = [cat['name'] for cat in cats]
print('COCO categories: \n{}\n'.format(' '.join(nms)))

nms = set([cat['supercategory'] for cat in cats])
print('COCO supercategories: \n{}'.format(' '.join(nms)))

COCO categories: 
person bicycle car motorcycle airplane bus train truck boat traffic light fire hydrant stop sign parking meter bench bird cat dog horse sheep cow elephant bear zebra giraffe backpack umbrella handbag tie suitcase frisbee skis snowboard sports ball kite baseball bat baseball glove skateboard surfboard tennis racket bottle wine glass cup fork knife spoon bowl banana apple sandwich orange broccoli carrot hot dog pizza donut cake chair couch potted plant bed dining table toilet tv laptop mouse remote keyboard cell phone microwave oven toaster sink refrigerator book clock vase scissors teddy bear hair drier toothbrush

COCO supercategories: 
appliance sports person indoor vehicle food electronic furniture animal outdoor accessory kitchen

# get all images containing given categories, select one at random
catIds = coco.getCatIds(catNms=['person', 'dog', 'skateboard'])
imgIds = coco.getImgIds(catIds=catIds)
imgIds = coco.getImgIds(imgIds=[335328])
img = coco.loadImgs(imgIds[np.random.randint(0, len(imgIds))])[0]

img

{'license': 4,
 'file_name': '000000335328.jpg',
 'coco_url': 'http://images.cocodataset.org/val2017/000000335328.jpg',
 'height': 640,
 'width': 512,
 'date_captured': '2013-11-20 19:29:37',
 'flickr_url': 'http://farm3.staticflickr.com/2079/2128089396_ddd988a59a_z.jpg',
 'id': 335328}

The code given by the official needs to decompress the image dataset:

# load and display image
# use url to load image
# I = io.imread(img['coco_url'])
I = io.imread('%s/images/%s/%s' % (dataDir, dataType, img['file_name']))
plt.axis('off')
plt.imshow(I)
plt.show()

We can use the zipfilemodule to read the image directly without decompressing it:

image_names[-1]

'E:/Data/coco/images/val2017.zip'

val_z = zipfile.ZipFile(image_names[-1])
I = image.imdecode(val_z.read('%s/%s' % (dataType, img['file_name']))).asnumpy()
plt.axis('off')
plt.imshow(I)
plt.show()

load and display instance annotations

plt.imshow(I)
plt.axis('off')
annIds = coco.getAnnIds(imgIds=img['id'], catIds=catIds, iscrowd=None)
anns = coco.loadAnns(annIds)
coco.showAnns(anns)

initialize COCO api for person keypoints annotations

annFile = '{}/annotations/person_keypoints_{}.json'.format(dataDir, dataType)
coco_kps = COCO(annFile)

loading annotations into memory...
Done (t=0.43s)
creating index...
index created!

load and display keypoints annotations

plt.imshow(I)
plt.axis('off')
ax = plt.gca()
annIds = coco_kps.getAnnIds(imgIds=img['id'], catIds=catIds, iscrowd=None)
anns = coco_kps.loadAnns(annIds)
coco_kps.showAnns(anns)

initialize COCO api for caption annotations

annFile = '{}/annotations/captions_{}.json'.format(dataDir, dataType)
coco_caps = COCO(annFile)

loading annotations into memory...
Done (t=0.06s)
creating index...
index created!

load and display caption annotations

annIds = coco_caps.getAnnIds(imgIds=img['id'])
anns = coco_caps.loadAnns(annIds)
coco_caps.showAnns(anns)
plt.imshow(I)
plt.axis('off')
plt.show()

A couple of people riding waves on top of boards.
a couple of people that are surfing in water
A man and a young child in wet suits surfing in the ocean.
a man and small child standing on a surf board  and riding some waves
A young boy on a surfboard being taught to surf.