[NLP] Q & A knowledge-based visual map

Q & A knowledge-based visual map

Foreword

We know now the mainstream VQA are neural networks, RNN signature sequence input or LSTM, do a classic such as Show Attend and Tell, Top down Attention attention on this and other mechanisms, are based on neural network to do a visual quiz, I am here to do based on the knowledge map, it is bound to involve ontology modeling, due to limited time, so I chose a relatively simple picture ontology modeling, and modeling only one picture. But if we can achieve automatic ontology modeling from pictures, information and labeling information for the entire data set modeling (human, financial enough, then), or by some method of drawing, then I have a feeling this is a process of great significance. Therefore, I propose an idea knowledge and simply achieve at Demo, do not apply on the project.

I chose this course mapping knowledge and motivation to do this project are from:

•Reasoning-RCNN: Unifying Adaptive Global Reasoning into Large-scale Object Detection

effect

Ontology Modeling Pictures

This picture is a picture from COCO data set, I carefully selected to find such a relatively simple Ha, process modeling is to use the mouse a little Protege is complete, it is mainly the relationship between "the left" , "the right side", "the top", "at the bottom", "lying", etc. such keywords describing the location of the last-established body shown in FIG. visualization:

Question Analysis

We enter a question from the command line, it certainly involves a machine to understand that we ask the right questions.

First, be sure the standard Jieba word, this is not explained.

and then? We want to know what he asked, drawing is asked what is inside it, or ask Turi how many cats do. So I go to match how much and what the words appear under each word as a question type. To do different treatments for each type of question.

What: 1

Where: 2

How much: 3

Of course, this is certainly not fully match to match, so it is best to train a Bayesian classifier to classify the results obtained jieba word corresponding to the type of questions.

Similarly, to match the name of the entity according to predefined entities, which got the question of entities and types of questions.

Then according to the order they appear, they can be matched with the issue of pre-defined, each corresponding to one kind of problem has predefined query, so to achieve the ultimate inquiry.

NLP should give this one a relatively large number of people do know, I do not do this so that relatively simple.

Inquire

SELECT ?object
FROM <http://test.com>
where {
    <http://example.com/{subject}> <http://example.com/{subject}> ?object
};

Different problems query, here is an example of the object of the investigation, to get results.

Vision Visualization

COCO data set provides a marked json format, use COCOAPI we get the convex hull Format Callout, and then use Opencv profile picture look just fine, as the dynamic effect is waitkey.

Code:

'''
@Descripttion: This is Aoru Xue's demo, which is only for reference.
@version: 
@Author: Aoru Xue
@Date: 2019-11-08 13:10:19
@LastEditors: Aoru Xue
@LastEditTime: 2019-11-08 13:18:19
'''
from pycocotools.coco import COCO
import numpy as np
import cv2 as cv
import copy
color_map = {
            0:[220, 20, 60],
            1:[119, 11, 32],
            2:[0, 0, 142],
            3:[0,255,255],
            4:[250, 170, 30],
            5:[100, 170, 30]
        }
def id2name(vid):
    dic = {
        0:"Remote Control",
        1:"Remote Control",
        2:"Sofa",
        3:"Frame",
        4:"Cat",
        5:"Cat"
    }
    return dic[vid]
class Viz():
    def __init__(self):
        annFile = './instances_val2017.json'
        self.coco = COCO(annFile)
        self.sid = "000000039769" 
        self.annIds = self.coco.getAnnIds(imgIds=int(self.sid), iscrowd=None)            
        self.img = cv.imread("./" + self.sid + ".jpg")
        self.show = copy.copy(self.img)
    def anns(self,ann_id = 0):
        return self.coco.loadAnns(self.annIds[ann_id:ann_id + 1])
    def draw_(self,ann_id = 0,is_mask = False):
        instance = self.anns(ann_id = ann_id)[0]
    
        seg = instance["segmentation"][0]
        
        if is_mask :
            
            seg = np.array(seg,dtype = np.int).reshape(len(seg)//2,2)
            cv.drawContours(self.show,[seg],-1,tuple(color_map[ann_id]),-1)
        else:
            seg = np.array(seg,dtype = np.int).reshape(len(seg)//2,2)
            self.draw_hull(seg)
        print(seg[0])
        cv.putText(self.show,id2name(ann_id),tuple(seg[0]),cv.FONT_HERSHEY_SIMPLEX,1,(255,255,255),2)
        cv.imshow("result",self.show)
    def clear_(self):
        self.show = copy.copy(self.img)
        cv.imshow("result",self.show)
    
    def draw_hull(self,hulls):
        for i in range(len(hulls)):
            cv.line(self.show,(hulls[i][0],hulls[i][1]),(hulls[(i+1)%len(hulls)][0],hulls[(i+1)%len(hulls)][1]),(0,0,255),3)
        
if __name__ == "__main__":
    viz = Viz()
    
    
    #q = int(input("which one to show?"))
    q = 4
    viz.draw_(q,is_mask = True)
    cv.waitKey(0)
    
"""
遥控器-右：0
遥控器-左：1
沙发：2
左边猫子：4
右边猫子：5
"""

Item code

NLP late to perfect that part of what made github.