A review of research on online sketch recognition

Summary

This paper reviews the current research status of online sketch recognition, including recognition timing, single stroke processing, single symbol recognition, image and text separation, and unconstrained stroke grouping. Finally, future development and key issues are summarized.

problem definition

Sketch recognition is to complete the mapping from handwriting space to cognitive space, that is, to give sketches specific semantic expressions in a specific application field.

The SBIM (sketch-based interfaces and modeling) conference discusses sketch-based modeling technology every year and publishes some of the latest results. In the GIS field, it is generally necessary to perform mapping or retrieval operations on topographic maps or aerial images. Hand-drawn mapping and new human-computer interaction integrated with speech recognition and other technologies provide new options for this operation.

First define the following terms formally:
a) Stroke . The trajectory of the pen tip movement from when the user puts down the pen to when he lifts the pen. It is the smallest unit for user input. The most common stroke form is a sequence composed of two-dimensional coordinate points. Some hardware devices also provide pressure, inclination and other information.
b) Basic primitives . Referred to as primitives, they are the smallest components of graphics, such as straight lines, arcs, circles, ellipses, etc. Usually a stroke drawn by the user includes one or more primitives.
c) Icon . Graphic symbols with fixed domain semantics can be divided into single-stroke symbols and multi-stroke symbols.

From different perspectives, sketch recognition can be divided into two-dimensional and three-dimensional, online (online) and offline (offline), symbolic sketches and non-symbolic sketches (line sketches) [7]. This article focuses on the recognition of online 2D symbolic sketches.

From the perspective of the complexity of the sketch composition level and the difficulty of processing, sketches can be divided into three categories:
single strokes, multi-stroke graphic symbols, and entire sketches composed of multiple symbols, as shown in Figure 1. The first two categories only include the recognition of a single graphic object, and the third category includes the segmentation and recognition of multiple objects, such as circuit diagrams, chemical molecular structure diagrams, etc. Since there are many graphic objects and complex structural relationships, higher-level semantic acquisition is involved. Research Researchers also call it sketch understanding [10] or semantic understanding of sketches [4].

Research status

2.1 Recognition timing

2.2 Single stroke processing

Stroke segmentation and primitive recognition
Pen gesture recognition

2.3 Single sketch symbol recognition

(icon recognition)

2.4 Sketch understanding

Image and text separation
a) Classification based on single stroke features.
b) Method based on stroke group features.
c) Context-based joint classification.

Stroke grouping
a) Context-based method
b) Perception-based method
c) Search-based method
d) Feature-based method

Challenges

Although there is some public sketch data, most of it is a single shape and lacks a public test library of stroke groups.
The main directions that require further research in the future are: a) Users hope to have greater freedom
in drawing while their sketches can be understood by computers . Online sketch understanding methods should support users' customized behaviors in more aspects . b) Recognition of partial sketch symbols. Currently, most methods for this problem are structural methods . How to use statistical methods to enhance the robustness of partial icon recognition is worthy of further study. c) Processing of complex graphics and text layouts, such as graphics and text, and text in multiple languages ​​and in any direction. d) Application and verification of new machine learning methods, such as deep learning, incremental learning, subspace learning, online learning, etc. e) Error handling. Although the recognition rate is an important indicator of the algorithm, the current technology is not 100% correct. How to automatically locate and correct errors is also an important issue in improving system performance. f) Practical application system of sketch recognition method. Application is always the ultimate goal of sketch recognition technology

Emphasis on text Emphasis on text

bold text bold text

mark text

Delete text

quoted text

H ₂ O is a liquid.

The result of 2 ¹⁰ operation is 1024.

Insert links and images

Link: link .

picture:

Pictures with dimensions:

Centered image:

Centered and sized image:

Of course, in order to make it more convenient for users, we have added the image drag and drop function.

How to insert a beautiful piece of code

Go to the blog settings page and choose a code fragment highlighting style you like. The same highlighting style is shown below 代码片.

// An highlighted block
var foo = 'bar';

Generate a list that works for you

• project
  • project
    • project

1. Project 1
2. Project 2
3. Project 3

• Scheduled Tasks
• mission accomplished

Create a form

A simple table is created like this:

project	Value
computer	$1600
cell phone	$12
catheter	$1

Set content to center, left, or right

Use :---------:center
. Use :----------left.
Use ----------:right.

first row	The second column	third column
The first column of text is centered	The second column of text is on the right	The third column of text is on the left

SmartyPants

SmartyPants converts ASCII punctuation characters into "smart" typographic punctuation HTML entities. For example:

TYPE	ASCII	HTML
Single backticks	'Isn't this fun?'	‘Isn’t this fun?’
Quotes	"Isn't this fun?"	“Isn’t this fun?”
Dashes	-- is en-dash, --- is em-dash	– is en-dash, — is em-dash

Create a custom list

Markdown

Text-to- HTML conversion tool

Authors

John

Luke

How to create a footnote

A text with footnotes. ¹

Comments are also essential

Markdown converts text to HTML .

KaTeX math formula

You can render LaTeX mathematical expressions using KaTeX :

Gamma infrastructure $C\left(n\right)=(n-1)!\forall n\in N$ is the integral via Euler

$$C\left(z\right)={\int }_0^{\infty }{t}^{z-1}{e}^{-t}dt.$$

You can find more information about LaTeX mathematical expressions here .

New Gantt chart function to enrich your articles

• Regarding Gantt chart syntax, please refer here ,

UML diagram

UML diagrams can be used for rendering. Mermaid . For example, a sequence diagram generated below:

This will produce a flowchart. :

• Regarding Mermaid syntax, refer here ,

FLowchart flowchart

We will still support flowchart flow charts:

• Regarding Flowchart flowchart syntax, refer here .

Export and import

Export

If you want to try using this editor, you can edit whatever you want in this article. When you finish writing an article, find the article export in the upper toolbar and generate a .md file or .html file for local saving.

import

If you want to load a .md file you have written, you can select the import function on the upper toolbar to import the file with the corresponding extension and
continue your creation.

---

1. Explanation of footnote↩︎