Get theproject codeandexperimental report, please >=click here=<
[0] Summary
In recent years, with the rapid rise of python, emerging disciplines such as artificial intelligence, image recognition, and computer vision have become hot. The development of Python is also accompanied by the development of its various derivative libraries and derivative editors. Among them, OpenCV is a classic cross-platform computer vision library based on BSD license (open source), which can run on Linux, Windows, Android and Mac. OS operating system. It is lightweight and efficient-consisting of a series of C functions and a small number of C++ classes, and provides interfaces for languages such as Python, Ruby, and MATLAB, and implements many general-purpose algorithms in image processing and computer vision.
This big assignment uses the Pycharm editor and the OpenCV class library to detect and recognize handwritten English letters based on convolutional neural networks. The technologies involved are: picture slicing, target detection, picture recognition, picture positioning, and rewriting the recognized letters into the picture.
Keywords: image cutting; handwriting recognition; letter recognition; OpenCV
[1] Chapter 1 Research Status of Object Detection
[1.1] Introduction to target detection
Target detection is a computer vision task that distinguishes targets in images or videos from other uninteresting areas, determines whether there is a target, determines the position of the target, and identifies the type of target. It is one of the core issues in the field of machine vision. Object detection has always been the most challenging problem in the field of machine vision due to the different appearance, shape, and posture of various objects, as well as the interference of factors such as illumination and occlusion during imaging. Therefore, target detection is a direction in computer vision.
In addition to image classification, the core problems to be solved in target detection are: ①The target may appear anywhere in the image. ② Targets come in various sizes. ③ Targets may have various shapes.
Object detection is a popular direction in computer vision and digital image processing. It is widely used in robot navigation, intelligent video surveillance, industrial inspection, aerospace and many other fields. It has important practical significance to reduce the consumption of human capital through computer vision. Therefore, target detection has become a research hotspot in theory and application in recent years. It is an important branch of image processing and computer vision, and it is also the core part of intelligent monitoring systems. At the same time, target detection is also a basic in the field of pan-identification Algorithms play a vital role in subsequent tasks such as face recognition, gait recognition, crowd counting, and instance segmentation. Due to the widespread use of deep learning, target detection algorithms have developed relatively rapidly.
[1.2] Development of target detection
[1.2.1] 2001
In 2001, Paul Viola and Michael Jones published an article of cross-age significance on CVPR. Later generations called the face detection algorithm in the article the Viola-Jones (VJ) detector. The VJ detector realized the real-time detection of faces for the first time under extremely limited computing resources 17 years ago. The speed is dozens or even hundreds of times faster than that of the detection algorithm in the same period, which greatly promoted the commercialization of face detection applications. . The idea of the VJ detector has profoundly influenced the development of the field of object detection for at least 10 years.
The VJ detector adopts the most traditional and conservative target detection method - sliding window detection, that is, traverses every scale and every pixel position in the image, and judges whether the current window is a face target one by one. This idea seems simple, but in fact the computational overhead is huge. The reason why the VJ face detector can realize real-time detection under limited computing resources has three key elements: fast calculation of multi-scale Haar features, effective feature selection algorithm and efficient multi-stage processing strategy.
[1.2.2] 2007
Deformable Part based Model (DPM) is the pinnacle of the development of detection algorithms based on classic manual features, and has won the detection champion of VOC for three consecutive years in 07, 08 and 2009. DPM was first proposed by P. Felzenszwalb and others at the University of Chicago, and was later improved by his doctoral student R. Girshick. In 2010, P. Felzenszwalb and R. Girshick were awarded the "Lifetime Achievement Award" by VOC. The main idea of DPM can be simply understood as splitting and transforming the detection problem of the whole target in the traditional target detection algorithm into the detection problem of each part of the model, and then aggregating the detection results of each part to obtain the final detection result, that is, " From the whole to the part, and from the part to the whole". For example, the detection problem of a car target can be decomposed into the detection problems of car windows, wheels, body and other parts under the idea of DPM, and the detection problem of pedestrians can be similarly decomposed into parts such as human head, limbs and torso. detection problem.
[1.2.3] Development of object detection
Since 2006, under the leadership of Hinton, Bengio, Lecun, etc., a large number of papers on deep neural networks have been published. Especially in 2012, Hinton's research group participated in the ImageNet image recognition competition for the first time, and won the first prize in one fell swoop through the constructed CNN network AlexNet. won the championship, and since then the neural network has received widespread attention.
The figure below shows the number of documents returned over the years for keywords related to target detection in Google Scholar Search. It can be seen that this field has attracted more and more attention from the academic community in the past 20 years. In 2018, nearly 1,200 related articles were published.
Since 2012, CNN has received widespread attention. The target detection algorithm is roughly divided into traditional detection methods before 2012 and detection methods based on deep learning that appeared after 2012.
Traditional methods such as VJ detection, HOG detection, and DPM algorithms.
The deep learning method is divided into two distinct technical paths: a single-stage detection algorithm and a two-stage detection algorithm.
In the past few years (2011-2013), the development of target detection algorithms has almost stagnated. Most people build complex models and more complex multi-model integration based on low-level feature expression to slowly improve detection accuracy. Since the deep convolutional network can learn a very robust and expressive feature representation, why not introduce it into the target detection process to extract features? When the convolutional neural network achieved great success in the ImageNet classification task in 2012,
Girshick et al. seized the opportunity to break the deadlock and first proposed the region convolutional network target detection framework (Regionswith CNN features, R-CNN) in 2014. . Since then, the field of object detection has developed at an unprecedented rate.
As the number of convolutional neural network layers continues to deepen, the abstraction ability, anti-translation ability and anti-scale change ability of the network are getting stronger and stronger. This is indeed a good thing for image classification tasks, but it brings another problem for detection tasks: the precise location of the target bounding box is becoming more and more difficult to obtain. As a result, a contradiction arises: if you want the detection algorithm to obtain stronger translation invariance and scale invariance, you must sacrifice the sensitivity of the feature to the position and scale of the target bounding box to a certain extent, that is, covariance; On the contrary, if you want to obtain more accurate target bounding box localization results, you must make some compromises in translation invariance and scale invariance. Therefore, if you want to effectively apply the convolutional network to the target detection problem, the key is how to effectively solve the contradiction between the translation/scale invariance of the deep network and the translation/scale covariance requirements in the target detection problem. This forces people to abandon the detection scheme based on feature map + sliding window, so as to turn their attention to finding a more accurate target candidate frame detection (Object Proposal Detection) algorithm.
In the past few years, with the development of deep learning target detection algorithms, many Object Proposal Detection algorithms have been proposed, such as Selective Search, Edge Boxes, BING, etc. It should be noted that Object Proposal-based detection algorithms are not exclusive to deep learning detection algorithms. As early as the traditional manual feature period, Uijlings and others tried to use Selective Search + Bag of Words (Bag of Words) features for object detection.
[2] Summary of target detection methods
[2.1] Classification of target detection algorithms
Algorithms based on region proposals, such as R-CNN, Fast R-CNN, Faster R-CNN, etc.
Detection algorithms based on target regression, such as YOLO, SSD, retinanet, EfficientDet.
Search-based target detection and recognition algorithms, AttentionNet, reinforcement learning.
Algorithms based on Anchor-free, such as CornerNet, CenterNet, FCOS, etc.
[2.2] R-CNN Algorithm Introduction
Object detection has two main tasks: object classification and localization. In order to accomplish these two tasks, R-CNN draws on the idea of sliding windows and adopts a scheme to identify regions, specifically:
input a picture, and extract 2000 candidate regions with independent categories (possible target regions) from the picture through a specified algorithm. );
for each candidate region, a convolutional neural network is used to obtain a feature vector;
for the corresponding feature vector of each region, a support vector machine (SVM) is used for classification, and a bounding box regression is used to adjust the size of the target bounding box.
[2.2.1] Extract candidate regions
R-CNN目标检测首先需要获取2000个目标候选区域，能够生成候选区域的方法很多，比如：
1 objectness
2 selective search
3 category-independen object proposals
4 constrained parametric min-cuts(CPMC)
[2.2.2] 提取特征向量
对于上述获取的候选区域，需进一步使用CNN提取对应的特征向量，使用模型AlexNet (2012)。（需要注意的是 Alexnet 的输入图像大小是 227x227，而通过 Selective Search 产生的候选区域大小不一，为了与 Alexnet 兼容，R-CNN 采用了非常暴力的手段，那就是无视候选区域的大小和形状，统一变换到 227x227 的尺寸）。
训练过程如下：
①有监督预训练：训练网络参数
样本：ImageNet；这里只训练和分类有关的参数，因为ImageNet数据只有分类，没有位置标注；图片尺寸调整为227x227；最后一层：4097维向量->1000向量的映射。
②特定样本下的微调 ：训练网络参数
采用训练好的AlexNet模型进行PASCAL VOC 2007样本集下的微调，学习率=0.001（PASCAL VOC 2007样本集上既有图像中物体类别标签，也有图像中物体位置标签）；mini-batch为32个正样本和96个负样本（由于正样本太少）；修改了原来的1000为类别输出，改为21维【20类+背景】输出。
[2.2.3] SVM分类
通过上述卷积神经网络获取候选区域的特征向量，进一步使用SVM进行物体分类，关键如下：
使用了一个SVM进行分类：向SVM输入特征向量，输出类别得分；用于训练多个SVM的数据集是ImageNet数据；将2000×4096维特征（2000个候选框，每个候选框获得4096的特征向量）与20个SVM组成的权值矩阵4096×20相乘（20种分类，SVM是二分类器，每个种类训练一个SVM，则有20个SVM），获得2000×20维矩阵表示每个建议框是某个物体类别的得分；分别对上述2000×20维矩阵中每列即每一类进行非极大值抑制剔除重叠建议框，得到该列即该类中得分最高的一些候选框。
[2.2.4] 边框修正
使用一个回归器进行边框回归：输入为卷积神经网络pool5层的4096维特征向量，输出为x、y方向的缩放和平移，实现边框的修正。在进行测试前仍需回归器进行训练。
在2014年R-CNN横空出世的时候，颠覆了以往的目标检测方案，精度大大提升。对于R-CNN的贡献，可以主要分为如下方面：
使用了卷积神经网络进行特征提取；使用bounding box regression进行目标包围框的修正；
但是，R-CNN仍然有一些问题：
耗时的selective search，对一张图像，需要花费2s；耗时的串行式CNN前向传播，对于每一个候选框，都需经过一个AlexNet提取特征，为所有的候选框提取特征大约花费47s；三个模块（CNN特征提取、SVM分类和边框修正）是分别训练的，并且在训练的时候，对于存储空间的消耗很大。
[2.3] Fast R-CNN算法介绍
面对R-CNN的缺陷，Ross在2015年提出的Fast R-CNN进行了改进，概述Fast R-CNN的解决方案：
首先采用selective search提取2000个候选框RoI；
使用一个卷积神经网络对全图进行特征提取；
使用一个RoI Pooling Layer在全图特征上摘取每一个RoI对应的特征；
分别经过为21和84维的全连接层（并列的，前者是分类输出，后者是回归输出）
Fast R-CNN通过CNN直接获取整张图像的特征图，再使用RoI Pooling Layer在特征图上获取对应每个候选框的特征，避免了R-CNN中的对每个候选框串行进行卷积（耗时较长）。
[2.4] YOLO v1算法介绍
这是继RCNN，fast-RCNN和faster-RCNN之后，rbg（RossGirshick）针对DL目标检测速度问题提出的另外一种框架。YOLO V1其增强版本GPU中能跑45fps，简化版本155fps。
[2.4.1] YOLO的核心思想
YOLO的核心思想就是利用整张图作为网络的输入，直接在输出层回归bounding box的位置和bounding box所属的类别；
faster RCNN中也直接用整张图作为输入，但是faster-RCNN整体还是采用了RCNN那种 proposal+classifier的思想，只不过是将提取proposal的步骤放在CNN中实现了,而YOLO则采用直接回归的思路。
[2.4.2] YOLO的实现方法
将一幅图像分成SxS个网格(grid cell)，如果某个object的中心 落在这个网格中，则这个网格就负责预测这个object。每个网格要预测B个bounding box，每个bounding box除了要回归自身的位置之外，还要附带预测一个confidence值。这个confidence代表了所预测的box中含有object的置信度和这个box预测的有多准两重信息，其值是这样计算的：
其中如果有object落在一个grid cell里，第一项取1，否则取0。 第二项是预测的bounding box和实际的groundtruth之间的IoU值。
每个bounding box要预测(x, y, w, h)和confidence共5个值，每个网格还要预测一个类别信息，记为C类。则SxS个网格，每个网格要预测B个bounding box还要预测C个categories。输出就是S x S x (5*B+C)的一个tensor。
[3] 程序实现
[3.1] 程序需求分析
随着计算机视觉方向的发展与各种开源库的涌现，目标检测与图像识别的步骤也越来越规范并且趋于简单化。
本次大作业采用Pycharm编辑器，应用Python的OpenCV图像处理库，基于深度学习的卷积神经网络来识别图像中的手写的大写英文字母。具体功能步骤是：对图像进行切片、目标检测、图像识别、图像定位、识别出来的字母重新写入到图片中。
[3.2] 程序开发技术介绍
[3.2.1] PyCharm编辑器
PyCharm是一种Python IDE，其带有一整套可以帮助用户在使用Python语言开发时提高其效率的工具，比如，调试、语法高亮、Project管理、代码跳转、智能提示、自动完成、单元测试、版本控制等等。此外，IDE提供了一些高级功能，以用于支持Django框架下的专业Web开发。同时支持Google App Engine，支持IronPython。这些功能在先进代码分析程序的支持下，使PyCharm成为Python专业开发人员和刚起步人员使用的有力工具。
PyCharm的主要功能有：
1、编码协助：其提供了一个带编码补全，代码片段，支持代码折叠和分割窗口的智能、可配置的编辑器，可帮助用户更快更轻松的完成编码工作。
2、项目代码导航：该IDE可帮助用户即时从一个文件导航至另一个，从一个方法至其申明或者用法甚至可以穿过类的层次。若用户学会使用其提供的快捷键的话甚至能更快。
3、代码分析：用户可使用其编码语法，错误高亮，智能检测以及一键式代码快速补全建议，使得编码更优化。
4、Python重构：有了该功能，用户便能在项目范围内轻松进行重命名，提取方法/超类，导入域/变量/常量，移动和前推/后退重构。
5、支持Django：有了它自带的HTML, CSS 和 JavaScript编辑器 ，用户可以更快速的通过Djang框架进行Web开发。此外，其还能支持CoffeeScript, Mako 和 Jinja2。
6、支持Google App引擎：用户可选择使用Python 2.5或者2.7运行环境，为Google APp引擎进行应用程序的开发，并执行例行程序部署工作。
7、集成版本控制登入，录出，视图拆分与合并–所有这些功能都能在其统一的VCS用户界面（可用于Mercurial, Subversion, Git, Perforce 和其他的 SCM）中得到。
8、图形页面调试器用户可以用其自带的功能全面的调试器对Python或者Django应用程序以及测试单元进行调整，该调试器带断点，步进，多画面视图，窗口以及评估表达式。
9、集成的单元测试用户可以在一个文件夹运行一个测试文件，单个测试类，一个方法或者所有测试项目。
[3.2.2] OpenCV图像处理库
OpenCV是一个基于BSD许可（开源）发行的跨平台计算机视觉库，可以运行在Linux、Windows、Android和Mac OS操作系统上。它轻量级而且高效——由一系列 C 函数和少量 C++ 类构成，同时提供了Python、Ruby、MATLAB等语言的接口，实现了图像处理和计算机视觉方面的很多通用算法。
OpenCV用C++语言编写，它的主要接口也是C++语言，但是依然保留了大量的C语言接口。该库也有大量的Python、Java and MATLAB/OCTAVE（版本2.5）的接口。这些语言的API接口函数可以通过在线文档获得。如今也提供对于C#、Ch、Ruby,GO的支持。
所有新的开发和算法都是用C++接口。一个使用CUDA的GPU接口也于2010年9月开始实现。
OpenCV于1999年由Intel建立，如今由Willow Garage提供支持。OpenCV是一个基于BSD许可（开源）发行的跨平台计算机视觉库，可以运行在Linux、Windows和Mac OS操作系统上。它轻量级而且高效——由一系列 C 函数和少量 C++ 类构成，同时提供了Python、Ruby、MATLAB等语言的接口，实现了图像处理和计算机视觉方面的很多通用算法。最新版本是3.4，2017年12月23日发布。
OpenCV 拥有包括 500 多个C函数的跨平台的中、高层 API。它不依赖于其它的外部库——尽管也可以使用某些外部库。
OpenCV 为Intel Integrated Performance Primitives（IPP）提供了透明接口。这意味着如果有为特定处理器优化的IPP库，OpenCV将在运行时自动加载这些库。
[3.3] 程序代码实现
[3.3.1] 程序文件介绍
程序的文件目录如图3-3所示。
其中，my_net2文件夹中，存放的是卷积神经网络的权重参数。
Cnn.pyfind_charater.pytotal.pywrite.py分别是编写的4个python文件。
First.jpg是文件进行识别的初始图片，result.jpg是文件识别之后导出的文件。
[3.3.2] 程序算法介绍
①. 首先，在total文件中，导入各个文件。
②. 调用cut（）函数。主要是对图像进行处理，灰度处理，高斯滤波虚化，二值化，之后对图片的轮廓进行检测识别，最后对图片进行切片。
在cut函数中，首先调用cv2.imread（）读取图片，然后通过cv2.cvtColor()函数来生成灰度图，通过cv2.GaussianBlur()来进行高斯滤波虚化，通过cv2.threshold(gray, 100, 255, 0)对图像进行二值化操作，最后用cv2.findContours()检测字母的轮廓并对图片进行提取切片。
③．调用reshape（）函数。对图像数组的格式进行变换，把28*28的二维数组改变成一维。
④．调用cnn（）函数。CNN是卷积神经网络，cnn（）函数的目的在于，把切片好的一组图片，送进cnn（）函数里面进行处理、比对标签，输出数组。
⑤．然后调用write（）函数。把上一步cnn（）函数导出的pre数组作为输入比对，比对出神经网络识别出的大写字母，最后把对应的大写字母写在原图片上面。
⑥．调用cv2.imwrite(‘result.png’,img)函数。把图片导出，图片名为result.jpg。
⑦．最后，cv2.imshow(‘a’,img)函数，显示导出的图片。
[3.3.3] Program code display
total.py file:
from find_charater import cut
from cnn import cnn
from writer import writer
import cv2
def main():
path='first.jpg'
num,p,q,img=cut(path)
num=num.reshape(-1,28*28)
pre=cnn(num)
img=writer(pre,p,q,img)
cv2.imwrite('result.png',img)
cv2.imshow('a',img)
cv2.waitKey(0)
cv2.destroyAllWindows()
main()
write.py file:
import cv2
def writer(num,p,q,img):
list = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H']
for i in range(len(p)):
a=list[num[i]]
img=cv2.putText(img,a,(p[i]-55,q[i]-55),cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 0), 2)
img = cv2.resize(img, (800, 600), interpolation=cv2.INTER_NEAREST)
return img
find_charater.py file:
import cv2 #opencv库
import matplotlib.pyplot as plt
import numpy as np
def cut(path):
img=cv2.imread(path) #读取
img = cv2.resize(img, (800, 600), interpolation=cv2.INTER_NEAREST)
a,b=img.shape[:2] #长和宽
MAX=a*b #面积
gray=cv2.cvtColor(img,cv2.COLOR_BGR2GRAY) #灰度图
gray = cv2.GaussianBlur(gray, (5, 5), 0) #高斯滤波虚化
ret, thresh = cv2.threshold(gray, 100, 255, 0) #二值化
image, contours, hierarchy = cv2.findContours(thresh, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE)
num=[]
p=[] #横坐标
q=[] #纵坐标
for i in range(len(contours)):
x, y, w, h = cv2.boundingRect(contours[i])
if cv2.contourArea(contours[i])>(MAX/500) and cv2.contourArea(contours[i])<(MAX/2):
cv2.rectangle(img,(x-25,y-25),(x+w+25,y+h+25),(150,199,199),4)
p.append(x)
q.append(y)
newimg=thresh[y-15:y + h+15,x-15:x + w+15]
newimg = cv2.resize(newimg, (28, 28), interpolation=cv2.INTER_LINEAR)
newimg=[(255-x)*1 for x in newimg]
num.append(newimg)
num=np.array(num)
return num ,p,q,img
cnn.py file:
import tensorflow as tf
import numpy as np
import os
import matplotlib.pyplot as plt
os.environ['TF_CPP_MIN_LOG_LEVEL'] = "4"
tf.set_random_seed(1)
np.random.seed(1)
LR = 0.001
tf_x = tf.placeholder(tf.float32, [None, 28*28])
image = tf.reshape(tf_x, [-1, 28, 28, 1])
tf_y = tf.placeholder(tf.int32, [None, 8])
def weight_variable(shape): # 初始化权重，初始化为接近0的很小的正值
initial = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(initial)
def bias_variable(shape): # 初始化偏置
initial = tf.constant(0.1, shape=shape)
return tf.Variable(initial)
def conv2d(x, W): # 卷积和池化，使用卷积步长为1（stride size）,0边距（padding size）池化用简单传统的2x2大小的模板做max pooling
return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
def max_pool_2x2(x):
return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
def cnn(img):
# 第一层 卷积层
W_conv1 = weight_variable([5, 5, 1, 16])
b_conv1 = bias_variable([16])
h_conv1 = tf.nn.relu(conv2d(image, W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1)
# 第二层 卷积层
W_conv2 = weight_variable([5, 5, 16, 32])
b_conv2 = bias_variable([32])
h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
h_pool2 = max_pool_2x2(h_conv2)
keep_prob = tf.placeholder("float")
h_pool2 = tf.nn.dropout(h_pool2, keep_prob)
flat = tf.reshape(h_pool2, [-1, 7*7*32])
output = tf.layers.dense(flat, 8) # 输出层
loss = tf.compat.v1.losses.softmax_cross_entropy(onehot_labels=tf_y, logits=output) # 计算
train_op = tf.compat.v1.train.AdamOptimizer(LR).minimize(loss)
accuracy = tf.compat.v1.metrics.accuracy(labels=tf.argmax(tf_y, axis=1), predictions=tf.argmax(output, axis=1),)[1]
with tf.Session() as sess:
saver = tf.train.Saver()
init_op = tf.group(tf.compat.v1.global_variables_initializer(), tf.local_variables_initializer())
sess.run(init_op)
saver.restore(sess, 'my_net2/save_net.ckpt')
pre = sess.run(output, {
tf_x: img,keep_prob:1})
pre_y=np.argmax(pre,1)
return pre_y
[4] Experimental results
[4.1] Experimental results display
The picture to be recognized:
the picture exported by the program:
references
[1] Yao Min. Digital Image Processing (Second Edition) [M]. Beijing: Machinery Industry Press, 2012. [2]
Bradski [US], Kaehler [US]; Yu Shiqi, Liu Ruizhen translation. Learning OpenCV (Chinese version) [M]. Beijing: Tsinghua University Press, 2009.
[3] Chen Shengyong. Realization of computer vision technology based on OpenCV [M]. Beijing: Science Press, 2014.
[4] Zhou Zhihua. Analysis depth Learning: Convolutional Neural Network Principles and Visual Practice [M]. Beijing: Electronics Industry Press, 2018. [
5] Mao Xingyun. Introduction to OpenCV3 Programming [M]. Beijing: Electronics Industry Press, 2015.