Realize Esp32-Cam model training and image recognition within half an hour

This project allows you to achieve model training and image recognition in half an hour, very simple. Click here to
play the effect video before starting

1. Web page display video stream

There are many ready-made resources, as long as you search a little bit and burn the program to Esp32-Cam, you can realize this function. You can go to study for details, so I won’t go into details here.

1. Linux-style routines

You can learn the routines of Anxinke official website, which is authoritative. Click to go
. The tutorial is very detailed. Brothers with Linux foundation can try it, otherwise don’t bother with it (such as vim editor use, shell script use, linux configuration, etc., it is very time-consuming, and few people have tried it successfully)

2. MicroPython routines

This way is to make Esp32-Cam have a python environment and be able to run py files. click to go

Step 1. Download Thonny

Download link: https://thonny.org/

Step 2. Burn Esp32-Cam firmware

Using Thonny, if the boot.py file cannot be displayed after burning the firmware, then there should be a problem with the base board, and you can buy the corresponding base board specified, but in fact, use the USB to ttl, and the DuPont line should be connected to 5V, GND, TXD and RXD.

Step 3. Run the corresponding code

import socket
import network
import camera
import time


# 连接wifi
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
if not wlan.isconnected():
    print('connecting to network...')
    wlan.connect('dongfeiqiu', 'wangmingdong1225')
    
    while not wlan.isconnected():
        pass
print('网络配置:', wlan.ifconfig())
 
 
# 摄像头初始化
try:
    camera.init(0, format=camera.JPEG)
except Exception as e:
    camera.deinit()
    camera.init(0, format=camera.JPEG)


# 其他设置：
# 上翻下翻
camera.flip(1)
#左/右
camera.mirror(1)

# 分辨率
camera.framesize(camera.FRAME_HVGA)
# 选项如下：
# FRAME_96X96 FRAME_QQVGA FRAME_QCIF FRAME_HQVGA FRAME_240X240
# FRAME_QVGA FRAME_CIF FRAME_HVGA FRAME_VGA FRAME_SVGA
# FRAME_XGA FRAME_HD FRAME_SXGA FRAME_UXGA FRAME_FHD
# FRAME_P_HD FRAME_P_3MP FRAME_QXGA FRAME_QHD FRAME_WQXGA
# FRAME_P_FHD FRAME_QSXGA
# 有关详细信息，请查看此链接：https://bit.ly/2YOzizz

# 特效
camera.speffect(camera.EFFECT_NONE)
#选项如下：
# 效果\无（默认）效果\负效果\ BW效果\红色效果\绿色效果\蓝色效果\复古效果
# EFFECT_NONE (default) EFFECT_NEG \EFFECT_BW\ EFFECT_RED\ EFFECT_GREEN\ EFFECT_BLUE\ EFFECT_RETRO

# 白平衡
# camera.whitebalance(camera.WB_HOME)
#选项如下：
# WB_NONE (default) WB_SUNNY WB_CLOUDY WB_OFFICE WB_HOME

# 饱和
camera.saturation(0)
#-2,2（默认为0）. -2灰度
# -2,2 (default 0). -2 grayscale 

# 亮度
camera.brightness(0)
#-2,2（默认为0）. 2亮度
# -2,2 (default 0). 2 brightness

# 对比度
camera.contrast(0)
#-2,2（默认为0）.2高对比度
#-2,2 (default 0). 2 highcontrast

# 质量
camera.quality(10)
#10-63数字越小质量越高

# socket UDP 的创建
s = socket.socket(socket.AF_INET,socket.SOCK_DGRAM,0)

try:
    while True:
        buf = camera.capture()  # 获取图像数据
        s.sendto(buf, ("192.168.31.53", 9090))  # 向服务器发送图像数据
        time.sleep(0.1)
except:
    pass
finally:
    camera.deinit()

3. Arduino-style routines

This is also the simplest implementation routine I found, and there are many resources, and the language involved is mainly C++. click to go

Step 1. Download Arduino

Download address: click to go

Step 2. Install the Esp32-Cam library

Method 1: Install in IDE.
(1). File→Preferences→Additional development board manager URL, modify the URL to

https://arduino.esp8266.com/stable/package_esp8266com_index.json
https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json

(2). Tools→Development Board→Development Board Manager, search for esp32, click to install
Method 2: Download the zip archive from github as a library
Download address: Click to go to
download the zip archive, project → include library → add .ZIP library

Step 3. Select a routine

Tools→Development Board→esp32→AI Thinker ESP32-CAM
Fill wifi and password in the following positions

const char* ssid = "Your wifi name";
const char* password = "wifi password";

The complete code is intercepted as follows

#include "esp_camera.h"
#include <WiFi.h>

//
// WARNING!!! Make sure that you have either selected ESP32 Wrover Module,
//            or another board which has PSRAM enabled
//

// Select camera model
//#define CAMERA_MODEL_WROVER_KIT
//#define CAMERA_MODEL_ESP_EYE
//#define CAMERA_MODEL_M5STACK_PSRAM
//#define CAMERA_MODEL_M5STACK_WIDE
#define CAMERA_MODEL_AI_THINKER

#include "camera_pins.h"

const char* ssid = "Your wifi name";
const char* password = "wifi password";

void startCameraServer();

void setup() {
    
    
  Serial.begin(115200);
  Serial.setDebugOutput(true);
  Serial.println();

  camera_config_t config;
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = Y2_GPIO_NUM;
  config.pin_d1 = Y3_GPIO_NUM;
  config.pin_d2 = Y4_GPIO_NUM;
  config.pin_d3 = Y5_GPIO_NUM;
  config.pin_d4 = Y6_GPIO_NUM;
  config.pin_d5 = Y7_GPIO_NUM;
  config.pin_d6 = Y8_GPIO_NUM;
  config.pin_d7 = Y9_GPIO_NUM;
  config.pin_xclk = XCLK_GPIO_NUM;
  config.pin_pclk = PCLK_GPIO_NUM;
  config.pin_vsync = VSYNC_GPIO_NUM;
  config.pin_href = HREF_GPIO_NUM;
  config.pin_sscb_sda = SIOD_GPIO_NUM;
  config.pin_sscb_scl = SIOC_GPIO_NUM;
  config.pin_pwdn = PWDN_GPIO_NUM;
  config.pin_reset = RESET_GPIO_NUM;
  config.xclk_freq_hz = 20000000;
  config.pixel_format = PIXFORMAT_JPEG;
  //init with high specs to pre-allocate larger buffers
  if(psramFound()){
    
    
    config.frame_size = FRAMESIZE_UXGA;
    config.jpeg_quality = 10;
    config.fb_count = 2;
  } else {
    
    
    config.frame_size = FRAMESIZE_SVGA;
    config.jpeg_quality = 12;
    config.fb_count = 1;
  }

#if defined(CAMERA_MODEL_ESP_EYE)
  pinMode(13, INPUT_PULLUP);
  pinMode(14, INPUT_PULLUP);
#endif

  // camera init
  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK) {
    
    
    Serial.printf("Camera init failed with error 0x%x", err);
    return;
  }

  sensor_t * s = esp_camera_sensor_get();
  //initial sensors are flipped vertically and colors are a bit saturated
  if (s->id.PID == OV3660_PID) {
    
    
    s->set_vflip(s, 1);//flip it back
    s->set_brightness(s, 1);//up the blightness just a bit
    s->set_saturation(s, -2);//lower the saturation
  }
  //drop down frame size for higher initial frame rate
  s->set_framesize(s, FRAMESIZE_QVGA);

#if defined(CAMERA_MODEL_M5STACK_WIDE)
  s->set_vflip(s, 1);
  s->set_hmirror(s, 1);
#endif

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    
    
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");

  startCameraServer();

  Serial.print("Camera Ready! Use 'http://");
  Serial.print(WiFi.localIP());
  Serial.println("' to connect");
}

void loop() {
    
    
  // put your main code here, to run repeatedly:
  delay(10000);
}

Step 4. View the running results

Tools→Serial monitor, then press the reset button of esp32-cam

Copy the URL and open it on the web page, you can watch the real-time content of the camera

2. Realize image recognition within half an hour

1. Web video streaming

Similar to the previous Arduino routine, but the included library is not the official library, but this: Click here
to download the zip library and include the library in the IDE operation, and then copy the following code as a new project ino file. Notice:To configure your own board, and then change to your own wifi and password

#include "eloquent.h"
#include "eloquent/networking/wifi.h"
#include "eloquent/vision/camera/esp32/webserver.h"
// 把 'm5wide' 替换成自己的模块，
// 支持的模块有 'aithinker', 'eye', 'm5stack', 'm5wide', 'wrover'
#include "eloquent/vision/camera/aithinker.h"//我用的是aithinker
void setup() {
    
    
    Serial.begin(115200);
    delay(2000);
    camera.jpeg();
    camera.qqvga();
    // 改成自己的wifi和密码
    while (!wifi.connectTo("Abc", "12345678"))
        Serial.println("Cannot connect to WiFi");
    while (!camera.begin())
        Serial.println("Cannot connect to camera");
    webServer.start();
    Serial.print("Camera web server started at http://");
    Serial.println(WiFi.localIP());
}
void loop() {
    
    
    // do nothing
}

After compiling and burning it on the esp32-cam board, open the serial monitor, get the URL (mine is 192.168.1.103), and then open it on the webpage, which is similar to the regular Arduino video stream routine.

The video window is set so small to make the video smoother.

2. Collect and train the target through the video stream

The training environment is Python. The brief introduction of Anaconda I recommend here
is: data visualization + JupyterNotebook + Spyder
download will not take long, and then we only need to use the IDE:Spyder
After downloading, install the everywhereml package , open the Anaconda Powershell Prompt and enter the following command, already indicates that the package has been installed

pip install everywhereml>=0.2.19

Step 1. Create a new Spyder project

project->new project

, then decompress the Python project for learning and training the model and add it to the project, click to get the Python project
and open the Spyder software as shown in the figure, the Python project will be displayed in the project file column on the left, and the interaction is also useful Interface and Data Visualization Display Interface

Step 2. Acquisition of training data

Copy the following code to the interface and press Enter,Capture the target picture from the video stream as the model data support

from logging import basicConfig, INFO
from everywhereml.data import ImageDataset
from everywhereml.data.collect import MjpegCollector
# 给将要存放数据的文件夹命名
base_folder = 'Images_Data'
# 视频流显示的那个网页地址
IP_ADDRESS_OF_ESP = 'http://192.168.1.103'
basicConfig(level=INFO)
try:
    image_dataset = ImageDataset.from_nested_folders(
        name='Dataset', 
        base_folder=base_folder
    )
except FileNotFoundError:
    mjpeg_collector = MjpegCollector(address=IP_ADDRESS_OF_ESP)
    image_dataset = mjpeg_collector.collect_many_classes(
        dataset_name='Dataset', 
        base_folder=base_folder,
        duration=30
    )
print(image_dataset)

Then it will pop up to ask you to name the created class. I don’t recognize anything first, so name it none and press Enter, as shown in the figure below. After that, it

will display a prompt to take 1272 pictures as the basis for model training, and ask if the class is ok

INFO:root:Captured 1272 images
Is this class ok? (y|n)

Then enter y and press Enter. If it is the first time, you will be prompted to create a folder Images_Data to save data

INFO:root:creating D:\Esp_Cam\Spyder_Demo\Esp32_Cam\Images_Data folder
INFO:root:creating D:\Esp_Cam\Spyder_Demo\Esp32_Cam\Images_Data\none folder
Which class are you going to capture? (leave empty to exit) 

Open the corresponding folder and you will find that the captured image data is stored in it.

Similarly, I have trained pen and napkin.


If you don’t want to add it, you don’t need to enter it and press Enter

Which class are you going to capture? (leave empty to exit) 
Are you sure you want to exit? (y|n) 

Then enter y and press Enter to exit, and the trained class will be displayed at this time

ImageDataset[Dataset](num_images=3704, num_labels=3, labels=['napkin', 'none', 'pen'])

Step 3. Data processing and model building

Step 2 obtained more than 1,000 pictures of tissues, pens, and blanks as data support

Gray the image first
Execute in the interface

image_dataset = image_dataset.gray().uint8()

You can execute the following code in the interactive interface to preview the data processing

image_dataset.preview(samples_per_class=10, rows_per_class=2, figsize=(20, 10), cmap='gray')

Then use the histogram of oriented gradients algorithm for processing
Histogram of Oriented Gradients (HOG for short), this algorithm is lightweight and very suitable for Esp32-cam.
Execute the following code in the interactive interface

from everywhereml.preprocessing.image.object_detection import HogPipeline
from everywhereml.preprocessing.image.transform import Resize
pipeline = HogPipeline(
    transforms=[
        Resize(width=40, height=30)#此处的分辨率会影响处理时间和模型建立的准确度，可自行调整
    ]
)
feature_dataset = pipeline.fit_transform(image_dataset)
feature_dataset.describe()

Then output a data set consisting of feature vectors

print(pipeline)

If you want to see the extracted feature quantity information, you can draw a pair plot (pairplot) to intuitively feel the data

feature_dataset.plot.features_pairplot(n=200, k=8)

It can be seen intuitively that the aggregation properties of these three classes (none, napkin, pen) are good, but to some extent they are mixed with each other.
Further optimization using dimensionality reduction algorithms
The dimensionality reduction algorithm used is Uniform Manifold Approximation and Projection (UMAP for short)

feature_dataset.plot.umap()

Analysis of point aggregation properties shows that the model of 1 (none) is the most ideal, while the models of 0 (napkin) and 2 (pen) are relatively poor.
In general, it can be used to characterize our data.
Finally, train the classifier to complete the model establishment
The modeling method used is called Random Forest (RF)

from everywhereml.sklearn.ensemble import RandomForestClassifier
for i in range(10):
    clf = RandomForestClassifier(n_estimators=5, max_depth=10)
    train, test = feature_dataset.split(test_size=0.4, random_state=i)
    clf.fit(train)
    print('Score on test set: %.2f' % clf.score(test))
clf.fit(feature_dataset)

Now, we have trained and built the model

3. Generate code and transplant it to Esp32-Cam

(1) Convert HOG and RF algorithms into C++ code that can run on Esp32-cam

HOG algorithm to obtain feature vector data set

print(pipeline.to_arduino_file(
    filename='path-to-sketch/HogPipeline.h',
    instance_name='hog'
))

RF Algorithm Training Classifier

print(clf.to_arduino_file(
    filename='path-to-sketch/HogClassifier.h',
    instance_name='classifier', 
    class_map=feature_dataset.class_map
))

At this time, two .h files will be generated in the path-to-sketch/ directory

(2) Create an Arduino project project

Ino file replaced with the following code

#include "eloquent.h"
#include "eloquent/print.h"
#include "eloquent/tinyml/voting/quorum.h"
// 支撑的有 'aithinker', 'eye', 'm5stack', 'm5wide', 'wrover'
#include "eloquent/vision/camera/aithinker.h"//我用的是aithinker
#include "HogPipeline.h"//Spyder里生成的
#include "HogClassifier.h"//Spyder里生成的
Eloquent::TinyML::Voting::Quorum<7> quorum;
void setup() {
    
    
  Serial.begin(115200);
  delay(3000);
  Serial.println("Begin");
  camera.qqvga();
  camera.grayscale();
  while (!camera.begin())
    Serial.println("Cannot init camera");
}
void loop() {
    
    
  if (!camera.capture()) {
    
    
      Serial.println(camera.getErrorMessage());
      delay(1000);
      return;
  }
  hog.transform(camera.buffer);
  uint8_t prediction = classifier.predict(hog.features);
  int8_t stablePrediction = quorum.vote(prediction);
  if (quorum.isStable()) {
    
    
    eloquent::print::printf(
      Serial, 
      "Stable prediction: %s \t(DSP: %d ms, Classifier: %d us)\n", 
      classifier.getLabelOf(stablePrediction),
      hog.latencyInMillis(),
      classifier.latencyInMicros()
    );
  }
  camera.free();
}

Find the two .h files produced earlier and include them in the project (copy the two .h files into the project)

(3) Burn to Esp32-Cam