ESP32cam series tutorial 001: Use webcam camera to view video in real time

1. ESP32cam introduction

ESP32-CAM is a small-sized camera module. This module can work independently as a minimum system with a size of only27*40.5*4.5mm. It can be widely used in various Internet of Things situations and is suitable for families. Smart devices, industrial wireless control, wireless monitoring, QR wireless identification, wireless positioning system signals and other IoT applications are ideal solutions for IoT applications. [^1]

Its product features are as follows:

• Adopts low-power dual-core 32-bit CPU, which can be used as an application processor
• Main frequency up to 240MHz, computing power up to 600 DMIPS
• Built-in 520 KB SRAM, external 8MB PSRAM
• Supports UART/SPI/I2C/PWM/ADC/DAC and other interfaces
• Supports OV2640 and OV7670 cameras, built-in flash
• Support picture WiFI upload
• Support TF card
• Supports multiple sleep modes.
• Embedded Lwip and FreeRTOS
• Support STA/AP/STA+AP working mode
• Supports Smart Config/AirKiss one-click network configuration
• Support secondary development

The interface pin diagram of ESP32cam is as follows:

2. arduino IDE

2.1 Install arduino IDE

Download the official website:https://www.arduino.cc/en/software
Download the IDE that matches your operating system version and install it .

2.2 Arduino IDE obtains ESP32 development environment

Since there is no development version of ESP32 in the arduino IDE, it needs to be installed manually. The installation method is as follows:

1. Open Arduino IDE, find File>Preferences, and fill in the configuration link of ESP32 into the additional development board management URL.

# 配置链接
	https://dl.espressif.com/dl/package_esp32_index.json

2. In the Arduino IDE, find Tools>Development Boards>Development Board Development Board Management, search for ESP32 or select ESP32 Wrover Module directly.
   

3 Real-time viewing of intranet videos

3.1 Select File>Example>ESP32>Camera>CameraWebServer to enter the sample code interface.

3.2 Modify relevant parameters in the sample code.

1. Modify the name of the wifi and password in the sample code.
2. Modify the camera type in the sample code to CAMERA_MODEL_AI_THINKER .

3.3 Running results

After the upload is successful, press the RST button on the ESP32cam development board to restart the development board.
Select Tools>Serial Monitor, view the IP output in the serial port, and open the IP with a browser to view the video screen in real time.

3.4 The procedure is as follows

#include "esp_camera.h"
#include <WiFi.h>

//
// WARNING!!! PSRAM IC required for UXGA resolution and high JPEG quality
//            Ensure ESP32 Wrover Module or other board with PSRAM is selected
//            Partial images will be transmitted if image exceeds buffer size
//

// Select camera model
// #define CAMERA_MODEL_WROVER_KIT // Has PSRAM
//#define CAMERA_MODEL_ESP_EYE // Has PSRAM
//#define CAMERA_MODEL_M5STACK_PSRAM // Has PSRAM
//#define CAMERA_MODEL_M5STACK_V2_PSRAM // M5Camera version B Has PSRAM
//#define CAMERA_MODEL_M5STACK_WIDE // Has PSRAM
//#define CAMERA_MODEL_M5STACK_ESP32CAM // No PSRAM
#define CAMERA_MODEL_AI_THINKER // Has PSRAM
//#define CAMERA_MODEL_TTGO_T_JOURNAL // No PSRAM

#include "camera_pins.h"

const char* ssid = "TP-LINK_1760";
const char* password = "987654321";

void startCameraServer();

void setup() {
  Serial.begin(115200);
  Serial.setDebugOutput(true);
  Serial.println();

  camera_config_t config;
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = Y2_GPIO_NUM;
  config.pin_d1 = Y3_GPIO_NUM;
  config.pin_d2 = Y4_GPIO_NUM;
  config.pin_d3 = Y5_GPIO_NUM;
  config.pin_d4 = Y6_GPIO_NUM;
  config.pin_d5 = Y7_GPIO_NUM;
  config.pin_d6 = Y8_GPIO_NUM;
  config.pin_d7 = Y9_GPIO_NUM;
  config.pin_xclk = XCLK_GPIO_NUM;
  config.pin_pclk = PCLK_GPIO_NUM;
  config.pin_vsync = VSYNC_GPIO_NUM;
  config.pin_href = HREF_GPIO_NUM;
  config.pin_sscb_sda = SIOD_GPIO_NUM;
  config.pin_sscb_scl = SIOC_GPIO_NUM;
  config.pin_pwdn = PWDN_GPIO_NUM;
  config.pin_reset = RESET_GPIO_NUM;
  config.xclk_freq_hz = 20000000;
  config.pixel_format = PIXFORMAT_JPEG;
  
  // if PSRAM IC present, init with UXGA resolution and higher JPEG quality
  //                      for larger pre-allocated frame buffer.
  if(psramFound()){
    config.frame_size = FRAMESIZE_UXGA;
    config.jpeg_quality = 10;
    config.fb_count = 2;
  } else {
    config.frame_size = FRAMESIZE_SVGA;
    config.jpeg_quality = 12;
    config.fb_count = 1;
  }

#if defined(CAMERA_MODEL_ESP_EYE)
  pinMode(13, INPUT_PULLUP);
  pinMode(14, INPUT_PULLUP);
#endif

  // camera init
  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK) {
    Serial.printf("Camera init failed with error 0x%x", err);
    return;
  }

  sensor_t * s = esp_camera_sensor_get();
  // initial sensors are flipped vertically and colors are a bit saturated
  if (s->id.PID == OV3660_PID) {
    s->set_vflip(s, 1); // flip it back
    s->set_brightness(s, 1); // up the brightness just a bit
    s->set_saturation(s, -2); // lower the saturation
  }
  // drop down frame size for higher initial frame rate
  s->set_framesize(s, FRAMESIZE_QVGA);

#if defined(CAMERA_MODEL_M5STACK_WIDE) || defined(CAMERA_MODEL_M5STACK_ESP32CAM)
  s->set_vflip(s, 1);
  s->set_hmirror(s, 1);
#endif

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");

  startCameraServer();

  Serial.print("Camera Ready! Use 'http://");
  Serial.print(WiFi.localIP());
  Serial.println("' to connect");
}

void loop() {
  // put your main code here, to run repeatedly:
  delay(10000);
}

4. Burn the program into the ESP32cam development board

4.1 Download through the supporting downloader

1. Install the downloader and ESP32cam together, use the data cable to connect to the computer, install the driver provided by the merchant, and then select the corresponding development board and serial port in the tool options.

2. Then click the Compilation Verification button in the upper left corner to compile. After the compilation is successful, click the button next to it to upload. button is burned into the ESP32cam development board.
   

4.2 Download via USB to TTL (CH340) downloader

The USB to TTL downloader is only different from the downloader provided with the connection line. The other download steps areSameof.

The link cable between USB to TTL downloader and ESP32cam is as follows:

1. USB to TTL VCC to ESP32cam 5V
2. USB to TTL GND to ESP32cam GND
3. USB to TTL RXD to ESP32cam TXD
4. USB to TTL TXD to ESP32cam RXD
5. When downloading, GPIO1 needs to be connected to GND to start the download mode.

5. Real-time viewing of external network videos

Real-time viewing of external network videos is divided into: 1. Programs running on the esp32cam development board; 2. Programs running on the server.
In the server that sends the video data through ESP32cam, the server runs the receiving program to receive and display it. The advantage of this is that it can be sent to an external public network server.
Please see Chapter 4 for program burning.

The program in esp32cam is as follows:

#include <Arduino.h>
#include <WiFi.h>
#include "esp_camera.h"
#include <vector>
 
const char *ssid = "TP-LINK_1760";
const char *password = "987654321";
const IPAddress serverIP(192,168,1,104); //欲访问的地址，即服务器的ip，可内网也可公网
uint16_t serverPort = 18080;         //服务器端口号

# MTU
#define maxcache 1430
 
WiFiClient client; //声明一个客户端对象，用于与服务器进行连接
 
//CAMERA_MODEL_AI_THINKER类型摄像头的引脚定义
#define PWDN_GPIO_NUM     32
#define RESET_GPIO_NUM    -1
#define XCLK_GPIO_NUM      0
#define SIOD_GPIO_NUM     26
#define SIOC_GPIO_NUM     27
 
#define Y9_GPIO_NUM       35
#define Y8_GPIO_NUM       34
#define Y7_GPIO_NUM       39
#define Y6_GPIO_NUM       36
#define Y5_GPIO_NUM       21
#define Y4_GPIO_NUM       19
#define Y3_GPIO_NUM       18
#define Y2_GPIO_NUM        5
#define VSYNC_GPIO_NUM    25
#define HREF_GPIO_NUM     23
#define PCLK_GPIO_NUM     22
 
static camera_config_t camera_config = {
    .pin_pwdn = PWDN_GPIO_NUM,
    .pin_reset = RESET_GPIO_NUM,
    .pin_xclk = XCLK_GPIO_NUM,
    .pin_sscb_sda = SIOD_GPIO_NUM,
    .pin_sscb_scl = SIOC_GPIO_NUM,
    
    .pin_d7 = Y9_GPIO_NUM,
    .pin_d6 = Y8_GPIO_NUM,
    .pin_d5 = Y7_GPIO_NUM,
    .pin_d4 = Y6_GPIO_NUM,
    .pin_d3 = Y5_GPIO_NUM,
    .pin_d2 = Y4_GPIO_NUM,
    .pin_d1 = Y3_GPIO_NUM,
    .pin_d0 = Y2_GPIO_NUM,
    .pin_vsync = VSYNC_GPIO_NUM,
    .pin_href = HREF_GPIO_NUM,
    .pin_pclk = PCLK_GPIO_NUM,
    
    .xclk_freq_hz = 20000000,
    .ledc_timer = LEDC_TIMER_0,
    .ledc_channel = LEDC_CHANNEL_0,
    
    .pixel_format = PIXFORMAT_JPEG,
    // .frame_size = FRAMESIZE_VGA,
    // FRAMESIZE_UXGA (1600 x 1200)
    // FRAMESIZE_QVGA (320 x 240)
    // FRAMESIZE_CIF (352 x 288)
    // FRAMESIZE_VGA (640 x 480)
    // FRAMESIZE_SVGA (800 x 600)
    // FRAMESIZE_XGA (1024 x 768)
    // FRAMESIZE_SXGA (1280 x 1024)
    .frame_size = FRAMESIZE_QVGA,
    .jpeg_quality = 24,
    // 图像质量（jpeg_quality) 可以是 0 到 63 之间的数字。数字越小意味着质量越高
    .fb_count = 1,
};
void wifi_init()
{
    WiFi.mode(WIFI_STA);
    WiFi.setSleep(false); //关闭STA模式下wifi休眠，提高响应速度
    WiFi.begin(ssid, password);
    while (WiFi.status() != WL_CONNECTED)
    {
        delay(500);
        Serial.print(".");
    }
    Serial.println("WiFi Connected!");
    Serial.print("IP Address:");
    Serial.println(WiFi.localIP());
}
esp_err_t camera_init() {
    //initialize the camera
    esp_err_t err = esp_camera_init(&camera_config);
    if (err != ESP_OK) {
        Serial.println("Camera Init Failed");
        return err;
    }
    sensor_t * s = esp_camera_sensor_get();
    //initial sensors are flipped vertically and colors are a bit saturated
    if (s->id.PID == OV2640_PID) {
    //        s->set_vflip(s, 1);//flip it back
    //        s->set_brightness(s, 1);//up the blightness just a bit
    //        s->set_contrast(s, 1);
    }
    Serial.println("Camera Init OK!");
    return ESP_OK;
}
 
void setup()
{
    Serial.begin(115200);
    wifi_init();
    camera_init();
}
 
void loop()
{
    Serial.println("Try To Connect TCP Server!");
    if (client.connect(serverIP, serverPort)) //尝试访问目标地址
    {
        Serial.println("Connect Tcp Server Success!");
        //client.println("Frame Begin");  //46 72 61 6D 65 20 42 65 67 69 6E // 0D 0A 代表换行  //向服务器发送数据
        while (1){       
          camera_fb_t * fb = esp_camera_fb_get();
          uint8_t * temp = fb->buf; //这个是为了保存一个地址，在摄像头数据发送完毕后需要返回，否则会出现板子发送一段时间后自动重启，不断重复
          if (!fb)
          {
              Serial.println( "Camera Capture Failed");
          }
          else
          { 
            //先发送Frame Begin 表示开始发送图片 然后将图片数据分包发送 每次发送1430 余数最后发送 
            //完毕后发送结束标志 Frame Over 表示一张图片发送完毕 
            client.print("Frame Begin"); //一张图片的起始标志
            // 将图片数据分段发送
            int leng = fb->len;
            int timess = leng/maxcache;
            int extra = leng%maxcache;
            for(int j = 0;j< timess;j++)
            {
              client.write(fb->buf, maxcache); 
              for(int i =0;i< maxcache;i++)
              {
                fb->buf++;
              }
            }
            client.write(fb->buf, extra);
            client.print("Frame Over");      // 一张图片的结束标志
            Serial.print("This Frame Length:");
            Serial.print(fb->len);
            Serial.println(".Succes To Send Image For TCP!");
            //return the frame buffer back to the driver for reuse
            fb->buf = temp; //将当时保存的指针重新返还
            esp_camera_fb_return(fb);  //这一步在发送完毕后要执行，具体作用还未可知。        
          }
          delay(20);//短暂延时 增加数据传输可靠性
        }
        /*
        while (client.connected() || client.available()) //如果已连接或有收到的未读取的数据
        {
            if (client.available()) //如果有数据可读取
            {
                String line = client.readStringUntil('\n'); //读取数据到换行符
                Serial.print("ReceiveData：");
                Serial.println(line);
                client.print("--From ESP32--:Hello Server!");    
            }
        }
        Serial.println("close connect!");
        client.stop(); //关闭客户端
        */
    }
    else
    {
        Serial.println("Connect To Tcp Server Failed!After 10 Seconds Try Again!");
        client.stop(); //关闭客户端
    }
    delay(10000);
}

Program running in the server (Python):

import socket
import threading
import time
import numpy as np
import cv2
 
begin_data = b'Frame Begin'
end_data = b'Frame Over'
 
#接收数据
# ESP32发送一张照片的流程
# 先发送Frame Begin 表示开始发送图片 然后将图片数据分包发送 每次发送1430 余数最后发送
# 完毕后发送结束标志 Frame Over 表示一张图片发送完毕
# 1430 来自ESP32cam发送的一个包大小为1430 接收到数据 data格式为b''
def handle_sock(sock, addr):
    temp_data = b''
    t1 = int(round(time.time() * 1000))
    while True:
        data = sock.recv(1430)
        # 如果这一帧数据包的开头是 b'Frame Begin' 则是一张图片的开始
        if data[0:len(begin_data)] == begin_data:
            # 将这一帧数据包的开始标志信息（b'Frame Begin'）清除   因为他不属于图片数据
            data = data[len(begin_data):len(data)]
            # 判断这一帧数据流是不是最后一个帧 最后一针数据的结尾时b'Frame Over'
            while data[-len(end_data):] != end_data:
                temp_data = temp_data + data  # 不是结束的包 讲数据添加进temp_data
                data = sock.recv(1430)# 继续接受数据 直到接受的数据包包含b'Frame Over' 表示是这张图片的最后一针
            # 判断为最后一个包 将数据去除 结束标志信息 b'Frame Over'
            temp_data = temp_data + data[0:(len(data) - len(end_data))]  # 将多余的（\r\nFrame Over）去掉 其他放入temp_data
            # 显示图片
            receive_data = np.frombuffer(temp_data, dtype='uint8')  # 将获取到的字符流数据转换成1维数组
            r_img = cv2.imdecode(receive_data, cv2.IMREAD_COLOR)  # 将数组解码成图像
            # r_img = r_img.reshape(480, 640, 3)
            # r_img = r_img.reshape(320, 240, 3)
            t2 = int(round(time.time() * 1000))
            fps = 1000//(t2-t1)
            cv2.putText(r_img, "FPS" + str(fps), (50, 50), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 0, 0), 2)
            cv2.imshow('server_frame', r_img)
            if cv2.waitKey(1) & 0xFF == ord('q'):
                break
            t1 = t2
            print("接收到的数据包大小：" + str(len(temp_data)))  # 显示该张照片数据大小
            temp_data = b''  # 清空数据 便于下一章照片使用
 
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# 这里的 ip 与端口是运行该程序的服务器的 ip 与端口，需要与 arduino 中的一致
server.bind(('192.168.1.104', 18080))
server.listen(5)
CONNECTION_LIST = []
 
#主线程循环接收客户端连接
while True:
    sock, addr = server.accept()
    CONNECTION_LIST.append(sock)
    print('Connect--{}'.format(addr))
    #连接成功后开一个线程用于处理客户端
    client_thread = threading.Thread(target=handle_sock, args=(sock, addr))
    client_thread.start()

6. References

1. CSDN:ESP32 cam from installation...
2. Arduino IDE official website
3. CSDN: ESP32cam and server TCP video transmission
4. WIFI and ADC2 in ESP32cam
5. CSDN:USB2TTL CH340

---

This article was first published on my blog:https://blog.gitnote.cn/post/esp32cam_001
Copyright information: CC BY-NC- SA 4.0 (Free to reproduce - non-commercial - share alike - maintain attribution)