Table of contents
1. Design pattern factory pattern
3. Relevant hardware operation function calls under the wiringPi library
4. Serial communication of voice module
5. Real-time monitoring and photography functions of the camera
7. Qt program runs cross-platform (compiled into Android APP)
1. Functional requirements
- When the flame sensor detects flame, the buzzer will alarm and will stop alarming when there is no flame.
- Voice control to turn on and off multiple LED lights (second floor lights, dining room lights, living room lights, bathroom lights)
- Turn on the camera with voice and monitor the screen in real time on the web page corresponding to the IP address
- Voice controls the camera to take photos and save them in the current folder (transfer the photos to the PC for viewing through filezilla)
- Turn on the face recognition function by voice, compare the taken photo with your own photo, the buzzer will beep once if the recognition is successful, and four if it fails.
- Through the socket network, the development board runs the server, the Android mobile phone runs the client APP or the PC host computer runs the client, and the mobile phone or the host computer remotely sends instructions to complete the above functions, and real-time data measured by the temperature and humidity sensor on the Android APP or PC. Displayed on the QT interface of the PC host computer
2. Development environment
1. Hardware:
Orangepi Zero2 Allwinner H616 development board, voice module SU-03T, camera module OV9726, buzzer, flame sensor, 4 LEDs, etc., 4-way relay group, 6v power supply, several DuPont lines.
2. Software:
MobaXterm、VS Code、FileZilla
3. Pin assignment:
Enter gpio readall in the MobaXterm command control terminal to view all pins on the development board. The pin connections of the voice module, buzzer, flame sensor and 4-way relay group are outlined in the figure below.
Since there are not enough pins on the board after adding the temperature and humidity sensor, the language recognition module will be replaced during the second video demonstration. The corresponding IO of the temperature and humidity sensor is, VCC--5V, GND--GND, DAT--3wPi.
3. Key points
1. Design pattern factory pattern
The factory pattern provides a way to encapsulate the instantiation process of an object in a factory class. This article uses the factory pattern to separate object creation and usage code, providing a unified interface to create different types of objects. In the factory mode, we do not expose the creation logic to the client when creating an object, and use a common interface to point to the newly created object (linked list).
//1.指令工厂初始化
pCommandHead = addVoiceContrlToInputCommandLink(pCommandHead);//串口
pCommandHead = addSocketContrlToInputCommandLink(pCommandHead);
//2.设备控制工厂初始化
//四个LED+火灾+蜂鸣器+摄像头
pdeviceHead = addBathroomLightToDeviceLink(pdeviceHead);
pdeviceHead = addUpstairLightToDeviceLink(pdeviceHead);
pdeviceHead = addRestaurantLightToDeviceLink(pdeviceHead);
pdeviceHead = addLivingroomLightToDeviceLink(pdeviceHead);
pdeviceHead = addFireToDeviceLink(pdeviceHead);
pdeviceHead = addBeepToDeviceLink(pdeviceHead);
pdeviceHead = addCameraToDeviceLink(pdeviceHead);2. Thread
Three threads are created in the main function: voice thread, flame detection thread, and network thread.
Voice Thread (voiceThread): Complete the configuration and initialization of the serial port. In the while loop, check whether there is a voice command word coming to the serial port every 0.3s, and execute the corresponding operation if there is one.
Flame check thread (socketThread): The while loop detects whether there is a flame every 0.5s. If there is a flame, the buzzer will sound an alarm to know that there is no flame.
Network thread (socketThread): Receive client instructions, realize the remote sending instructions of the host computer to complete the functions described in [Functional Requirements]; send the measured data of the temperature and humidity sensor to the client, and display it on the QT interface in real time.
//1、接收客户端的指令控制灯和摄像头 执行指令功能与语音控制复用
//2、向客户端发送温湿度数据
void *socket_thread(){
struct sockaddr_in c_addr;
memset(&c_addr,0,sizeof(struct sockaddr_in));
int clen = sizeof(struct sockaddr_in);
socketHandler = findCommandByName("socketServer", pCommandHead);
if(socketHandler == NULL){
printf("find socketHandler error\n");
pthread_exit(NULL);
}
printf("%s init success\n", socketHandler->commandName);
socketHandler->Init(socketHandler, NULL, NULL);
while(1){
c_fd = accept(socketHandler->sfd, (struct sockaddr *)&c_addr, &clen);
piThreadCreate(read_thread);
pthread_tempAndHumi_create();
}
}In the network thread, first initialize the configuration of the socket, including the configuration of IPV4 Internet protocol and TCP protocol (socket), bind the IP address and port number (bind), and listen to the corresponding port through the socket identifier (listen) And wait for the client to access (accept).
After the client connects, it creates a thread for reading data. In the while loop, the process is blocked in the read function until the client issues an instruction. The execution function of the instruction is multiplexed with the execution function of the voice instruction of the voice module (the voiceContrlFunc function is in the fourth point exhibit). For example: press the [Turn on living room light] button on the QT interface, the client sends the string "OLL" through the network, reads the command on the server and puts it into socketHandler->command, and calls the function voiceContrlFunc(socketHandler->command) to execute the command .
At the same time, after accessing, the client will call the function pthread_tempAndHumi_create() to create a thread for sending data, that is, send the data of temperature and humidity in real time, and cooperate with the signal slot in QT to receive the data and display it on the QT interface. The code is as follows.
void *read_thread(){
while(1){
int n_read = 0;
memset(socketHandler->command, '\0', sizeof(socketHandler->command));
n_read = read(c_fd, socketHandler->command, sizeof(socketHandler->command));//n_read是读到字节数
voiceContrlFunc(socketHandler->command);
if(n_read == -1){
perror("read");
}
else if(n_read>0){
printf("\nget: %d, %s\n",n_read, socketHandler->command);
}
else{
printf("client quit\n");
break;
}
}
}QT program, function to control the lights in the living room:
void Widget::on_livingRoomLight_clicked()
{
if(livingRoomLightFlag == 1){
//客户端向服务端发送消息
if(tcpSocket->state() == QAbstractSocket::ConnectedState){
ui->livingRoomLight->setText("关客厅灯");
tcpSocket->write("OLL");
ui->textBrowser->append("> 客厅灯已打开\n");
livingRoomLightFlag = 0;
}
else{
ui->textBrowser->append("请先与服务端连接！");
}
}
else{
if(tcpSocket->state() == QAbstractSocket::ConnectedState){
ui->livingRoomLight->setText("开客厅灯");
tcpSocket->write("CLL");
ui->textBrowser->append("> 客厅灯已关闭\n");
livingRoomLightFlag = 1;
}
else{
ui->textBrowser->append("请先与服务端连接！");
}
}
}Signal slot connection and TCP network reading function in QT program:
connect(tcpSocket, SIGNAL(readyRead()), this, SLOT(receiveMessages()));
void Widget::receiveMessages()
{
QByteArray tmpByteArray = tcpSocket->readAll();
char* tempHumi;
tempHumi = tmpByteArray.data();
ui->tempLabel->setText(QString::number(tempHumi[0]));
ui->humiLabel->setText(QString::number(tempHumi[2]));
} 
PC-side QT interface
APP interface
3. Relevant hardware operation function calls under the wiringPi library
Including the initialization of the wiringPi library, the input and output pin configuration and high and low level settings of the buzzer, flame sensor, and relay group. Serial port configuration and initialization under wiringP library.
4. Serial communication of voice module
This article uses the serial port device /dev/ttyS5 in the Allwinner H616 chip with a baud rate of 115200 to realize serial communication with the voice module. The voice module receives the command words from us, converts the command words into hexadecimal data and sends them to the development board through the serial port, and completes the operations of receiving, storing, judging which command is the serial port data, and executing the corresponding command in the program, including Switch the lights on the second floor, dining room lights, living room lights, bathroom lights, turn on the camera, take a photo and face recognition.
The voice module SU-03T needs to be burned into the SDK corresponding to the command. The SDK configured in this article is completed on the intelligent AD/AI product zero-code platform and is free.
int get_voice_type(char *cmd)
{
if(!strcmp("OLL", cmd)) return OLL;
if(!strcmp("ORL", cmd)) return ORL;
if(!strcmp("OUL", cmd)) return OUL;
if(!strcmp("OBL", cmd)) return OBL;
if(!strcmp("CLL", cmd)) return CLL;
if(!strcmp("CRL", cmd)) return CRL;
if(!strcmp("CUL", cmd)) return CUL;
if(!strcmp("CBL", cmd)) return CBL;
if(!strcmp("OC" , cmd)) return OC ;
if(!strcmp("TAP", cmd)) return TAP;
if(!strcmp("OFR", cmd)) return OFR;
perror("voice recognition failure");
}
void voiceContrlFunc(char *cmd){
switch(get_voice_type(cmd)){
case OLL://OLL ASCII对应的16进制4f 4c 4c
printf("open livingroom light\n");
struct Devices *tmpOpenLivingroomLight = findDeviceByName("livingroomLight", pdeviceHead);
tmpOpenLivingroomLight->open(tmpOpenLivingroomLight->pinNum);
break;
case ORL://ORL 4f 52 4c
printf("open restaurant light\n");
struct Devices *tmpOpenRestaurantLight = findDeviceByName("restaurantLight", pdeviceHead);
tmpOpenRestaurantLight->open(tmpOpenRestaurantLight->pinNum);
break;
case OUL://OUL 4f 55 4c
printf("open upstair light\n");
struct Devices *tmpOpenUpstairLight = findDeviceByName("upstairLight", pdeviceHead);
tmpOpenUpstairLight->open(tmpOpenUpstairLight->pinNum);
break;
case OBL://OBL 4f 42 4c
printf("open bathroom light\n");
struct Devices *tmpOpenBathroomLight = findDeviceByName("bathroomLight", pdeviceHead);
tmpOpenBathroomLight->open(tmpOpenBathroomLight->pinNum);
break;
case CLL://CLL 43 4c 4c
printf("close livingroom light\n");
struct Devices *tmpCloseLivingroomLight = findDeviceByName("livingroomLight", pdeviceHead);
tmpCloseLivingroomLight->close(tmpCloseLivingroomLight->pinNum);
break;
case CRL://CRL 43 52 4c
printf("close restaurant light\n");
struct Devices *tmpCloseRestaurantLight = findDeviceByName("restaurantLight", pdeviceHead);
tmpCloseRestaurantLight->close(tmpCloseRestaurantLight->pinNum);
break;
case CUL://CUL 43 55 4c
printf("close upstair light\n");
struct Devices *tmpCloseUpstairLight = findDeviceByName("upstairLight", pdeviceHead);
tmpCloseUpstairLight->close(tmpCloseUpstairLight->pinNum);
break;
case CBL://CBL 43 42 4c
printf("close bathroom light\n");
struct Devices *tmpCloseBathroomLight = findDeviceByName("bathroomLight", pdeviceHead);
tmpCloseBathroomLight->close(tmpCloseBathroomLight->pinNum);
break;
case OC://OC 4f 43
printf("open camera\n");
printf(" -------------------------------------------------------------------\n");
printf(" --\033[1;32m 已开启摄像头，请到指定网页观看画面 https//192.168.43.206:8081 \033[0m--\n");//黄色字体
printf(" -------------------------------------------------------------------\n");
printf("\n");
break;
case TAP://TAP 54 41 50
printf("take a picture\n");
struct Devices *tmpTakeAPictureCamera = findDeviceByName("camera", pdeviceHead);
tmpTakeAPictureCamera->takeAPicture();
printf(" --------------------------------------\n");
printf(" --\033[1;32m 已拍照，请在当前文件夹下查看照片 \033[0m--\n");//黄色字体
printf(" --------------------------------------\n");
printf("\n");
break;
case OFR://OFR 4f 46 52
printf("open face recognition\n");
struct Devices *tmpFaceRecCamera = findDeviceByName("camera", pdeviceHead);
tmpFaceRecCamera->faceRecognition();
break;
}
}
5. Real-time monitoring and photography functions of the camera
Refer to this article: (1031 messages) How to configure the USB camera on the Orange Pi Zero 2 development board_Aaron’s blog where he is still writing code-CSDN blog
6. Face recognition
Use the command word [Face Recognition] to let the camera take a picture and compare it with the local photo. If it succeeds, the buzzer will beep once, and if it fails, it will beep four times. The face recognition processing program calls the face recognition API of Xiangyun OCR face recognition (netocr.com) , the interface address https://netocr.com/api/faceliu.do is the address of the https protocol, and the https protocol It adds an extra layer between http and tcp for authentication and data encryption.
If you want to access the address of the https protocol, you need to use Libcurl, a cross-platform network protocol library. With the OpenSSL library, you can access the interface of the https protocol. (Compile OpenSSL to support Libcurl's https access. If you compile Libcurl directly, you can only access http but not https. You need the OpenSSL library to access https)
camera.c
#include "contrlDevices.h"
#include <stdio.h>
#include <curl/curl.h>
#include <string.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
extern struct Devices *pdeviceHead;//extern表面变量或函数是定义在其他文件中，声明 为全局变量在该源文件使用
struct Devices* findDeviceByName(char *name, struct Devices *phead);
#define true 1
#define false 0
struct Devices camera;
char buf[1024] = {'\0'};//全局
void cameraTakeAPicture(){
system("(fswebcam -d /dev/video0 --no-banner -r 1280x720 -S 5 ./image.jpg) > tmpFile");//照片存放在当前目录下
}
size_t readData(void *ptr, size_t size, size_t nmemb, void *stream){
strncpy(buf, ptr, 1024);
}
char *getPicBase64FromFile(char *filePath){
char *bufPic = NULL;
char cmd[128] = {'\0'};
sprintf(cmd, "base64 %s > tmpFile", filePath);
system(cmd);//图片的base64流数据存入tmpFile文件中
int fd = open("./tmpFile", O_RDWR);
int filelen = lseek(fd, 0, SEEK_END);
lseek(fd, 0, SEEK_SET);//重新让文件的光标回到初始位置
bufPic = (char *)malloc(filelen + 2);//+1也可以 多加点没毛病
memset(bufPic, '\0', filelen + 2);
read(fd, bufPic, filelen);
close(fd);
system("rm -f tmpFile");
return bufPic;
}
void cameraFaceRecognition(){
camera.takeAPicture();
CURL *curl;
CURLcode res;
char *postString;
char *img1;
char *img2;
char *key = "DYRrmZz2rTwYGywyWdhKzR";
char *secret = "56bc8e083a9b4d9fbf590413ddcb3a61";
int typeId = 21;
char *format = "xml";
char *bufPic1 = getPicBase64FromFile("./image.jpg");
char *bufPic2 = getPicBase64FromFile("./zyl.jpg");
int len = strlen(key) + strlen(secret) + strlen(bufPic1) + strlen(bufPic2) + 128;
postString = (char *)malloc(len);
memset(postString, '\0', len);//sizeof(postString)替换成len,因为postString是指针
sprintf(postString, "&img1=%s&img2=%s&key=%s&secret=%s&typeId=%d&format=%s", bufPic1, bufPic2, key, secret, 21, format);//拼接字符串
curl = curl_easy_init();
if (curl){
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "/tmp/cookie.txt"); // 指定cookie文件
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, postString); // 指定post内容
curl_easy_setopt(curl, CURLOPT_URL,"https://netocr.com/api/faceliu.do"); // 指定url
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, readData);//当有数据回来 调用回调函数
res = curl_easy_perform(curl);
struct Devices *beepHandler = findDeviceByName("beep", pdeviceHead);
if(strstr(buf, "是") != NULL){
beepHandler->open(beepHandler->pinNum); usleep(300000);
beepHandler->close(beepHandler->pinNum);
printf("\n");
printf(" -----------------------------------\n");
printf(" --\033[1;32m 人脸识别成功: the same person \033[0m--\n");//绿色字体
printf(" -----------------------------------\n");
printf("\n");
}
else{
int i = 4;
while(i--){
beepHandler->open(beepHandler->pinNum); usleep(200000);
beepHandler->close(beepHandler->pinNum); usleep(100000);
}
printf("\n");
printf(" ------------------------------------\n");
printf(" --\033[1;31m 人脸识别失败: different person \033[0m--\n");//红色字体
printf(" ------------------------------------\n");
printf("\n");
}
curl_easy_cleanup(curl);
}
}
//实例化对象
struct Devices camera = {
.deviceName = "camera",
.takeAPicture = cameraTakeAPicture,
.faceRecognition = cameraFaceRecognition
};
struct Devices* addCameraToDeviceLink(struct Devices *phead){
if(phead == NULL){
return &camera;
}
else{//头插
camera.next = phead;
phead = &camera;
return phead;
}
}7. Qt program runs cross-platform (compiled into Android APP)
Installation packages required to build the environment:
4. Compile and run
When compiling, you need to use some library files and header files in the library files. When using the temperature and humidity sensor, add tempAndHumi.c
gcc bathroomLight.c livingroomLight.c restaurantLight.c upstairLight.c socketContrl.c voiceContrl.c fireDetection.c beep.c camera.c usartContrl.c main.c -I ../httpHandler/curl-7.71.1/_install/include/ -L ../httpHandler/curl-7.71.1/_install/lib/ -lcurl -lwiringPi -lpthreadrun:
sudo ./a.out5. Video function display
Smart home function display
QT interface function display on PC
Android app function display