Intelligent sign language digital real-time translation based on Android+OpenCV+CNN+Keras - deep learning algorithm application (including Python, ipynb engineering source code) + data set (5)

---

Preface

This project relies on the Keras deep learning model and is designed to classify and recognize sign language in real time. To achieve this goal, the project incorporates relevant algorithms from the OpenCV library to capture the position of the hands, enabling real-time recognition of sign language in video streams and images.

First, the project uses algorithms from the OpenCV library to capture hand positions in video streams or images. This can involve technologies such as skin color detection, motion detection, or gesture detection to pinpoint sign language gestures.

Next, the project uses a CNN deep learning model to classify the captured sign language. After training, it can recognize different sign language gestures as specific categories or characters.

During the real-time recognition process, the sign language gestures in the video stream or image are passed to the CNN deep learning model, which makes inferences and recognizes the gestures into the corresponding categories. This enables the system to recognize sign language gestures in real time and convert them into text or other forms of output.

Overall, this project combines computer vision and deep learning technology to provide a real-time solution for sign language recognition. This is a beneficial tool for hearing-impaired people and sign language users to help them communicate and understand others more easily.

overall design

This part includes the overall system structure diagram and system flow chart.

Overall system structure diagram

The overall structure of the system is shown in the figure.

System flow chart

The system flow is shown in the figure.

Operating environment

This part includes Python environment, TensorFlow environment, Keras environment and Android environment .

Module implementation

This project includes 6 modules: data preprocessing, data enhancement, model construction, model training and preservation, model evaluation and model testing. The functions and related codes of each module are introduced below.

1. Data preprocessing

Download the corresponding data set on Kaggle. The download address is https://www.kaggle.com/ardamavi/sign-language-digits-dataset .

See blog for details .

2. Data enhancement

In order to facilitate the display of the effect of generated images and fine-tuning of parameters, this project does not use keras to directly train the generator. Instead, an enhanced data set is first generated and then used for model training.

See blog for details .

3. Model construction

After the data is loaded into the model, the model structure needs to be defined and the loss function optimized.

See blog for details .

4. Model training and saving

This section includes code related to model training and model saving.

See blog for details .

5. Model evaluation

Due to the lack of sign language recognition related models on the Internet, in order to facilitate the selection of the optimal model among various models and the optimization of the model, before the model is applied to the Android project, it is necessary to use the Python file on the PC device for preliminary running testing. In order to verify whether the sign language recognition strategy of this solution is feasible and select the optimal classification model.

See blog for details .

6. Model testing

After evaluating the feasibility of the overall model, apply the sign language recognition model to the Android Studio project to complete the APP. Specific steps are as follows.

See blog for details .

System test

This part includes training accuracy, test results and model application.

1. Training accuracy

The changes in accuracy loss during the training process are shown in Figures 1 and 2.

Figure 1 Model accuracy

Figure 2 Model loss value

2. Test effect

In the preliminary evaluation, after using Spyder to compile and run the relevant evaluation code, the hand position can be better captured under white walls and various lighting conditions, and a total of 10 sign language gestures from 0 to 9 can be accurately recognized, as shown in Figure 3 and As shown in Figure 4.

Figure 3 Effect diagram of capturing hand area

Figure 4 Sign language recognition renderings

3. Model application

This section includes program download and operation, application usage instructions and test results.

1) Download and run the program

After the Android project is compiled successfully, it is recommended to run the project on a real machine for testing. The emulator runs slowly and is not recommended. The method to run it on the real machine is as follows:

(1) Connect the mobile phone data cable to the computer, turn on the developer mode, turn on USB debugging, click the run button of the Android project, the option to connect to the phone will appear, click it.

(2) Android Studio generates an apk file and sends it to the mobile phone. Download the apk file on the mobile phone and install it.

2) Application instructions

Open the APP, the initial interface is as shown in the figure.

The three buttons from top to bottom on the interface are [Go to Image Recognition], [Start Translation of Sign Language], and [Stop Translation]. The interface sequentially displays the recognition results and confidence level, the captured hand area, and the accumulated sentence translations recognized.

Click the [Start Recognition] button, and the result is as shown in the figure.

Click the [Go to Image Recognition] button to jump to the image recognition interface. Click the [Photo Recognition] button to call the camera to take pictures and switch between the front and rear cameras. Click [Select from Album] to bring up the album interface. After selecting the image, the APP will display the selected image and return the sign language recognition results, as shown in the figure.

3) Test results

The effect of gesture recognition "520' is shown in the figure.

The effect of other gestures on picture recognition is shown in the figure.

The complex background effect is shown in the figure.

Related other blogs

Intelligent sign language digital real-time translation based on Android+OpenCV+CNN+Keras - deep learning algorithm application (including Python, ipynb engineering source code) + data set (1)

Intelligent sign language digital real-time translation based on Android+OpenCV+CNN+Keras - deep learning algorithm application (including Python, ipynb engineering source code) + data set (2)

Intelligent sign language digital real-time translation based on Android+OpenCV+CNN+Keras - deep learning algorithm application (including Python, ipynb engineering source code) + data set (3)

Intelligent sign language digital real-time translation based on Android+OpenCV+CNN+Keras - deep learning algorithm application (including Python, ipynb engineering source code) + data set (4)

Project source code download

For details, please see my blog resource download page

---

Download other information

If you want to continue to understand the learning routes and knowledge systems related to artificial intelligence, you are welcome to read my other blog " Heavyweight | Complete Artificial Intelligence AI Learning - Basic Knowledge Learning Route, all information can be downloaded directly from the network disk without following any routines.》
This blog refers to Github’s well-known open source platform, AI technology platform and experts in related fields: Datawhale, ApacheCN, AI Youdao and Dr. Huang Haiguang, etc., which has nearly 100G of related information. I hope it can help all my friends.