An Attempt to Intelligentize Robot Arm Control Based on Speech Recognition Technology
introduce:
In the movie Iron Man, we see Tony Stark communicating with the artificial intelligence Jarvis while building the device. Tony described the parts he needed to Jarvis, and Jarvis controlled the robotic arm to assist Tony in completing the task. With today's technology, this realization is only a matter of time. Therefore, I decided to try to implement this function by myself, using voice control to operate the robotic arm, to achieve a simple application of artificial intelligence.
I was solely responsible for connecting and controlling the robotic arm and voice development board, triggering the movement of the robotic arm through specific commands. This is a challenging project, because voice control technology involves multiple aspects, such as speech recognition, signal processing and robot control, which requires me to learn and understand related knowledge. Through this article, I hope to share my project implementation process, achievements and lessons learned to inspire and help fellow students working in the field of robotics and voice control.
In order to implement this project, I first needed to solve two main problems: how to process the voice signal and how to control the robotic arm.
Process speech signals:
Steps to process a speech signal:
● Hardware selection: choose a suitable development board (Arduino, Raspberry Pi, etc.).
● Microphone signal processing: select a suitable microphone for collection, and convert the collected voice signal into a digital signal through an analog-to-digital converter.
● Signal processing: the collected signal is processed by noise reduction and filtering. Useful features are extracted, and feature signals are processed using speech recognition algorithms to output digital signals to the computer.
Control the robotic arm:
● Use a simple and easy-to-operate mechanical arm.
The above is the solution to two main problems. However, after research, developing a development board that can handle speech signals is a lot of work! Fortunately, there are integrated speech development boards on the market now, and we can directly buy one to use.
HLK-V20 Voice Development Board
The HLK-V20 development board is a complex board with two IO interfaces - one for voice output and the other for voice input. It can be connected to a computer via a serial port.
MyCobot 280 M5Stack 2023 Robotic Arm
For the robotic arm, I chose the popular desktop robotic arm, myCobot 280 M5Stack 2023, developed by Elephant Robotics. The robotic arm has recently undergone a major software update, enabling a serial communication speed of 20ms and a rich interface compatible with popular programming languages such as Python and C++. I chose this robotic arm for several reasons, including its compact size, rich Python API interface (I am good at programming), and ease of installation and use.
Programming language: Python 3.7+
Platform: windows11
Python libraries used: serial, pymycobot, time.
project development
In the development of the project, four main steps are involved:
● Voice input
● The development board retrieves the voice information and transmits it to the PC
● PC processes received information
● Send motion control commands to the robotic arm for movement
To observe how the board transmits data and what type of data it returns, use the serial library in Python to manipulate the serial port. The library has methods to open, close, read and write data from the serial port.
'''
For instance, the serial.Serial() method is used to open the serial port and connect it to the PC, while the serial.read() method reads the data transmitted from the serial port.
'''
import serial
s = serial.Serial("COM9",115200)
r = s.read()
while True:
print(r)copy
This is the data obtained from read().
By analyzing the data transmitted from the development board, the wakeup word of the development board is recognized as "wakeup_uni", and the keyword "uArTcp" indicates the start of the next command.
It is found that the received data is concatenated, and each time a command is entered, it is concatenated with the previous command.
It is found that the received data is concatenated, and each time a command is entered, it is concatenated with the previous command. To check if the board has received voice input, the if a in b: statement is used, prepended with the input command and its corresponding keyword. For example, the wakeup word is set to "wakeup_uni" and 4 prompt words are set for testing.
#set prompt word
hi_mycobot = b'openled'
dancing = b'zengjialiangdu'
nod_head = b'jianxiaoliangdu'
go_back = b'closeled'
# run frame
while True:
r += s.read()
if wake_up in r:
print("wake")
wake = True
r = b''
if wake and hi_mycobot in r:
print("Hi myCobot")
r = b''
if wake and dancing in r:
print("dancing")
r = b''
if wake and nod_head in r:
print('nod your head')
r = b''
if wake and go_back in r:
print('Go back')
r = b''
if wake and _exit in r:
print("exit")
r = b''
# print(r)scopy
Here are the test results
The pymycobot library is a robotic arm API call interface library developed by Elephant Robotics. The specific usage method is on Gitbook (provided by Elephant Robotics).
# import library
from pymycobot import Mycobot
import time
# Create an instance object for later use
mc = Mycobot('COM9',115200)
#Control the mechanical arm to move at a speed of 70 according to the angle in the list，send_angles([angles],speed)
mc.send_angles([0.87,(-50.44),47.28,0.35,(-0.43),(-0.26)],70)
# Execute the next command after a delay of x seconds (the movement of the robotic arm takes time)
time.sleep(x)copy
Next, combine the two parts of the code.
import serial
from pymycobot import MyCobot
import time
s = serial.Serial("COM6",115200)
mc = MyCobot('COM3',115200)
r = b''
wake_up = b'wakeup_uni'
_exit = b'exitUni'
hi_mycobot = b'openled'
dancing = b'zengjialiangdu'
nod_head = b'jianxiaoliangdu'
go_back = b'closeled'
wake = False
while True:
r += s.read()
if wake_up in r:
print("wake")
wake = True
r = b''
if wake and hi_mycobot in r:
print("Hi myCobot")
# say hi shake with you
mc.send_angles([0.87,(-50.44),47.28,0.35,(-0.43),(-0.26)],70)
time.sleep(1)
for count in range(3):
mc.send_angle(1,30,80)
time.sleep(1)
mc.send_angle(1,(-30),80)
time.sleep(1)
mc.send_angles([0,0,0,0,0,0],70)
r = b''
if wake and dancing in r:
print("dancing")
mc.send_angles([0,0,0,0,0,0],80)
time.sleep(1)
for count in range(2):
mc.send_angles([(-0.17),(-94.3),118.91,(-39.9),59.32,(-0.52)],80)
time.sleep(1.2)
mc.send_angles([67.85,(-3.42),(-116.98),106.52,23.11,(-0.52)],80)
time.sleep(1.7)
mc.send_angles([(-38.14),(-115.04),116.63,69.69,3.25,(-11.6)],80)
time.sleep(1.7)
mc.send_angles([2.72,(-26.19),140.27,(-110.74),(-6.15),(-11.25)],80)
time.sleep(1)
mc.send_angles([0,0,0,0,0,0],80)
r = b''
if wake and nod_head in r:
print('nod your head')
mc.send_angles([0,0,0,0,0,0],70)
time.sleep(1)
mc.send_angles([3.07,(-86.3),75.32,11.86,2.72,(-0.35)],70)
time.sleep(0.5)
for count in range(4):
mc.send_angle(4,13,70)
time.sleep(0.5)
mc.send_angle(4,(-60),70)
time.sleep(1)
mc.send_angle(4,13,70)
time.sleep(0.5)
mc.send_angles([0,0,0,0,0,0],70)
r = b''
if wake and go_back in r:
print('Go back')
mc.send_angles([12.83,(-138.95),156.09,(-12.3),(-12.91),35.41],70)
r = b''
if wake and _exit in r:
print("exit")
r = b''III. Experience and lessons learned
In the process of completing this project, I gained many valuable experiences and insights. First of all, I realized that the completion of a project depends not only on the mastery of technology, but also requires prior research and understanding. When I was researching the production of the speech recognition development board, I found that the workload was very heavy and it would be very difficult to implement. However, there are already many mature techniques and tools in the market. We just need to choose a reasonable combination and integration. Second, the successful completion of a project requires a clear framework and adequate preparation. In this project, I need to understand speech recognition algorithms, robotic arm design and control technology, master the usage and performance characteristics of hardware devices and software development tools.
In this project, I successfully combined speech recognition technology with robotic arm control technology to realize voice control of the robotic arm. Although the scale of this project is relatively small, it represents the application and development trend of artificial intelligence technology in real life. Although this is just a prototype of an AI project, its completion was a valuable experience for me. In future research, I will continue to explore more relevant information to further improve this project. If you have any good ideas, please leave a comment below.