Researchers at Florida State University and Rutgers University have developed a 3D mesh technology based on Wi-Fi sensing: Wi-Mesh, which claims to recognize and create a reliable 3D human mesh, which can assist computer vision, Applications such as AR/VR positioning.
To put it simply, Wi-Mesh obtains 3D information of people in the room by analyzing the reflected signals received by multiple Wi-Fi antennas, and generates a three-dimensional grid. The Wi-Fi signal source is a low-cost indoor positioning method. The disadvantage is that the accuracy is not high, and it is difficult to improve the performance through algorithms. Wi-Fi signal can also supplement GPS positioning to solve the problem that GPS has no signal indoors.
It is understood that the research team focuses on cutting-edge Wi-Fi sensing research. Previous research (E-eyes, WiFinger system) has been able to sense a series of human activities and objects through Wi-Fi, such as large movements of the human body, or Small gesture changes, in addition to detecting sleep activity and recognizing everyday objects.
In the Wi-Mesh research, researchers focused on testing common Wi-Fi communication solutions to evaluate their 3D grid generation capabilities (such as whether they can accurately capture the height, weight, body shape, body proportions, and body deformations of different people). etc). In addition, a Wi-Fi sensing model is trained to classify daily activities, gestures, and match preset actions.
In fact, early research has shown that commercial Wi-Fi can classify some predefined human movements and detect subtle activities, such as vital signs (using Wi-Fi to detect sleep, falls, etc.). However, these systems cannot generate complex, fine-grained 3D meshes because 3D human body meshes typically have thousands of vertices, far more than can be identified for 3D pose capture. In addition, high-quality training data and deeper neural networks are required, and it is sensitive to various factors such as the use environment, making it difficult to be universal in different scenarios.
The birth of Wi-Mesh
In the latest study, however, the researchers demonstrated the ability of Wi-Fi signals to build 3D human meshes, which was previously difficult to achieve. Compared with the solution described above, the biggest difference of Wi-Mesh is that it uses multi-antenna Wi-Fi devices to analyze and deduce the shape of the human body from the data reflected from the Wi-Fi signal to the human body, and more accurately locate the 3D target. . The advancement of commercial Wi-Fi technology has made Wi-Mesh possible. Usually, the new generation of Wi-Fi 6/7 receivers are equipped with 8 to 16 antennas, which can capture the reflected signals after reaching the 3D surface of the human body and pass them through The signal incident angle and AoA information visualize the shape and movement of the human body.
Researchers pointed out that Wi-Mesh calculates the two-dimensional angle of arrival (2D AoA) through the reflected Wi-Fi signal, and calculates the 3D structure of the physical environment, the 3D visualization model of the human body and static objects, and then extracts the 2D AoA image of the human body. (Similar to the grayscale image captured by the camera), and then use the depth model to convert the 2D image into a 3D grid for various human-computer interactions. To put it simply, this method allows Wi-Fi to perceive the surrounding environment very well, just like giving it "vision".
The AoA positioning here refers to sensing the angle of arrival of signals sent by other devices through the hardware device, and calculating the relative direction of the receiving node and the sending node to calculate the position. Generally speaking, this positioning method mainly relies on a multi-antenna array. The Bluetooth 5.1 standard introduces the AoA feature, which can achieve centimeter-level precision positioning.
Strengths and Limitations
It has been verified that commercial Wi-Fi has 3D sensing and modeling capabilities. Therefore, in the future, Wi-Fi can not only communicate, but also visualize people and objects around it, bringing a new sensing method.
After a series of indoor experiments, the researchers found that the average vertex position error of Wi-Mesh is 2.81 cm, and the joint position recognition error is 2.4 cm, which is equivalent to the effect of dedicated 3D sensing hardware (such as two common RF positioning systems, FMCW radar, millimeter wave radar solution). It is worth noting that Wi-Mesh does not require additional hardware and can use existing Wi-Fi equipment, so it has an advantage in cost. In addition, the adaptability of Wi-Mesh in various scenarios is good enough, and it can also be recognized and 3D modeled by people who have never been in contact with it, and the running results are stable and effective.
Compared with 3D capture solutions based on cameras and vision algorithms, Wi-Mesh has the advantage that it is based on RF radio frequency, can penetrate walls and obstacles, and can be positioned even if there is visual occlusion or in an environment with insufficient light. Plus, Wi-Mesh can identify even dark, loose clothing.
Compared with traditional motion capture solutions, Wi-Mesh does not require additional wearable sensors or markers, so it is more natural and easier to use. It can also be used in existing Wi-Fi communication systems, and the cost is low enough to be suitable for a wide range of applications.
However, it still has some limitations, such as only one person in the space can be 3D modeled, and it is not suitable for crowded spaces such as train stations and shopping malls. The sensing distance is about a few meters and does not make full use of the full communication range of Wi-Fi. This is because Wi-Mesh relies on signal reflection and is less efficient than visual solutions. Therefore, more powerful directional antennas may be required, or multiple Wi-Fi devices may be covered. In addition, the computational cost also needs to be further reduced.
future application
Wi-Mesh has the ability to capture 3D human body data in a room and generate a grid, which can be used in a variety of potential scenarios, such as: AR/VR content development, virtual try-on, motion detection, animation/motion capture, Clothing body reconstruction and rendering and more. In fact, if it is used in offline large-space scenes, it may be possible to achieve a large-scale multiplayer AR game experience.
Researchers said: We are working on a smart home monitoring/security system that uses Wi-Fi signals for 3D positioning, which can be used in scenarios such as human body weight recognition (Re-ID), such as by detecting height, weight, and body proportions. , gait and other information for identification. Compared with camera monitoring, Wi-Fi positioning monitoring is more versatile. Even if the target changes appearance and posture, it can also identify the identity. In addition, it can also make up for the limitation of the visual solution due to occlusion. Reference: acm