On April 23, the Perceptual Computing Laboratory of Carnegie Mellon University put its OpenPose, a set of libraries that can read human body language, on GitHub, and open-sourced the core in June and July. Face and hand recognition source code. Source URL: https://github.com/CMU-Perceptual-Computing-Lab/openpose
According to the information on GitHub, OpenPose is the world's first C++ library for real-time multi-person keypoint detection and multi-threading. It belongs to Carnegie Mellon University and was just launched in June, aiming to promote artificial intelligence research and education. The CMU AI Program. Anyone, as long as there is no commercial use involved, can use it to build their own limb tracking system.
There are 6 developers of the entire OpenPose project, namely Ginés Hidalgo Martínez, Zhe Cao, Shih-En Wei, assistant researchers from the School of Robotics, PhD students Tomas Simon, Hanbyul Joo, and associate professor Yaser Sheikh who provided guidance. And OpenPose is actually the result of a number of computer vision projects they are doing, such as real-time multi-person 2D pose estimation, dynamic 3D reconstruction, and hand key point detection.
In addition, they also developed a face tracking library IntraFace and a large multi-view system Panoptic Studio shaped like an igloo. These have also been used in the development of OpenPose to expand its functions. Therefore, now OpenPose can not only track human torso and limbs, Even facial movements and individual fingers can be captured.
The general process of recognition is that first a 2D image is captured by the camera, and then the keypoint detector in OpenPose will identify and mark the parts of the body, helping the body tracking algorithm to understand the performance of each pose at different angles, and 3D color. The form of a stickman is presented. The recognition process is closely related to the camera system and computing power. Therefore, in Panoptic Studio, which consists of more than 500 cameras, the performance of OpenPose is very exaggerated. It can detect 130 key points of the human body in real time, and run it on a personal computer to track people. The more, the longer the calculation and rendering time, and it is not impossible to take more than 100 hours. Therefore, OpenPose needs to configure the multi-threaded module to speed up the processing.
In general, OpenPose is equivalent to an upgraded version of the body tracking technology used in previous somatosensory games, but compared to Microsoft Kinect tracking 20 key points, OpenPose is much more detailed. For the same action, Kinect perceives that a person is lifting hand, and OpenPose can observe that the person is actually pointing a finger at something. In terms of facial tracking, the entire head in Kinect is just a point, while in OpenPose, the eyebrows, eyes, nose, and mouth can be depicted by dozens of key points, not to mention body language, even expressions can be recognized.
In the words of developer Yaser Sheikh himself, OpenPose essentially opens up a new way of human-computer interaction. Compared with keyboard and mouse, human body movements and facial expressions can express much richer content than keyboard and mouse. For example, current somatosensory devices cannot tell the difference. Is this dancing person in front of him excited or out of anger? And vent.
When OpenPose is used in the fashionable artificial intelligence and VR/AR fields, even if it is separated from the keyboard and mouse, there will be no obstacles in the communication between people and computers. Instead, it is closer to the communication between people in reality and more natural. The device can be regarded as an object that can roughly understand your emotions. Then, in fields such as rehabilitation therapy and social interaction, artificial intelligence can figure out the user's psychology based on the situation at that time and provide more personalized solutions. Or, when it "sees" a group of people waiting for a red light, and suddenly a person walks across the road, it can immediately issue a warning, which is also one of the broader applications of multi-person tracking.
At present, the open source of OpenPose has attracted thousands of users to participate in the improvement, and more than 20 enterprises including the automobile group have also become interested in this project. The Perceptual Computing Lab is developing software downloads and commercial licenses, which will soon be used in real production and life.