Cressy is sent from the concave non-si
qubit | public account QbitAI
This new achievement of MIT makes the fetching robot smarter!
Not only can it understand natural language instructions, but it can also pick up unseen objects.
Ma Ma no longer has to worry that I can't find something!
The researchers embed 2D features into 3D space to construct a feature field (F3RM) for controlling the robot.
In this way, image features and semantic data constructed in 2D images can be understood and used by 3D robots.
Not only is the operation simple, but the sample size required in the training process is also small.
Easy retrieval with low training samples
We can see that with the help of F3RM, the robot can adeptly pick up the target object.
Even if you want to find objects that the robot has not encountered before, it is not a problem.
For example... Dabai (doll).
For the same kind of items in the scene, they can be distinguished according to information such as color.
For example, pick up two different blue and red screwdrivers in the same scene.
Not only that, but the robot can also be asked to grasp specific locations of objects.
For example, for this cup, we can specify the robot to grab the cup body or handle.
In addition to the picking problem, you can also let the robot put the picked things in the designated position.
For example, put the cups on wooden and transparent supports respectively.
The team provided complete, unfiltered experimental results. They randomly selected out-of-distribution (outside the training set) test samples around the laboratory.
Among them, the feature field using CLIP ResNet features was successfully grasped and placed in more than 30% of the test samples (78%). On tasks based on open artificial language instructions, the success rate is 60%. The results are not cherry-picked and thus provide an objective description of how the eigenfield behaves in the zero-fine-tuning scenario.
So, how to use F3RM to help robots work?
Project 2D features into 3D space
The following picture roughly describes the workflow of using F3RM to help robots pick up items.
F3RM is a characteristic field. To make it work, we must first obtain relevant data.
The first two links in the figure below are to obtain F3RM information.
First, the robot scans the scene through the camera.
During the scanning process, RGB images from multiple angles will be obtained, and image features will be obtained at the same time.
Using NeRF technology, extract 2D density information from these images and project them into 3D space.
The extraction of image and density features uses the following algorithm:
This results in a 3D feature field of the scene, which can be used by the robot.
After obtaining the characteristic field, the robot also needs to know how to operate different objects in order to pick them up.
During this process, the robot will learn the corresponding arm movement information of six degrees of freedom.
If an unfamiliar scene is encountered, the similarity to known data is calculated.
Then by optimizing the action, the similarity is maximized to realize the operation of the unknown environment.
The process of natural language control is very similar to the previous step.
First, the feature information will be found from the CLIP data set according to the instructions, and the DEMO with the highest similarity will be retrieved from the knowledge base of the machine.
Then the predicted pose is also optimized to achieve the highest similarity.
After the optimization is completed, the object can be picked up by performing the corresponding action.
After such a process, a language-controlled fetching robot with a low sample size is obtained.
Team Profile
The research team members are all from MIT's CSAIL Laboratory (Computer Science and Artificial Intelligence Laboratory).
The laboratory is the largest laboratory at MIT, formed in 2003 by the merger of CS and AI laboratories.
The co-authors are Chinese doctoral student William Shen and Chinese postdoctoral fellow Yang Ge, supervised by Phillip Isola and Leslie Kaelbling. They come from MIT CSAIL (Computing and Artificial Intelligence Laboratory) and IAIFI (Institute of Artificial Intelligence and Fundamental Interaction). Yang Ge is the co-organizer of the 2023 CSAIL Embodied Intelligence Seminar.
Left: William Shen, Right: Yang Ge
Paper Address:
https://arxiv.org/abs/2308.07931
Project Homepage:
https://f3rm.github.io
MIT Embodied Intelligence Team
https://ei.csail.mit.edu/people.html
Embodied Intelligence Seminar Will
https://www.youtube.com/channel/UCnXGbvgu9071i3koFooncAw