Guide: Alibaba Amoy Technology has developed a new 3D AI algorithm, which can accurately search for the corresponding 3D model based on 2D images. The accuracy rate is greatly increased by 10%, which can reduce the threshold of 3D printing, VR viewing, and scene shopping guide. The research results have been included in the AI Conference NeurIPS 2020.
3D research is currently one of the most popular topics in the industry, and it is the core foundation for creating a 3D intelligent world. In the process of exploring the construction of 3D digital home furnishings, the latest 2D image search 3D model (IBSR) program proposed by the Alibaba Taoist Technology Department has been recognized by the academic community, and related papers have been accepted and published by the international artificial intelligence summit NeurIPS 2020.
IBSR aims to search out the CAD model corresponding to the object in the picture from a given 3D pool based on the 2D picture. With the rapid increase in the number of 3D CAD models, the research and design of high-precision IBSR systems is of great significance. It is conducive to quickly restore the real 3D scene, and also plays a vital role in the database-driven 3D model reconstruction problem. The general idea to solve the IBSR problem is to map the 2D graph and the 3D model to the same space and learn the similarity measurement function. In the past few years, top universities including Carnegie Mellon University, Massachusetts Institute of Technology, and Stanford University have all established IBSR benchmark data sets and invested in related research.
motivation
The biggest difficulty of IBSR lies in how to overcome the appearance gap between 2D images and 3D models, and decouple a unified geometric surface feature expression without texture information interference. We found that the IBSR problem has some special properties, including (1) 3D models are usually independent individuals (categories); (2) the same 3D model may correspond to different textured surfaces in reality. These characteristics have not been well modeled and learned in the past, making traditional methods unsatisfactory in fine 3D model recommendation.
Method overview
In response to these characteristics, Alibaba Taoist Technology has proposed a multi-view metric learning architecture driven by texture synthesis. Specifically, we design the Conditional Adversarial Generative Network (cGAN) for texture generation to create Hard Triplets for metric learning, so that the network can effectively suppress the adverse interference of the rich texture of the 2D image during the training process, so that more attention The geometric details of the object. At the same time, in order to make it easier for the network to learn geometric surface features, we use the saliency and perspective attention mechanism to eliminate as much as possible the cluttered background of 2D images and the interference of unconstrained projection perspective.
Experimental results
Our solution has achieved SOTA results on multiple open source data sets, including Pix3D, Stanford Cars, Comp Cars, and 3D-FUTURE, and the 3D model search accuracy rate exceeds the traditional method by about 10%. Among them, 3D-FUTURE is the industry's first large-scale 3D furniture model data set full of texture details, which is led and open sourced by the Alibaba Amoy Department Technology Department. Everyone is welcome to use and extract suggestions.
3D-FUTURE data set link: https://tianchi.aliyun.com/specials/promotion/alibaba-3d-future
✿Related reading
Author | Amoy Technology
Edit| Orange
Produced| Alibaba's new retail technology