Researchers have unveiled Objaverse, a "massive open dataset of 3D objects with textual descriptions." It contains approximately 800 000 3D models with text descriptions.
The Objaverse dataset can be downloaded fromhuggingface and was created from a 3D model shared on Sketchfab, an online platform owned by Epic Games . The team only uses 3D models shared under a Creative Commons license. In other words, if you share 3D models on Sketchfab using a CC license, they may be included in the Objaverse, and this may be the case even if you use the NoAI tag, which is meant to prevent any use of AI.
NSDT tool recommendation: Three.js AI texture development kit - YOLO synthetic data generator - GLTF/GLB online editing - 3D model format online conversion - Programmable 3D scene editor - REVIT export 3D model plug-in - 3D model semantic search engine
1. Why create this data set?
Matt Deitke et al. explain why they created this dataset in their paperObjaverse: A Universe of Annotated 3D Objects. They emphasize that in terms of text or images, massive data sets are already available, which is why AI has made such huge advances in recent years/months. In other words, tools such as ChatGPT and StableDiffusion will not be able to create text or images without a data set to train on, regardless of whether these data sets are open and available for commercial use.
So far, only moderately sized 3D datasets are available, with limited diversity of object classes. Of course, this limits their use.
With large-scale data sets, new AI tools can be created. For example, you can train an AI to create 3D models from text descriptions, or create LOD/retopology assets, recognize what a 3D object should be, or animate a 3D character. Such datasets can also be used in the field of computer vision, not only as training data but also as benchmarks.
Comparison between Objaverse and existing 3D object datasets
2. Objaverse: Objects from Sketchfab
At this stage, you probably realize that datasets like the Objaverse have huge potential for artificial intelligence. It can be used both as training data and as a baseline. To create Objaverse, the researchers explained, they obtained 3D models, descriptions, and labels from Sketchfab. Objaverse contains over 800,000 assets designed by over 100,000 artists. This includes 3D scans, 3D models created from scratch, and even animated assets.
It should be emphasized that this dataset is derived only from assets shared using Creative Commons licenses (most of which follow the CC-By license).
3. What can Objaverse be used for?
Objaverse has just been released, but is already being used by several research projects. For example, Text2Tex is a text-to-texture tool trained using Objaverse:
Text2Tex generates high-quality textures of 3D meshes given text cues, an approach that incorporates inpainting into a pre-trained depth-aware image diffusion model to progressively synthesize high-resolution partial textures from multiple viewpoints. To avoid artifacts, Text2Tex proposes an automatic view sequence generation scheme to determine the next best view for updating partial textures. Extensive experiments show that the Text2Tex method is significantly better than existing text-driven methods and GAN-based methods.
Matt Deitke, lead author of the Objaverse paper, gives other examples, such as Zero-1-to-3, which can be obtained from A system for creating 3D models from a single image:
4. The first reaction to problems caused by Objaverse
The CC license itself allows scraping of resources, but this practice raises some issues.
Many artists and creators have long been uploading 3D models on Sketchfab, so some of their assets were shared before the rise of AI. Additionally, Objaverse doesn't seem to take into account the "NoAI" tag, which is now available on Sketchfab to publicly state that you don't want your assets to be used to train AI. Of course, in this case, the team behind Objaverse would not be the ones violating the license, and such abuse of assets shared on Sketchfab is already possible.
We should also emphasize that many 3D assets shared using Creative Commons licenses...are not actually licensed under CC. For example, do a quick search and you'll find an asset that was ripped from a Nintendo game and only slightly tweaked by the user who uploaded it. This asset is too close to the original copyrighted asset and cannot be shared under a CC license.
When they learned about Objaverse, some artists chose to delete their Sketchfab accounts, while others suggested (perhaps jokingly) that one way to deal with the problem was to upload "assets with non-manifold geometry to Sketchfab and tag them with common "label" to create bad data. In other words, the data set scraped from Sketchfab will not be used to train AI. Of course, this may be regarded as a junk asset by Sketchfab and other users of the platform.
5. How do I check if my 3D model is included in this dataset?
The creators of the Objaverse have built an exploration tool available here. Looking up your Sketchfab handle or entering the name of one of your 3D models should help check if your 3D model is included in the Objaverse dataset.
6. What does Sketchfab think about this?
Sketchfab CEO and Sketchfab co-founder Alban Denoyel (as a reminder, Sketchfab is owned by Epic Games and will soon be merged into Fab) responded on Twitter.
His answer highlighted four key points:
- He emphasized that "these models are aggregated at scale by the objaverse without their knowledge" and that "their advantage is absolutely zero when something like this happens."
- He also explained that the dataset was created before Sketchfab implemented the NoAI tag, which might explain why it wasn't taken into account.
- He also emphasized that the dataset relies on a "set of user-downloadable CC content." In other words, even if they didn't anticipate it, they did technically allow their assets to be used in this way.
- Last but not least, he explained that Sketchfab/Epic Games are "looking at what steps they can take."
The official Sketchfab account also posted several tweets on the topic, explaining that they "understand the artist's concerns and are looking into it."
It's unclear what Sketchfab can do about this. An interesting topic to explore relates to textual descriptions. The Sketchfab Terms of Use state that the license applies to "3D assets", but is the description part of the asset? If not, you can still grab 3D models shared on Sketchfab under a CC license and share them as datasets, but without descriptions. This would make the data set less interesting for training AI.
We asked Sketchfab if they could help us shed some light on this issue, and we'll update the article accordingly. We also asked the creators of Objaverse what their plans are (specifically, whether they will exclude 3D models on Sketchfab that now carry the NoAI label) and what they will do with 3D models on Sketchfab that are shared under a CC license, but this Apparently protected by copyright).
7. Uncertain times
The situation highlights that the “NoAI” label used by some digital art platforms is not a perfect solution to the rise of artificial intelligence, as the data may have already been zoned out by the time they are implemented. Objaverse also reminds us that uploading assets under a CC license may lead to situations that the artist cannot foresee.
Last but not least, the announcement highlights the fact that claiming a tool was trained on non-copyrighted data is not enough if the data has not been thoroughly inspected. In fact, the Objaverse does contain copyrighted material as well as the creations of artists who do share their work under CC licenses but do not want their work to be used to train artificial intelligence. This raises ethical and legal questions. Hopefully the team behind Objaverse will take these issues into consideration.
In the meantime, if you have a Sketchfab account, you can use the Settings/Account page to add a "NoAI" tag to all uploaded content (if you wish). This will assign the "NoAI" meta tag to all of your past and future uploads and disable generative AI from using them. Of course, this won't have any impact on data that may have been downloaded.
Original link:Objaverse large 3D data set - BimAnt