[CS588] Lecture PR: Retrieval-Augmented 3D Generation
This post summarizes a lecture survey on Retrieval-Augmented 3D Generation presented at KAIST CS588. The survey explores how external 3D contexts can assist neural networks in generating 3D content more efficiently.
Survey Overview
The core objective of this survey is to address how to prevent hallucinations and incorporate up-to-date information without constantly updating model parameters. By leveraging retrieval, models can overcome the limitations of 3D datasets compared to 2D vision or NLP models.
Why 3D Generation Needs Retrieval
- Memory: Externalizing information reduces the burden on model parameters.
- Generalization: Accessing various views in a database improves model robustness.
- Dynamics: Retrieval allows for the integration of new information during the generation process.
- Details: High-frequency details can be retrieved to refine the final output.
3D Representation and Embedding
Diverse 3D Representations
The survey covers multiple ways to represent 3D data, including Voxels, Meshes, Point Clouds, and Implicit Functions.
- Point Cloud: A set of coordinates $(N, 3)$ that is memory-efficient and simple, though it lacks topological information.
Multi-modal Alignment (Connecting 3D to CLIP)
A key part of the survey is how 3D data is aligned with the CLIP space to enable cross-modal retrieval.
- ULIP / ULIP-2: Uses (image, text, point cloud) triplets to pre-align 3D encoders.
- Uni3D: A scalable framework that uses Vision Transformers (ViT) as a unified 3D encoder, scaling up to 1 billion parameters.
Frameworks for Retrieval-Augmented Generation
Capturing Similarity
The survey highlights that semantic similarity alone is insufficient for high-quality 3D generation.
- Fine-grained Similarity: Utilizing part-level retrieval (e.g., specific armrests or wheels for a chair).
- Style Similarity: Focusing on materials, textures, and the overall design of the object.
Diffusion Models in 3D
The survey categorizes 3D diffusion into two main paradigms:
- Direct 3D Diffusion: Generating 3D representations like point clouds (Point-E) or implicit functions (Shap-E).
- Multi-view 2D Diffusion: Generating 2D images from different angles and combining them into a 3D object.
Challenges and Solutions
| Challenge | Surveyed Solutions |
|---|---|
| Hallucination in Novel Views | Zero-1-to-3: Conditioning the model with reference images and camera poses. |
| Geometry Consistency | Phidias: Using Canonical Coordinate Maps (CCM) from retrieved references to guide geometry learning. |
Conclusion
This survey concludes that the main challenge in 3D generation lies in improving quality through spatio-temporal consistency and efficient data representations. Retrieving “useful” 3D objects as references is more important than simple matching to ensure the generated model is practical and detailed.