NVIDIA Omniverse Blueprint: 3D Conditioning for Precise Visual Generative AI

Image generated with live interactive demo.

Developers can integrate NVIDIA NIM microservices to build applications that leverage controllable generative AI for efficient 3D scene creation.

In this workflow guide, we outline how to create a system that allows users to quickly generate and manipulate scene elements, using an espresso machine sample asset. We then describe how other NIM microservices, including USD Code, USD Search, Edify 360 and Edify 3D, can be used to extend that base workflow for enhanced features such as scene layout.

This guide includes:

1. The standard workflow to replicate what is on the NVIDIA Omniverse Blueprint web page.
2. Optional NIM that you could integrate on your own to further accelerate scene layout beyond the demonstration in the standard workflow guide.

NVIDIA NIM microservices are a set of accelerated inference microservices that allow developers to easily deploy AI models on NVIDIA GPUs anywhere.

For this workflow, we specifically explore microservices that enable developers to use generative backgrounds, while also taking advantage of OpenUSD for 3D application and workflow development.

Image generated with live interactive demo.

These workflows may include:

• SDXL + ComfyUI: A fast generative text-to-image model that can synthesize photorealistic images from a text prompt in a single network evaluation with a graph/nodes interface for advanced developers.

  The following NIM are optional microservices.

• Edify 360 NIM: Shutterstock Early Access preview of Generative 3D service for 360 HDRI (High Dynamic Range Image) generation. Trained on NVIDIA Edify using Shutterstock's licensed creative libraries.

• Edify 3D NIM: Shutterstock Generative 3D service for 3D asset generation. Trained on NVIDIA Edify using Shutterstock's licensed creative libraries.

• USD Code NIM: A language model that answers OpenUSD knowledge queries and generates USD Python code.

• USD Search NIM: An AI-powered search for OpenUSD data, 3D models, images, and assets using text or image-based inputs.

All microservices are currently available as a preview on the NVIDIA API Catalog, where developers can make API calls for evaluation.

By the end of this workflow guide, you will be able to run the application on it's own or if you want to customize your own kit application, you will learn how to create a custom workflow with AI to enable and accelerate your creative teams.

Table of Contents

• Workflow Summary
  • Standard Workflow
  • Workflow Diagram
• Get Started
  • System Requirements
  • Preview API Key (optional, required for custom workflow)
• Run the Application (no custom workflow)
• Create Your Own Application (optional, custom workflow)
  • Add Extensions
    • Configure Extensions with NIM
  • ComfyUI Overview
    • Requirements & Installation
    • Graph Setup
    • Configure Image Generation Service
    • Assigning References & Models
    • Graph Breakdown
    • Connecting Kit Application to Image Generation Service
• Prompting Best Practices
• Add Other NIMs(optional)
  • Scene Layout With NIM
  • Augment the Content Library with Edify
• Scene Structure and Assets
• Scene Navigation with ActionGraph
• Messaging for Web Based Front End
• Deployment
  • Creating a Deployable Service for ComfyUI
  • Building Containers from Source
  • Run Locally without Kubernetes
• Additional Usage
  • Batch Prompt Ideation
  • Improved Lighting
  • Reuse Masks
  • NSFW Filtering
• Known Issues and Limitations
• Troubleshooting