Depth Pro: Sharp Monocular Metric Depth in Less Than a Second!

[HighOnBuffs]

release code & weights at https://github.com/apple/ml-depth-pro
can this be implemented? how does it compare to any_depth_L2

[nagadomi]

It seems to only support Depth Resolution=1536.
It may be great for high-resolution input, but it is inefficient and too slow for low-resolution input.
If there is an efficient way to handle low resolution inputs(like 360p), I can support it with iw3.

[nagadomi]

I have added support for this to depth_pro branch.
However, this model has heavy depth flickering problems.
Maybe it is unstable to minor image differences such as compression artifacts.
So the change may be discarded.

[gituser123456789000]

Thank you for adding a Depth Pro branch.. I'll be trying it soon. First impressions of the demo results I tried look good.. slightly more detailed and certainly higher res than Depth-Anything v2. Maybe it's not good for video... I see here, people saying there's a lot of flicker, but it seems like it may be great for single images. I'll see how the accuracy is soon.

[nagadomi]

I will not remove the model/code, but currently it is hidden on the GUI.
When supporting MoGe, I wish to address the video flickering problem by referring to MoGe's method, if possible.

[gituser123456789000]

My first time hearing about MoGe also. It looks great... like it's nearing 2D to VR180 capability if there were some sort of inpainting of the empty areas. Very impressive accuracy.

https://wangrc.site/MoGePage/

[gituser123456789000]

Looking at those MoGe examples, I can imagine the future of 2D to VR180. That's going to be amazing, standing inside movies and photos.

Or VR360*, whichever.. 6DOF, 6 degrees of freedom, walking around in photos and movies

[andy500]

how do l add this model to your program to work

[loawizard]

you need to download the specific branch connected to that and then download the model and add it to the checkpoints

[andy500]

where do l get the specific branch connected to download

[andy500]

have found it thanks

[loawizard]

it's a flicker fest depth pro. any v2 is better

[andy500]

get this error float' object has no attribute numerator why

[andy500]

l try it on my shadow pc working ok 8min 4k clip took about 50min depth anything v2 took 30min could not see problem in 3d full sbs depth pro hd gpu rtx6000

[luckyaigit]

Comparison of Depth Pro and Depth Anything V2: Differences and Advantages

1. Model Architecture and Design

• Depth Pro:

  • Utilizes a multi-scale Vision Transformer (ViT) architecture, combining high resolution and detailed depth estimation.
  • Focuses on generating high-resolution depth maps from a single image without relying on camera metadata such as focal length.
  • Excels in speed and high resolution, producing 2.25 million-pixel depth maps in just 0.3 seconds.

• Depth Anything V2:

  • Also based on ViT but emphasizes training with large-scale synthetic data and incorporating unlabeled real image data to improve the model's generalization capability.
  • Offers models of different sizes (ranging from 25M to 1.3B parameters) to support various applications, from lightweight to large-scale models.
  • Uses a more complex data processing pipeline, including synthetic data and large-scale pseudo-labeled real images.

2. Training Data

• Depth Pro:

  • Primarily trained on both synthetic and real datasets, focusing on images without camera metadata, which increases model flexibility.

• Depth Anything V2:

  • Initially trained entirely on synthetic data, followed by pseudo-labeling large-scale unlabeled real data for further training.
  • V2 mitigates the distribution shift issue from synthetic data by leveraging the diversity and real-world coverage of pseudo-labeled real images.

3. Inference Speed and Efficiency

• Depth Pro:

  • Highly focused on inference speed, capable of generating high-resolution depth maps in an extremely short time (0.3 seconds), ensuring real-time performance.

• Depth Anything V2:

  • Provides various options for inference speed depending on the model size, with the smallest model achieving 60ms and larger models requiring 213ms.

4. Detail Handling and Boundary Precision

• Depth Pro:

  • Enhances boundary precision, particularly in retaining fine details in complex scenes (e.g., hair, plants), with excellent detail preservation capabilities.

• Depth Anything V2:

  • Emphasizes strong robustness for complex scenes, including transparent objects and reflective surfaces, and excels in handling thin structures and transparent objects, especially in cross-domain scenarios involving synthetic and real-world data.

5. Application Scenarios and Suitability

• Depth Pro:

  • More suitable for real-time applications requiring fast high-resolution depth map generation, such as image editing and viewpoint synthesis.

• Depth Anything V2:

  • Due to its diverse model sizes and generalization capabilities, it is better suited for more complex scenarios, including handling transparent and reflective objects with stronger performance in complex layouts.

6. Evaluation and Benchmarking

• Depth Pro:

  • Primarily focuses on inference speed and high-resolution generation, excelling in efficiency and boundary detail.

• Depth Anything V2:

  • Introduced a new benchmark (DA-2K), covering diverse scenes and high-resolution images, evaluating its performance in handling transparent objects and complex reflective scenarios, demonstrating superior performance in complex and non-real image scenes.

Summary

• Depth Pro has significant advantages in speed, boundary precision, and high-resolution depth estimation, making it ideal for applications with high real-time requirements.
• Depth Anything V2 emphasizes model generalization, especially for complex and diverse scenarios, making it more suitable for a broader range of applications and cross-domain tasks.

[vizdotlife]

I've been comparing results between two different depth map models for human faces. The two models I'm doing the comparison for are Depth Pro and Depth Anything V2.

What I'm seeing in the results is Depth Pro is much better at recognizing hair strands and details around the outside of an object, but not nearly as good at estimating depth for the inner parts of facial features like the nose, shape around the eyes, cheeks, and chin compared to Depth Anything V2.

Based on my initial assessment it seems Depth Anything V2 is much better at generating depth maps that can be used to convert 2d images to 3d mesh in Blender. This is just based on a quick assessment.

[gituser123456789000]

I get errors:
No module named 'pillow_heif'
then once I add that and get it in python\Lib\site-packages I get
No module named '_pillow_heif'
with an underscore in front.. so I copy/paste the other folder and put an underscore in front of it
Then I get error
module '_pillow_heif' has no attribute 'get_lib_info'
no idea what to do from there