Implement new depthmap called Marigold

[loawizard]

Hello
I wonder if it could be possible to have the depthmaps from Marigold implemented in Iw3?

here is a link to GitHub and demo

https://github.com/prs-eth/marigold
https://huggingface.co/spaces/prs-eth/marigold-lcm

Thank you so much!

[donrikk]

https://huggingface.co/spaces/prs-eth/marigold-lcm

Update in the newest marigold depth map model seems it may be even better than depth anything I’ll have some sample pictures up in a few but here is the huggingface link to test out images and it even seems to be able to do videos aswell which is why im sharing this with you to see if u could impliment it! This is there latest fastest model the creator just got back to me and shared this with me he also answers my question. He says it is possible to code this to make it do image batching so technically this can be implemented into pictures and videos he just hasn’t done that with it but he says it’s possible! here are some examples of how accurate marigold is! lcm model is the fast model btw which is the one i used for these examples.

[donrikk]

and all of those examples are at 3 depth in 3Dcombinewith a convergence of .1 or in ur program it would be the equicalent to 1.0 convergence

[donrikk]

here is also the examples of the depth maps themselves! as u can see these maps are crazy accurate!

[donrikk]

now i do think the maps could use some tuning for video, it sometimes picks up to much in the image and can make distrant object seem unnatural so something like an autogain and contrast adjustment may be needed. but if u just add it as is we would be happy to test it out for u and give feedback!

[donrikk]

this example is a bit more on the normal side for most movie scenes and this is at only 1 depth, these maps could give usthe ability to get amazing 3D videos even at 1 depth!

[nagadomi]

I know of it, but have not implemented it because I did not think it is a practical model.
LCM version is probably improving on them, so it may be worth reviewing, but I'm not too enthusiastic about it.

[donrikk]

i share the same sentiment but it would be nice to test as it seems they are tryin to add video demo capabilties. honestly i think combining marigold and depth anything together like the zoedepth and anthing model are could hold some interesting benefits but i wouldnt wven know how t oachieve that but it must be possible as its done with zoe depth and anything model. especially this faster model it doesnt do to much for the foreground but it really helps with background details and edges i wish thre was a way to get the benefit of the background enhancement of marigold with the foreground accuracy of depth anything.

[nagadomi]

The original marigold is over 100x slower than DepthAnything.
LCM version is 10~20 times faster than that. But it is still very slow.

Maybe optimizing the batch size and resolution will make it much faster.
Once their video demo is released, we can get a more detailed processing time.

[nagadomi]

I have tried the video demo locally.
It was 0.9 FPS for 720x720 video (only depth, Linux RTX3070ti). With iw3 + Any_B, it is 40 FPS (depth + stereo generation).
It is slow, but faster than I thought.
Also, background depth is quite flickering.
You can try it by cloning the above huggingface space repository locally.

[gituser123456789000]

The Marigold depth maps are not accurate by the way, the mid-range is too bright, which causes a 'bulge', or 'hill' or 'wave' effect in the mid-range of the 3D effect.. this makes Marigold's maps very unnatural, which is easily seen if you compare Depth-Anything's results next to it side by side, back and forth

The only good thing about Marigold currently is the depth it pulls out of the background of an image. Through testing we've combined the foreground and the mid-range of Depth-Anything with the background of Marigold. The process hasn't been perfected, and is likely different for every image, but it can give excellent results. Depth-Anything's backgrounds are flat in comparison

[gituser123456789000]

This is the process:

I'm not sure how we can automate this, but I've tested with other images and results are along the same lines.. Marigold pulls out more background detail while performing poorly in the midrange, due to being too bright for its natural proper depth.

First I save the black and white Marigold results... then you have to invert the colors, because Marigold outputs in black front / white rear instead fo the opposite for pretty much every other depth map program.

In paint.net, I use the Magic Wand to select a bright white area.. change the Flood Mode on top to Global (to select all of the whitest, brightest parts of the Marigold depth map)... then adjust the Tolerance % on top to leave only the background details, like pictured here as an example:
https://media.discordapp.net/attachments/1218511652241150099/1222040713479327805/paintdotnet.PNG

Then delete all of that foreground and mid-range with the delete key on your keyboard

Then you make a layer over the Depth-Anything map (or you could have done this step first).. and copy/paste what's left of the Marigold background on top of it:
https://media.discordapp.net/attachments/1218511652241150099/1222053355300585584/og.png

Now if you know depth maps, you can see the background is too bright and would become foreground.. so we need to fix the brightness between the two depth maps. Darkening the background is a possibility, or part of the possibility.. it will depend on the image, but as you darken, you'll start losing the detail you want.. so in this case, we go to the Depth-Anything layer and brighten it until everything looks to be at the correct depth:
https://media.discordapp.net/attachments/1218511652241150099/1222053355866554378/og2.png

Then I merged the layers and darkened the combined depth maps image, to make the background more black (and therefore depth for the 3d effect). This will vary by image:
https://media.discordapp.net/attachments/1218511652241150099/1222054131146031104/og3.png

And this gave the result of Depth-Anything's high accuracy from the front to near the background, along with Marigold's detailed background depth. Here are the results, combined with the original image, in SBS, crosseyed, and anaglyph.

Original image used:
https://images.squarespace-cdn.com/content/v1/5a7f41ad8fd4d236a4ad76d0/1670121236520-E89JZMD2FV78P3TB6WO0/DAK_Pandora_LowAngleNight_DC_v04MerchHD.jpg

SBS:
https://media.discordapp.net/attachments/1218511652241150099/1222055415483994182/ggSBS.jpg

Crosseyed:
https://media.discordapp.net/attachments/1218511652241150099/1222055416133845032/ggCross.jpg

Anaglyph:
https://media.discordapp.net/attachments/1218511652241150099/1222055416503209995/ggAnaglyph.jpg

If someone can figure out how to automate this process.. or we get a new depth map program that can do this (be accurate throughout, while also pulling out background depth detail).. we're going to have a nice upgrade to our 2D to 3D conversions.

[nagadomi]

I can't see some of the images, so this may contain some misguided comments.

DepthAnything outputs Reletive Depth. In fact Relative Depth is Disparity, not Depth.
Disparity is the distance between two corresponding points in the left and right images of a stereo pair.
So it is normal for it to be flatter(darker) the farther away.

If you want to enhance the background disparity, you can edit using Tone Curve.

original output of Any_B resolution=770

Editing with GIMP's Color Curves tool.

In this example, banding artifact is occurring, but this is due to saving the image as 8-bit. iw3 handles the image as a floating point number, so this problem does not occur.

This Tone Curve editing feature is implemented in iw3 as --mapper option, which is used internally by --foreground-scale option. Currently, only curves that enhance foreground are supported.
It is easy to add negative options to --forground-scale that enhances the background.

[gituser123456789000]

I noticed the image links broke soon after I posted, so they should be attached below

I never got good results from using Foreground Scale. I always see unnatural distortions in the results compared to not using this setting.

[gituser123456789000]

I have an update request to add the ability to export the depth maps only. So we can see exactly what our changes are doing in the depth map, or for users who want the depth map so that they can modify them manually.

Edited To Add: Now that I've played with the Curve option.. I see this would probably be very difficult to get right just by putting in a value into the GUI. You really need to see the real-time effect that the changing Curve values make to the depth map. Even being able to see the curve making changes in real-time, it is still difficult to pull out extra detail without harming other parts of the depth map. So my second opinion on a Background Scale option would be that it may be interesting to tinker with, but I think getting actual improvements from it, without distortions, probably isn't likely to happen.

I see you had good looking results with this method though, which can't be argued with, so it seems you can probably pull this off.

[gituser123456789000]

Feedback on the dubios2 anaglyph addition: I get poor results with all images. I'm seeing dark, distorted bands in the cyan, right eye.. usually around the edges of the subject. These problem areas look to be a very bright red color.

[gituser123456789000]