Normals are not rendered correctly

[marian42]

Using the simple offscreen rendering example, my models have incorrect normals (see screenshot on the right).
If I display the mesh with trimesh's mesh.show(), it looks correct (left part of the screenshot).
This happens with all models that I tested and even if I supply a mesh without normals.

[marian42]

Ok, I figured out that I need to supply the smooth option:
pyrender.Mesh.from_trimesh(mesh, smooth = False).

You might want to change the default for that since most users probably don't want to have their model smoothened.

[LittleYuer]

@marian42, may I ask how to get normal image using pyrender?

Thanks.

[marian42]

Are you looking for a normal buffer of the rendered frame?

[LittleYuer]

Thanks for your reply @marian42.

Both normal buffer and normal map.

[marian42]

I recently had to generate a normal buffer with pyrender, so here is what I did:

Define a custom vertex shader in shaders/mesh.vert:

#version 330 core

// Vertex Attributes
layout(location = 0) in vec3 position;
layout(location = NORMAL_LOC) in vec3 normal;
layout(location = INST_M_LOC) in mat4 inst_m;

// Uniforms
uniform mat4 M;
uniform mat4 V;
uniform mat4 P;

// Outputs
out vec3 frag_position;
out vec3 frag_normal;

void main()
{
    gl_Position = P * V * M * inst_m * vec4(position, 1);
    frag_position = vec3(M * inst_m * vec4(position, 1.0));

    mat4 N = transpose(inverse(M * inst_m));
    frag_normal = normalize(vec3(N * vec4(normal, 0.0)));
}

Define a custom fragment shader in shaders/mesh.frag:

#version 330 core

in vec3 frag_position;
in vec3 frag_normal;

out vec4 frag_color;

void main()
{
    vec3 normal = normalize(frag_normal);

    frag_color = vec4(normal * 0.5 + 0.5, 1.0);
}

Now you can generate a normal buffer like this:

import trimesh
import pyrender
import numpy as np
from PIL import Image

class CustomShaderCache():
    def __init__(self):
        self.program = None

    def get_program(self, vertex_shader, fragment_shader, geometry_shader=None, defines=None):
        if self.program is None:
            self.program = pyrender.shader_program.ShaderProgram("shaders/mesh.vert", "shaders/mesh.frag", defines=defines)
        return self.program

camera_pose = np.array(
    [[ 1,  0,  0,  0],
     [ 0,  0, -1, -4],
     [ 0,  1,  0,  0],
     [ 0,  0,  0,  1]]
)

scene = pyrender.Scene(bg_color=(0, 0, 0))
scene.add(pyrender.Mesh.from_trimesh(trimesh.primitives.Capsule(), smooth = False))
camera = pyrender.PerspectiveCamera(yfov=np.pi / 3.0, aspectRatio=1.0, znear = 0.5, zfar = 40)
scene.add(camera, pose=camera_pose)

renderer = pyrender.OffscreenRenderer(512, 512)
renderer._renderer._program_cache = CustomShaderCache()

normals, depth = renderer.render(scene)
world_space_normals = normals / 255 * 2 - 1

image = Image.fromarray(normals, 'RGB')
image.show()

Now if you want a normal buffer it might be useful to disable antialiasing. To do that, I changed renderer.py of pyrender, which in my case was in /usr/local/lib/python3.5/dist-packages/pyrender/renderer.py. Here are my changes:

        # Clear it
        glClearColor(*scene.bg_color)
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
        glDisable(GL_MULTISAMPLE) ###### change here

# .....

            # Generate multisample buffer
            self._main_cb_ms, self._main_db_ms = glGenRenderbuffers(2)
            glBindRenderbuffer(GL_RENDERBUFFER, self._main_cb_ms)
            glRenderbufferStorage( ###### change here
                GL_RENDERBUFFER, GL_RGBA, ###### change here
                self.viewport_width, self.viewport_height
            )
            glBindRenderbuffer(GL_RENDERBUFFER, self._main_db_ms)
            glRenderbufferStorage( ###### change here
                GL_RENDERBUFFER, GL_DEPTH_COMPONENT24, ###### change here
                self.viewport_width, self.viewport_height
            )

Maybe @mmatl could add an option to pyrender to disable antialiasing.

Here is my result:

[LittleYuer]

I'll have a try.

Thank you very much for help @marian42.

[LogWell]

@LittleYuer map points to it's norms, and then render color image

[ndahlquist]

Thanks for your detailed description @marian42 ! Works like a charm

[ndahlquist]

One more addition: if you're on Mac, you'll also need to change this in render.py:

# Resize for macos if needed
if sys.platform == 'darwin':
    color_im = self._resize_image(color_im, False) # Change last parameter (anti-alias) to False.

[madhawav]

This is a very helpful thread. This is the second time I am referring to this thread, this time for a different project. It will be very useful if the developers can bring this as a feature.

Maybe we can add a RenderFlag called "DISABLE_ANTI_ALIASING" and then use that to condition multi-sample in a render call. (E,g,: #175). But, this mechanism cannot handle the macOS specific cases mentioned above by @ndahlquist.

@mmatl

[fishfishson]

Thanks for your scripts @marian42 ! Would you pls tell me the coordinate of your output normal map? world space or camera space?

[hummat]

Thanks @marian42, still works out of the box! One small addition: The CustomShaderCache might not (no longer?) be necessary. Simply point shader_dirs of the pyrender ShaderProgramCache class to the directory where you have created your shader files:

import pyrender
from pyrender.shader_program import ShaderProgramCache

renderer = pyrender.OffscreenRenderer(640, 480)
renderer._renderer._program_cache = ShaderProgramCache(shader_dir="shaders")

[Guptajakala]

I cannot compile the shader:
RuntimeError: ("Shader compile failure (0): b'0:5(8): error: compile-time constant expressions require GLSL 4.40 or ARB_enhanced_layouts\\n0:6(8): error: compile-time constant expressions require GLSL 4.40 or ARB_enhanced_layouts\\n'", [b'#version 330 core\n\n// Vertex Attributes\nlayout(location = 0) in vec3 position;\nlayout(location = NORMAL_LOC) in vec3 normal;\nlayout(location = INST_M_LOC) in mat4 inst_m;\n\n// Uniforms\nuniform mat4 M;\nuniform mat4 V;\nuniform mat4 P;\n\n// Outputs\nout vec3 frag_position;\nout vec3 frag_normal;\n\nvoid main()\n{\n gl_Position = P * V * M * inst_m * vec4(position, 1);\n frag_position = vec3(M * inst_m * vec4(position, 1.0));\n\n mat4 N = transpose(inverse(M * inst_m));\n frag_normal = normalize(vec3(N * vec4(normal, 0.0)));\n}'], GL_VERTEX_SHADER)

Any ideas?