Offscreen rendering with mjr_render is extremely slow

[michaelala25]

Intro

Hi!

I am a graduate student using mujoco for research into ML for robotics.

My setup

I'm using MuJoCo version 3.2.1, Python version 3.10.14, Ubuntu 22.04.4 LTS with libglfw3-dev 3.3.6-1. Output of glxinfo | grep "OpenGL version" is OpenGL version string: 4.6 (Compatibility Profile) Mesa 23.2.1-1ubuntu3.1~22.04.2.

My question

I'm posting this question because I'm not really sure how to proceed into investigating this issue and I'd like some help. I'm trying to render both to a window and offscreen simultaneously, and for some reason offscreen rendering takes astronomically longer. I'd like to render at around 20Hz and a single call to render offscreen is taking >50ms right now, with the culprit being mjr_render and mjr_readPixels.

Below is the output of cProfile.run on my env.render() call:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.008    0.008 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 __init__.py:1289(window_should_close)
        1    0.003    0.003    0.003    0.003 __init__.py:1388(get_framebuffer_size)
        1    0.001    0.001    0.001    0.001 __init__.py:1851(poll_events)
        1    0.000    0.000    0.000    0.000 __init__.py:2347(make_context_current)
        1    0.002    0.002    0.002    0.002 __init__.py:2370(swap_buffers)
        5    0.000    0.000    0.000    0.000 __init__.py:684(errcheck)
        1    0.000    0.000    0.008    0.008 mujoco_sim.py:143(render)
        1    0.000    0.000    0.008    0.008 env.py:205(render)
        2    0.000    0.000    0.000    0.000 env.py:340(get_curr_time)
        1    0.000    0.000    0.000    0.000 renderer.py:349(make_context_current)
        1    0.000    0.000    0.000    0.000 renderer.py:352(terminated)
        1    0.000    0.000    0.008    0.008 renderer.py:387(render)
        1    0.000    0.000    0.008    0.008 renderer.py:403(update)
        1    0.000    0.000    0.000    0.000 renderer.py:620(_create_overlay)
       14    0.000    0.000    0.000    0.000 renderer.py:92(add_overlay)
        2    0.000    0.000    0.000    0.000 {built-in method _ctypes.pointer}
        1    0.000    0.000    0.008    0.008 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 {built-in method builtins.round}
        1    0.000    0.000    0.000    0.000 {built-in method mujoco._functions.mjv_updateScene}
        2    0.000    0.000    0.000    0.000 {built-in method mujoco._render.mjr_overlay}
        1    0.000    0.000    0.000    0.000 {built-in method mujoco._render.mjr_render}

Notice that the calls to mjr_render and mjr_overlay take <1ms here.

Below is the output of cProfile.run on my camera.render() call (offscreen rendering):

    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.069    0.069 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 __init__.py:2347(make_context_current)
        1    0.000    0.000    0.000    0.000 __init__.py:30(make_current)
        1    0.000    0.000    0.000    0.000 __init__.py:684(errcheck)
        1    0.000    0.000    0.000    0.000 mujoco_sim.py:501(compile)
        1    0.000    0.000    0.069    0.069 mujoco_sim.py:738(render)
        1    0.000    0.000    0.000    0.000 renderer.py:205(make_context_current)
        1    0.000    0.000    0.069    0.069 renderer.py:214(render)
        1    0.000    0.000    0.069    0.069 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 {built-in method mujoco._functions.mjv_updateScene}
        1    0.055    0.055    0.055    0.055 {built-in method mujoco._render.mjr_readPixels}
        1    0.012    0.012    0.012    0.012 {built-in method mujoco._render.mjr_render}}

Now for some reason mjr_render is taking 12ms and mjr_readPixels is taking a whopping 55ms. This is at an offscreen viewport resolution set to 1920 x 1080 but curiously even with a viewport resolution of 64 x 64, the mjr_readPixels call takes 30ms, so it appears the bottleneck doesn't have to do too much with resolution. Additionally I'd like to cautiously add that the profiler output indicates the issue is isolated to solely mujoco calls, and doesn't appear to have anything to do with my code (though perhaps I simply haven't spotted it yet).

Frankly I'm at a loss for how to even investigate this further, as GLFW and rendering in general isn't really my area of expertise. One thing I know is that GLFW is the only OpenGL interface that seems to work on my system (neither EGL nor osmesa seem to work). I'm not even sure how that piece of information would be relevant but, figured I'd include it.

What should I do next to investigate the source of this problem?

Minimal model and/or code that explain my question

No response

Confirmations

• I searched the latest documentation thoroughly before posting.
• I searched previous Issues and Discussions, I am certain this has not been raised before.