mjpython and matplotlib

[pab47]

Intro

Interested in learning about MuJoCo for fun.

My setup

I am using python 3.11, mujoco 3.1.6 on a mac intel (sonoma)

My question

I am trying to graph using matplotlib.pyplot. My code is something like this and I run it like this on a mac

mjpython <filename.py>

My question: This code crashes at the plt.figure(). Is there no way to use plt.figure with mjpython? If not, are there other options to make plots. NOTE: I could save the data in a csv and plot in a separate call to python3 but I want to avoid that.

import mujoco
import mujoco.viewer
import matplotlib.pyplot as plt

with mujoco.viewer.launch_passive(model=model, data=data) as viewer:
      <code>
     while viewer.is_running():
           <code>
          mujoco.mj_step(model, data)
          viewer.sync()

plt.figure()
<cdoe for plotting>

Minimal model and/or code that explain my question

If you encountered the issue in a complex model, please simplify it as much as possible (while still reproducing the issue).

Model:

minimal XML

<mujoco>
  <worldbody>
    <light pos="0 0 1"/>
    <geom type="sphere" size="1" rgba="1 0 0 1"/>
  </worldbody>
</mujoco>

Code:

import mujoco
import mediapy as media
model = mujoco.MjModel.from_xml_string(xml)
data = mujoco.MjData(model)

with mujoco.Renderer(model) as renderer:
  mujoco.mj_forward(model, data)
  renderer.update_scene(data)

  media.show_image(renderer.render())

Confirmations

• I searched the latest documentation thoroughly before posting.
• I searched previous Issues and Discussions, I am certain this has not been raised before.

[psclklnk]

I'd also be curious to know whether there is a fix for this problem.

I started encountering it when I needed to start using mjpython to run a passive viewer on my Mac while still plotting some data with matplotlib on the side.

In my particular case, the error message is the result of an NSInternalInconsistencyException with the message 'NSWindow should only be instantiated on the main thread!'.

Digging a bit into the problem, the exception is raised when instantiating the FigureManagerMac class in the backend_macosx.py file of the Matplotlib library (version 3.7.5).

This error makes sense to me given that mjpython seems to essentially make the user code run in a newly created thread which is different from the main thread.

[Balint-H]

There's probably multiple different ways to do it, but as help I can offer this tutorial exercise I wrote that uses real time plotting of a mujoco scene with matplotlib in one script, and PyQt in another. It is done with the interactive viewer not the passive, but it may be adapted in a straightforward fashion to it.

https://github.com/Balint-H/ssnr_sim/tree/main/challenges/Day_3/foot_drop

As @psclklnk suggests, the solution is with multithreading/multiprocessing, posting data in a queue from mujoco, and consuming it in the visualisation thread.

[psclklnk]

Thanks for the suggestion! I assume that you are not forced to run the Python script with the mjpython executable though, correct? (Since you are using the interactive viewer). I'm asking because I think the problem of @pab47 is related to mjpython specifically.

Regarding the use of multiprocessing to avoid the issue: The problem here is that we are not able to get a hold of the original main process if we execute with mjpython and therefore cannot execute our own rendering code in it, correct? So I guess we would need to use shared memory to communicate between a process that runs with mjpython for the passive viewer and a "normal" python process. That feels a bit heavy given that mjpython should be a drop-in replacement for the python executable.

BTW I also realized that the use of mjpython seems to break the Camera class of dm-control (Version 1.0.16). At least I get the same error as for matplotlib when trying to create an instance. The Renderer class that is shipped with MuJoCo works like a charm with mjpython and python.

Unfortunately, I do not have proper knowledge of the implementation differences of these two classes. Would be super helpful to have a person with in-depth knowledge explaining the difference in how they set up the OpenGL contexts.