How to address textures data pipeline

[ManuCorrea]

As I mentioned in my PR: #295 the current data pipeline for the renderer seems that could have some problem with the texture data.

"I am also working on adding textures to the renderer. Using the current way all the information is dumped into the brax system json. Encoding de picture in the json makes the file really big and can slow down the process. I can try to finish out the texture feature so you could judge and we could discuss the implementation if necessary"

I think dumping the information of a texture(raw) in the JSON is quite insufficient and doesn't seem good practice, the RGBA data can be passed and doesn't seem big problem, since its a small amount of data.

The JSON file is the only current way of passing the system to the renderer as far as I know. The textures (raw readed/loaded from MuJoCo file reader) could be save into a jpeg image or some format as another option.

I could just implement it in JSON as first to test and develop the feature and then decide how to address it.

[btaba]

Hi @ManuCorrea thanks for taking a look! Are you dumping the raw pixels or the encoding into the json (dumping raw pixels doesn't seem like it'd be too efficient, but I'm not sure if that's what you're referring to)? Dumping the jpeg to disk and loading the jpeg in THREE js https://threejs.org/docs/#api/en/textures/Texture without writing to the json file might be a good approach

[ManuCorrea]

Yes saving the picture is what I think is the best approach. I may need to use OpenCV or Pillow to encode the image since MuJoCo gives the raw pixels. Once I close the color PR I will implement the textures in Brax.

With threejs you can also send raw pixels, but loading the jpeg makes more sense.