-
Notifications
You must be signed in to change notification settings - Fork 516
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve robustness for multiple client connections #803
Conversation
These changes look good to me. I did a quick test locally to confirm that these changes address the issues seen in #783, at least for my setup. I still get these errors intermittently but the rosbridge server continues working and accepting new connections.
|
I think we can simply remove the rosbridge_suite/rosbridge_server/src/rosbridge_server/websocket_handler.py Lines 207 to 211 in eadde22
|
Amazing! From my testing, both the hanging and the error spam are solved! I also intermittently got the error: [rosbridge_websocket-3] ERROR:tornado.application:Exception in callback <function main.<locals>.<lambda> at 0x7ffa191bbf70>
[rosbridge_websocket-3] Traceback (most recent call last):
[rosbridge_websocket-3] File "/usr/lib/python3/dist-packages/tornado/ioloop.py", line 1229, in _run
[rosbridge_websocket-3] return self.callback()
[rosbridge_websocket-3] File "/opt/greenroom/whiskey_ros_gs_external/lib/rosbridge_server/rosbridge_websocket", line 330, in <lambda>
[rosbridge_websocket-3] spin_callback = PeriodicCallback(lambda: rclpy.spin_once(node, timeout_sec=0.01), 1)
[rosbridge_websocket-3] File "/opt/ros/galactic/lib/python3.8/site-packages/rclpy/__init__.py", line 176, in spin_once
[rosbridge_websocket-3] executor.spin_once(timeout_sec=timeout_sec)
[rosbridge_websocket-3] File "/opt/ros/galactic/lib/python3.8/site-packages/rclpy/executors.py", line 712, in spin_once
[rosbridge_websocket-3] raise handler.exception()
[rosbridge_websocket-3] File "/opt/ros/galactic/lib/python3.8/site-packages/rclpy/task.py", line 239, in __call__
[rosbridge_websocket-3] self._handler.send(None)
[rosbridge_websocket-3] File "/opt/ros/galactic/lib/python3.8/site-packages/rclpy/executors.py", line 418, in handler
[rosbridge_websocket-3] await call_coroutine(entity, arg)
[rosbridge_websocket-3] File "/opt/ros/galactic/lib/python3.8/site-packages/rclpy/executors.py", line 343, in _execute_subscription
[rosbridge_websocket-3] await await_or_execute(sub.callback, msg)
[rosbridge_websocket-3] File "/opt/ros/galactic/lib/python3.8/site-packages/rclpy/executors.py", line 107, in await_or_execute
[rosbridge_websocket-3] return callback(*args)
[rosbridge_websocket-3] File "/opt/greenroom/whiskey_ros_gs_external/lib/python3.8/site-packages/rosbridge_library/internal/subscribers.py", line 218, in callback
[rosbridge_websocket-3] for callback in callbacks:
[rosbridge_websocket-3] RuntimeError: dictionary changed size during iteration
[rosbridge_websocket-3] [WARN] [1665466611.521004707] [rosbridge_websocket]: WebSocketClosedError: Tried to write to a closed websocket
[rosbridge_websocket-3] ERROR:tornado.application:Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7ffa182e38b0>, <Future finished exception=WebSocketClosedError()>)
[rosbridge_websocket-3] Traceback (most recent call last):
[rosbridge_websocket-3] File "/usr/lib/python3/dist-packages/tornado/ioloop.py", line 758, in _run_callback
[rosbridge_websocket-3] ret = callback()
[rosbridge_websocket-3] File "/usr/lib/python3/dist-packages/tornado/stack_context.py", line 300, in null_wrapper
[rosbridge_websocket-3] return fn(*args, **kwargs)
[rosbridge_websocket-3] File "/usr/lib/python3/dist-packages/tornado/ioloop.py", line 779, in _discard_future_result
[rosbridge_websocket-3] future.result()
[rosbridge_websocket-3] File "/usr/lib/python3/dist-packages/tornado/gen.py", line 326, in wrapper
[rosbridge_websocket-3] yielded = next(result)
[rosbridge_websocket-3] File "/opt/greenroom/whiskey_ros_gs_external/lib/python3.8/site-packages/rosbridge_server/websocket_handler.py", line 197, in prewrite_message
[rosbridge_websocket-3] future = self.write_message(message, binary)
[rosbridge_websocket-3] File "/usr/lib/python3/dist-packages/tornado/websocket.py", line 259, in write_message
[rosbridge_websocket-3] raise WebSocketClosedError()
[rosbridge_websocket-3] tornado.websocket.WebSocketClosedError But, this just happens once when the client disconnects. I really appreciate your and @p0rys work on this - has helped me a heap at just the time I needed it :) |
This exception is alarming, but I found the reason for it: In rosbridge_suite/rosbridge_library/src/rosbridge_library/internal/subscribers.py Lines 213 to 215 in eadde22
|
Small nitpick with 95905f8 The reason I think the comment would help is because other |
rosbridge_library/src/rosbridge_library/internal/subscribers.py
Outdated
Show resolved
Hide resolved
Co-authored-by: Jacob Bandes-Storch <[email protected]>
Sorry for commenting on a long-ago closed PR but I have a hunch that the issue I opened a few days ago might explain why the exceptions keep being thrown in the background and why they keep being thrown in the first place. If anyone is still interested in this feel free to chime in. |
Public API Changes
None
Description
Supersedes #741
Fixes #783
Fixes #758
This PR improves multi-client connection robustness by
The deadlock could be easily reproduced by rapidly connecting and disconnecting clients while one client was connected and receiving data.
subscribe.json
(example):