Potential Port Problems

[rstyd]

During this morning's meeting, we discussed potential issues with ports. Currently, the WFM, TM, and scheduler along with the neo4j instance each require a unique port for each user (neo4j actually requires 3). We need to figure out a solution for how to handle this at scale since each user will need a unique set of ports.

One potential solution is to just look for available ports when the user first configures bee. This could serve as a temporary fix, but there's potential for overlap later on. Another possible solution would be to use unix sockets instead of TCP/IP ports since most of our communication will be done locally. This has a few issues. Flask and neo4j do not appear to natively support unix sockets instead of ports. Additionally, when we have federated scheduling of jobs across multiple HPC resources the workflow manager and task manager will require TCP sockets to be accessed from another machine.

This issue kind of lies in the realm of good problems since we'd need a decent number of users for it to even show up, but it's something we should consider.

[jtronge]

I'm thinking it might be a good idea to try to use unix sockets. This would solve both our port problems, at least for our flask-based components, and also adds extra security at the file-system level that we're currently lacking. I think we might be able to do this with gunicorn: https://docs.gunicorn.org/en/stable/settings.html?highlight=socket#bind (I haven't tried it though)

As for neo4j, I guess we'd have to do exactly what you said about looking for ports (or switch to something else).