You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I recently deployed flows in a Linux container environment and encountered errors that were not showing in my windows dev environment. After some testing, I believe that was due to the default start method of multiprocessing that Prefect LocalDaskExecutor would call in two different operating systems (spawn in Windows and fork in Linux). The errors would go away if I use None as the flow executor or set the number of workers to 1 in LocalDaskExecutor.
I don't think I could modify my patterns so that tasks could be forked safely, I want to ask if there is a way to set the multiprocess start method in Prefect LocalDaskExecutor in a Linux environment to spawn?
I did try setting the underlying multiprocessing to spawn (if I did the import order correctly), but did not seem to have resolved the problem:
import multiprocessing as mp
import ...
if __name__ == '__main__':
import ...
mp.set_start_method('spawn')
flow.run(executor=LocalDaskExecutor(scheduler="processes", num_workers=5))
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Version: Prefect 1.x
I recently deployed flows in a Linux container environment and encountered errors that were not showing in my windows dev environment. After some testing, I believe that was due to the default start method of multiprocessing that Prefect LocalDaskExecutor would call in two different operating systems (spawn in Windows and fork in Linux). The errors would go away if I use None as the flow executor or set the number of workers to 1 in LocalDaskExecutor.
I don't think I could modify my patterns so that tasks could be forked safely, I want to ask if there is a way to set the multiprocess start method in Prefect LocalDaskExecutor in a Linux environment to spawn?
I did try setting the underlying multiprocessing to spawn (if I did the import order correctly), but did not seem to have resolved the problem:
Beta Was this translation helpful? Give feedback.
All reactions