Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement pipeline merge/node borrow #23

Open
insujang opened this issue Apr 11, 2024 · 0 comments
Open

Implement pipeline merge/node borrow #23

insujang opened this issue Apr 11, 2024 · 0 comments
Assignees

Comments

@insujang
Copy link
Member

During handling failures, if some pipeline doesn't have enough number of nodes, Oobleck is supposed to borrow nodes from other pipelines or merge pipelines.
Previous implementation had a prototype implementation, but during refactoring with colossalai backend it is gone. As a result, when there is no pipeline template for the remaining number of nodes in the pipeline, training terminates with an error in OobleckPlugin._instantiate_pipelines().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant