Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move model to device before wrapping with FSDP #1801

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Move model to device before wrapping with FSDP #1801

wants to merge 1 commit into from

Conversation

mieshkiwrk
Copy link

@mieshkiwrk mieshkiwrk commented Feb 26, 2025 •
edited
Loading

Seems to be a better approach due to known issue with moving view tensors to device within FSDP: pytorch/pytorch#147321

Also PyTorch recommendation in FSDP tutorial does it like that:
https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html

model = Net().to(rank)
model = FSDP(model)

@mieshkiwrk
Move model to device before wrapping with FSDP
67bc4c2 
Seems to be a better approach due to known issue with moving view tensors to device within FSDP: pytorch/pytorch#147321
@mieshkiwrk mieshkiwrk requested a review from regisss as a code owner February 26, 2025 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@regisss regisss Awaiting requested review from regisss regisss is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@mieshkiwrk