Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added Accelerate support from HuggingFace - MultiGPU & distributed support! #4

Merged
merged 17 commits into from
Dec 20, 2023
Merged

Added Accelerate support from HuggingFace - MultiGPU & distributed support! #4

merged 17 commits into from
Dec 20, 2023

Conversation

gcervantes8
Copy link
Owner

@gcervantes8 gcervantes8 commented Dec 20, 2023 •
edited
Loading

Added Features

  • Optimizers are now available in the configuration file! You can easily run with 8 bit optimizers.
  • Mixed precision with bf16.
  • Multi-GPU support
  • Train loop modified! It is now more memory efficient. Saving up to 1 GB more.
  • Updated to Torchvision Transforms.v2
  • Refactored location of files, now there are only 2 public .py files in the src directory.

These changes add 2 more dependencies:

  • bitsandbytes - for their 8 bit optimizers
  • accelerate - to simplify the multi-gpu training

PyTorch requirement is moved from 1.9+ to 2.0+
Python requirement went from 3.7+ to 3.8+

@gcervantes8
Prototype using HF Accelerate library
a51ebfc
@gcervantes8
Cleaned up code using Accelerator, Note: I'm noticing mixed precision…
1597f3e 
… with Accelerator is not giving speed improvements
@gcervantes8
Cleaned up Steps for Generator and Discriminator, and cleaned up comp…
98cf1ec 
…iling and mxp code
@gcervantes8
Went back to quick training, by not using Accelerator's mxp
5008357
@gcervantes8
Fixed using MXP Accelerator, it is much faster now. Updated to transf…
ed1f3d7 
…orms.v2. Forced input to be float32, reduces VRAM significantly.
@gcervantes8
README small fix
7c450d3
@gcervantes8
Small improvements to the code all around, GANModel now has dtype
ce0dcc0
@gcervantes8
Gradient Accumulation is now reflected in the step number, and fixed …
795d96e 
…dtype of labels and latent vectors
@gcervantes8
Fixed bug in data normalization during evaluation
a4d7d02
@gcervantes8
Added HuggingFace Accelerate to README
6e4bd17
@gcervantes8
Fixed model saving
6868e65
@gcervantes8
Updated data transforms to newest package
a0542e5
@gcervantes8
Restructured file locations
74c79bc
@gcervantes8
Fixed imports, and added starting code for changing optimizer through…
7672bd0 
… config
@gcervantes8
Added optimizer parameter to configuration files
1b8993b
@gcervantes8
PyTorch 2.0+ is now required
995d96e
@gcervantes8
Removed support for Python 3.7, since PyTorch 2.0 no longer supports it
68325fd
@gcervantes8 gcervantes8 merged commit 7831e70 into master Dec 20, 2023
@gcervantes8 gcervantes8 deleted the hf-accelerate branch December 20, 2023 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@gcervantes8