Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the default of ngl be -1 #707

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+5 −5
Open

Make the default of ngl be -1 #707

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/ramalama.1.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ pass --group-add keep-groups to podman (default: False)
Needed to access the gpu on some systems, but has an impact on security, use with caution.

#### **--ngl**
number of gpu layers (default: 999)
number of gpu layers (default: -1)
Comment on lines 115 to +118
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Clarify the meaning of -1 for ngl.

The documentation should explicitly state what a value of -1 means for the number of GPU layers. Does it mean "use all available layers" or something else? Also, "gpu" should be "GPU".

Suggested change
Needed to access the gpu on some systems, but has an impact on security, use with caution.
#### **--ngl**
number of gpu layers (default: 999)
number of gpu layers (default: -1)
Needed to access the GPU on some systems, but has an impact on security, use with caution.
#### **--ngl**
number of GPU layers (default: -1). A value of -1 means use all available GPU layers

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with sorcery, you need to explain the value of -1?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rhatdan! I'm glad you agree. @, it's important to make this clear in the documentation so users understand how to use this option. Please update the documentation to explain the meaning of -1, as suggested in my previous comment. Let me know if you have any questions about how to best phrase this.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 magic numbers are bad.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your feedback, we will generate more comments like this in the future.


#### **--nocontainer**
do not run RamaLama in the default container (default: False)
Expand Down
2 changes: 1 addition & 1 deletion docs/ramalama.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@

# Default number of layers offloaded to the gpu
#
#ngl = 999
#ngl = -1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Explain -1 above


# Specify default port for services to listen on
#
Expand Down
2 changes: 1 addition & 1 deletion docs/ramalama.conf.5.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ RAMALAMA_IMAGE environment variable overrides this field.
Pass `--group-add keep-groups` to podman, when using podman.
In some cases this is needed to access the gpu from a rootless container

**ngl**=999
**ngl**=-1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please explain the meaning of -1.


Default number of layers to offload to the gpu

Expand Down
2 changes: 1 addition & 1 deletion ramalama/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ def configure_arguments(parser):
"--ngl",
dest="ngl",
type=int,
default=config.get("ngl", 999),
default=config.get("ngl", -1),
help="Number of layers to offload to the gpu, if available"
)
parser.add_argument(
Expand Down
2 changes: 1 addition & 1 deletion ramalama/model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ def setup_container(self, args):
def gpu_args(self, args, runner=False):
gpu_args = []
if (
args.gpu
args.gpu > 0
or os.getenv("HIP_VISIBLE_DEVICES")
or os.getenv("ASAHI_VISIBLE_DEVICES")
or os.getenv("CUDA_VISIBLE_DEVICES")
Expand Down
Loading