Add CUDA forward compatibility hook #906

elezar · 2025-02-06T16:58:40Z

With #877 the default behaviour of the NVIDIA Container Runtime / NVIDIA Container Runtime Hook was changed to not mount compat libraries from the container into the container. This removed "automatic" support for CUDA Forward compatibility.

This change attempts to address this by adding a createContainerHook that will create a file in /etc/ld.so.conf.d/ in the container to ensure that the /usr/local/cuda/compat libraries are added to the ldcache over the libraries mounted from the host. Here the provided driver version number is used to indicate whether this should be done or not.

In the case of the legacy runtime, this behaviour is only triggered if the allow-cuda-compat-libs-from-container feature flag is not enabled. The CDI spec generation has also been extended to include this hook.

elezar · 2025-02-06T22:25:43Z

cmd/nvidia-cdi-hook/compat-libs/compat-libs.go

+
+	// Create the 'compat-libs' command
+	c := cli.Command{
+		Name: "compat-libs",


Should this hook have a better name?

Currently it reads:

nvidia-cdi-hook compat-libs --driver-version 999.88.77

Changed this to:

nvidia-cdi-hook cuda-compat --host-driver-version=999.88.77

We could use this hook for other compat functions such as enhanced compatibility.

internal/modifier/gated.go

elezar · 2025-02-10T13:44:11Z

internal/modifier/gated.go

+		compatLibHookDiscoverer := discover.NewCUDACompatHookDiscoverer(logger, cfg.NVIDIACTKConfig.Path, driver)
+		discoverers = append(discoverers, compatLibHookDiscoverer)
+		if cfg.NVIDIAContainerRuntimeConfig.Mode == "legacy" {
+			ldcacheIpdateHookDiscoverer, err := discover.NewLDCacheUpdateHook(


Note: Since the nvidia-container-runtime-hook is invoked as a prestart hook, this is done BEFORE the createContainer hook that we insert above. This means that we need to once again run the update ldcache hook.

@klueska one option would be to create the file in /etc/ld.so.conf.d when invoking the nvidia-container-runtime-hook instead -- before calling out to the nvidia-container-cli. We don't have as ready access to the driver version, but we could extract it there.

If we create the file in nvidia-container-runtime-hook, then wouldn't that remove the need to add the cuda-compat createContainer hook altogether? And as a result, the forward compatibility support in the legacy stack would not require users to use NVIDIA container Runtime, correct?

I did try it, and my initial assumption was incorrect. Because we invoke ldconfig as:

argv = (char * []){cnt->cfg.ldconfig, "-f", "/etc/ld.so.conf", "-C", "/etc/ld.so.cache", cnt->cfg.libs_dir, cnt->cfg.libs32_dir, NULL};

in libnvidia-container, this means that the cnt->cfg.libs_dir and cnt->cfg.libs32_dir folders take precedence over the files in /etc/ld.so.conf.d and the CUDA libraries present there are used.

We could rework the libnvidia-container implementation further, but the intent of this change is also to provide the functionality for #910 so that we can remove the legacy code path by default.

Got it. So for legacy mode, we are adding two createContainer hooks ( the cuda-compat hook and the update-ldcache) to ensure the compat libs are used, the caveat being that we execute ldconfig twice.

cmd/nvidia-cdi-hook/cudacompat/cudacompat.go

cmd/nvidia-cdi-hook/cudacompat/cudacompat_test.go

internal/modifier/gated.go

This change adds an nvidia-cdi-hook cuda-compat hook that checks the container for cuda compat libs and updates /etc/ld.so.conf.d to include their parent folder if their driver major version is sufficient. This allows CUDA Forward Compatibility to be used when this is not available through the libnvidia-container. Signed-off-by: Evan Lezar <[email protected]>

This change adds the cuda-compat hook to the incomming OCI runtime spec if the allow-cuda-compat-libs-from-container feature flag is not enabled. An update-ldcache hook is also injected to ensure that the required folders are processed. Signed-off-by: Evan Lezar <[email protected]>

Signed-off-by: Evan Lezar <[email protected]>

cdesiniotis

I left a few questions. But overall this LGTM. Would be good to get another pair of eyes on this.

cdesiniotis · 2025-02-15T00:42:16Z

internal/runtime/runtime_factory.go

@@ -128,8 +128,8 @@ func supportedModifierTypes(mode string) []string {
 		return []string{"nvidia-hook-remover", "mode"}
 	case "csv":
 		// For CSV mode we support mode and feature-gated modification.
-		return []string{"nvidia-hook-remover", "mode", "feature-gated"}
+		return []string{"nvidia-hook-remover", "feature-gated", "mode"}


Question -- why does the ordering matter?

cdesiniotis · 2025-02-15T00:46:20Z

tests/e2e/nvidia-container-toolkit_test.go

@@ -166,4 +168,51 @@ var _ = Describe("docker", Ordered, func() {
 			Expect(referenceOutput).To(Equal(out4))
 		})
 	})
+
+	Describe("CUDA Forward compatibility", Ordered, func() {


Nice! I notice the test cases get skipped since our holodeck installs the latest driver: https://github.com/NVIDIA/nvidia-container-toolkit/actions/runs/13331020349/job/37235261007?pr=906#step:10:95

Any plans to downgrade the version of the driver we install in our e2e tests so we actually test CUDA forward compatibility?

cdesiniotis · 2025-02-15T00:59:25Z

internal/modifier/gated.go

@@ -78,5 +79,10 @@ func NewFeatureGatedModifier(logger logger.Interface, cfg *config.Config, image
 		discoverers = append(discoverers, d)
 	}

+	if !cfg.Features.AllowCUDACompatLibsFromContainer.IsEnabled() {


Question -- are features in the config only applicable to libnvidia-container? If yes, how can we make this more clear to users? From the user's perspective, it may be a bit confusing that compat libs do end up being used even if features.allow-cuda-compat-libs-from-container=false is set in the config.

AO696 approved these changes Feb 6, 2025

View reviewed changes

elezar force-pushed the add-compat-lib-hook branch from f21f7a6 to d617fb5 Compare February 6, 2025 22:24

elezar commented Feb 6, 2025

View reviewed changes

elezar force-pushed the add-compat-lib-hook branch 3 times, most recently from 606aed5 to ba30397 Compare February 6, 2025 23:08

elezar requested review from klueska, tariq1890 and cdesiniotis February 7, 2025 08:41

elezar changed the title ~~Add compat lib hook~~ Add CUDA forward compatibility hook Feb 7, 2025

elezar force-pushed the add-compat-lib-hook branch 3 times, most recently from 0c4dd3d to 0b50458 Compare February 7, 2025 13:07

elezar marked this pull request as ready for review February 7, 2025 13:10

elezar added the must-backport The changes in PR need to be backported to at least one stable release branch. label Feb 7, 2025

elezar force-pushed the add-compat-lib-hook branch 2 times, most recently from d1e97a0 to eeef750 Compare February 7, 2025 21:08

cdesiniotis reviewed Feb 8, 2025

View reviewed changes

internal/modifier/gated.go Show resolved Hide resolved

elezar force-pushed the add-compat-lib-hook branch 2 times, most recently from 88841fa to 8e0c69b Compare February 10, 2025 13:38

elezar commented Feb 10, 2025

View reviewed changes

elezar force-pushed the add-compat-lib-hook branch 3 times, most recently from 77bfb3c to 1140fcf Compare February 10, 2025 15:08

tariq1890 reviewed Feb 10, 2025

View reviewed changes

cmd/nvidia-cdi-hook/cudacompat/cudacompat.go Outdated Show resolved Hide resolved

elezar force-pushed the add-compat-lib-hook branch from 1140fcf to cb6b4f6 Compare February 10, 2025 18:18

cdesiniotis reviewed Feb 10, 2025

View reviewed changes

internal/modifier/gated.go Outdated Show resolved Hide resolved

elezar force-pushed the add-compat-lib-hook branch 2 times, most recently from f39a194 to aaf5ed8 Compare February 11, 2025 14:23

elezar force-pushed the add-compat-lib-hook branch 2 times, most recently from 12cc9aa to 0060b39 Compare February 14, 2025 13:35

elezar added 7 commits February 14, 2025 14:37

Add ldconfig hook in legacy mode

9cf60a4

Signed-off-by: Evan Lezar <[email protected]>

[no-relnote] Add basic CDI generate test

f63d72b

Signed-off-by: Evan Lezar <[email protected]>

Add cuda-compat hook to CDI spec generation

b352780

Signed-off-by: Evan Lezar <[email protected]>

Ensure that mode hook is executed last

1beef7a

Signed-off-by: Evan Lezar <[email protected]>

Add basic integration tests for forward compat

b47130b

Signed-off-by: Evan Lezar <[email protected]>

elezar force-pushed the add-compat-lib-hook branch 2 times, most recently from c0cf008 to b47130b Compare February 14, 2025 14:17

cdesiniotis reviewed Feb 15, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CUDA forward compatibility hook #906

Add CUDA forward compatibility hook #906

elezar commented Feb 6, 2025 •

edited

Loading

elezar Feb 6, 2025

elezar Feb 7, 2025 •

edited

Loading

elezar Feb 10, 2025

elezar Feb 10, 2025

cdesiniotis Feb 10, 2025 •

edited

Loading

elezar Feb 11, 2025 •

edited

Loading

cdesiniotis Feb 14, 2025

cdesiniotis left a comment

cdesiniotis Feb 15, 2025

cdesiniotis Feb 15, 2025 •

edited

Loading

cdesiniotis Feb 15, 2025

Add CUDA forward compatibility hook #906

Are you sure you want to change the base?

Add CUDA forward compatibility hook #906

Conversation

elezar commented Feb 6, 2025 • edited Loading

elezar Feb 6, 2025

Choose a reason for hiding this comment

elezar Feb 7, 2025 • edited Loading

Choose a reason for hiding this comment

elezar Feb 10, 2025

Choose a reason for hiding this comment

elezar Feb 10, 2025

Choose a reason for hiding this comment

cdesiniotis Feb 10, 2025 • edited Loading

Choose a reason for hiding this comment

elezar Feb 11, 2025 • edited Loading

Choose a reason for hiding this comment

cdesiniotis Feb 14, 2025

Choose a reason for hiding this comment

cdesiniotis left a comment

Choose a reason for hiding this comment

cdesiniotis Feb 15, 2025

Choose a reason for hiding this comment

cdesiniotis Feb 15, 2025 • edited Loading

Choose a reason for hiding this comment

cdesiniotis Feb 15, 2025

Choose a reason for hiding this comment

elezar commented Feb 6, 2025 •

edited

Loading

elezar Feb 7, 2025 •

edited

Loading

cdesiniotis Feb 10, 2025 •

edited

Loading

elezar Feb 11, 2025 •

edited

Loading

cdesiniotis Feb 15, 2025 •

edited

Loading