Several Issues and Errors in building pre-compiled driver image for Ubuntu 22.04

[runitmisra]

I am trying to build a pre-compiled driver image for kernel version 6.5.0-1020-aws since NVIDIA does not provide any pre-compiled driver images for kernels other than 5.15. I am following this documentation. I have encountered several issues when trying to build using the Dockerfile in the ubuntu22.04/precompiled directory.

For reference, I used the following command to build the image:

```bash
$ docker build -t driver --build-arg DRIVER_BRANCH=535 --build-arg DRIVER_VERSION=535.183.01 --build-arg KERNEL_VERSION=6.5.0-1020-aws ubuntu22.04/precompiled

• The CUDA version is hardcoded, so I had to change it manually in the Dockerfile (It should be a build argument)
• I get this error during execution of the RUN statement at Dockerfile line 21:

[+] Building 28.5s (8/17)                                                                                                                            docker:rancher-desktop
 => [internal] load build definition from Dockerfile                                                                                                                   0.0s
 => => transferring dockerfile: 3.29kB                                                                                                                                 0.0s
 => [internal] load metadata for nvcr.io/nvidia/cuda:12.2.2-base-ubuntu22.04                                                                                           0.0s
 => [internal] load .dockerignore                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                        0.0s
 => CACHED [ 1/13] FROM nvcr.io/nvidia/cuda:12.2.2-base-ubuntu22.04                                                                                                    0.0s
 => [internal] load build context                                                                                                                                      0.0s
 => => transferring context: 35B                                                                                                                                       0.0s
 => [ 2/13] RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections                                                                         0.2s
 => [ 3/13] RUN apt-key del 7fa2af80 &&     apt-key adv --fetch-keys "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/3bf863cc.pub"         2.8s
 => ERROR [ 4/13] RUN dpkg --add-architecture i386 &&     apt-get update && apt-get install -y --no-install-recommends         apt-utils         build-essential      25.5s
------
 > [ 4/13] RUN dpkg --add-architecture i386 &&     apt-get update && apt-get install -y --no-install-recommends         apt-utils         build-essential         ca-certificates         curl         kmod         file         libelf-dev         libglvnd-dev         pkg-config &&     rm -rf /var/lib/apt/lists/*:
0.805 Get:1 http://ports.ubuntu.com/ubuntu-ports jammy InRelease [270 kB]
1.197 Ign:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa  InRelease
1.591 Get:3 http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease [128 kB]
1.774 Get:4 http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease [127 kB]
1.955 Get:5 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease [129 kB]
2.141 Get:6 http://ports.ubuntu.com/ubuntu-ports jammy/restricted arm64 Packages [24.2 kB]
2.290 Get:7 http://ports.ubuntu.com/ubuntu-ports jammy/multiverse arm64 Packages [224 kB]
2.590 Get:8 http://ports.ubuntu.com/ubuntu-ports jammy/universe arm64 Packages [17.2 MB]
3.086 Ign:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa  InRelease
3.924 Ign:9 http://ports.ubuntu.com/ubuntu-ports jammy/multiverse i386 Packages
4.060 Ign:10 http://ports.ubuntu.com/ubuntu-ports jammy/main i386 Packages
4.197 Ign:11 http://ports.ubuntu.com/ubuntu-ports jammy/universe i386 Packages
4.335 Get:12 http://ports.ubuntu.com/ubuntu-ports jammy/main arm64 Packages [1758 kB]
4.885 Ign:13 http://ports.ubuntu.com/ubuntu-ports jammy/restricted i386 Packages
5.019 Ign:14 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main i386 Packages
5.153 Get:15 http://ports.ubuntu.com/ubuntu-ports jammy-updates/universe arm64 Packages [1473 kB]
5.982 Ign:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa  InRelease
6.174 Ign:16 http://ports.ubuntu.com/ubuntu-ports jammy-updates/universe i386 Packages
6.309 Get:17 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main arm64 Packages [2556 kB]
7.122 Get:18 http://ports.ubuntu.com/ubuntu-ports jammy-updates/restricted arm64 Packages [3006 kB]
8.052 Get:19 http://ports.ubuntu.com/ubuntu-ports jammy-updates/multiverse arm64 Packages [30.6 kB]
8.186 Ign:20 http://ports.ubuntu.com/ubuntu-ports jammy-updates/restricted i386 Packages
8.318 Ign:21 http://ports.ubuntu.com/ubuntu-ports jammy-updates/multiverse i386 Packages
8.451 Get:22 http://ports.ubuntu.com/ubuntu-ports jammy-backports/universe arm64 Packages [33.3 kB]
8.585 Ign:23 http://ports.ubuntu.com/ubuntu-ports jammy-backports/universe i386 Packages
8.889 Get:24 http://ports.ubuntu.com/ubuntu-ports jammy-backports/main arm64 Packages [81.0 kB]
9.462 Ign:25 http://ports.ubuntu.com/ubuntu-ports jammy-backports/main i386 Packages
9.607 Ign:26 http://ports.ubuntu.com/ubuntu-ports jammy-security/multiverse i386 Packages
9.751 Ign:27 http://ports.ubuntu.com/ubuntu-ports jammy-security/universe i386 Packages
9.898 Get:28 http://ports.ubuntu.com/ubuntu-ports jammy-security/multiverse arm64 Packages [24.2 kB]
10.08 Get:29 http://ports.ubuntu.com/ubuntu-ports jammy-security/restricted arm64 Packages [2879 kB]
10.90 Err:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa  InRelease
10.90   Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 23.212.249.207 443]
11.04 Get:30 http://ports.ubuntu.com/ubuntu-ports jammy-security/main arm64 Packages [2263 kB]
11.77 Ign:31 http://ports.ubuntu.com/ubuntu-ports jammy-security/restricted i386 Packages
11.91 Ign:32 http://ports.ubuntu.com/ubuntu-ports jammy-security/main i386 Packages
12.06 Get:33 http://ports.ubuntu.com/ubuntu-ports jammy-security/universe arm64 Packages [1186 kB]
12.50 Ign:9 http://ports.ubuntu.com/ubuntu-ports jammy/multiverse i386 Packages
12.79 Ign:10 http://ports.ubuntu.com/ubuntu-ports jammy/main i386 Packages
12.93 Ign:11 http://ports.ubuntu.com/ubuntu-ports jammy/universe i386 Packages
13.06 Ign:13 http://ports.ubuntu.com/ubuntu-ports jammy/restricted i386 Packages
13.21 Ign:14 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main i386 Packages
13.35 Ign:16 http://ports.ubuntu.com/ubuntu-ports jammy-updates/universe i386 Packages
13.49 Ign:20 http://ports.ubuntu.com/ubuntu-ports jammy-updates/restricted i386 Packages
13.62 Ign:21 http://ports.ubuntu.com/ubuntu-ports jammy-updates/multiverse i386 Packages
13.76 Ign:23 http://ports.ubuntu.com/ubuntu-ports jammy-backports/universe i386 Packages
13.90 Ign:25 http://ports.ubuntu.com/ubuntu-ports jammy-backports/main i386 Packages
14.04 Ign:26 http://ports.ubuntu.com/ubuntu-ports jammy-security/multiverse i386 Packages
14.18 Ign:27 http://ports.ubuntu.com/ubuntu-ports jammy-security/universe i386 Packages
14.48 Ign:31 http://ports.ubuntu.com/ubuntu-ports jammy-security/restricted i386 Packages
14.62 Ign:32 http://ports.ubuntu.com/ubuntu-ports jammy-security/main i386 Packages
14.77 Ign:9 http://ports.ubuntu.com/ubuntu-ports jammy/multiverse i386 Packages
14.91 Ign:10 http://ports.ubuntu.com/ubuntu-ports jammy/main i386 Packages
15.06 Ign:11 http://ports.ubuntu.com/ubuntu-ports jammy/universe i386 Packages
15.20 Ign:13 http://ports.ubuntu.com/ubuntu-ports jammy/restricted i386 Packages
15.35 Ign:14 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main i386 Packages
15.50 Ign:16 http://ports.ubuntu.com/ubuntu-ports jammy-updates/universe i386 Packages
15.64 Ign:20 http://ports.ubuntu.com/ubuntu-ports jammy-updates/restricted i386 Packages
15.79 Ign:21 http://ports.ubuntu.com/ubuntu-ports jammy-updates/multiverse i386 Packages
15.93 Ign:23 http://ports.ubuntu.com/ubuntu-ports jammy-backports/universe i386 Packages
16.20 Ign:25 http://ports.ubuntu.com/ubuntu-ports jammy-backports/main i386 Packages
16.33 Ign:26 http://ports.ubuntu.com/ubuntu-ports jammy-security/multiverse i386 Packages
16.47 Ign:27 http://ports.ubuntu.com/ubuntu-ports jammy-security/universe i386 Packages
16.60 Ign:31 http://ports.ubuntu.com/ubuntu-ports jammy-security/restricted i386 Packages
16.73 Ign:32 http://ports.ubuntu.com/ubuntu-ports jammy-security/main i386 Packages
16.86 Ign:9 http://ports.ubuntu.com/ubuntu-ports jammy/multiverse i386 Packages
16.99 Ign:10 http://ports.ubuntu.com/ubuntu-ports jammy/main i386 Packages
17.12 Ign:11 http://ports.ubuntu.com/ubuntu-ports jammy/universe i386 Packages
17.25 Ign:13 http://ports.ubuntu.com/ubuntu-ports jammy/restricted i386 Packages
17.38 Ign:14 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main i386 Packages
17.52 Ign:16 http://ports.ubuntu.com/ubuntu-ports jammy-updates/universe i386 Packages
17.81 Ign:20 http://ports.ubuntu.com/ubuntu-ports jammy-updates/restricted i386 Packages
17.96 Ign:21 http://ports.ubuntu.com/ubuntu-ports jammy-updates/multiverse i386 Packages
18.11 Ign:23 http://ports.ubuntu.com/ubuntu-ports jammy-backports/universe i386 Packages
18.25 Ign:25 http://ports.ubuntu.com/ubuntu-ports jammy-backports/main i386 Packages
18.40 Ign:26 http://ports.ubuntu.com/ubuntu-ports jammy-security/multiverse i386 Packages
18.54 Ign:27 http://ports.ubuntu.com/ubuntu-ports jammy-security/universe i386 Packages
18.69 Ign:31 http://ports.ubuntu.com/ubuntu-ports jammy-security/restricted i386 Packages
18.83 Ign:32 http://ports.ubuntu.com/ubuntu-ports jammy-security/main i386 Packages
18.98 Ign:9 http://ports.ubuntu.com/ubuntu-ports jammy/multiverse i386 Packages
19.13 Ign:10 http://ports.ubuntu.com/ubuntu-ports jammy/main i386 Packages
19.27 Ign:11 http://ports.ubuntu.com/ubuntu-ports jammy/universe i386 Packages
19.59 Ign:13 http://ports.ubuntu.com/ubuntu-ports jammy/restricted i386 Packages
19.75 Ign:14 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main i386 Packages
19.90 Ign:16 http://ports.ubuntu.com/ubuntu-ports jammy-updates/universe i386 Packages
20.06 Ign:20 http://ports.ubuntu.com/ubuntu-ports jammy-updates/restricted i386 Packages
20.21 Ign:21 http://ports.ubuntu.com/ubuntu-ports jammy-updates/multiverse i386 Packages
20.37 Ign:23 http://ports.ubuntu.com/ubuntu-ports jammy-backports/universe i386 Packages
20.52 Ign:25 http://ports.ubuntu.com/ubuntu-ports jammy-backports/main i386 Packages
20.68 Ign:26 http://ports.ubuntu.com/ubuntu-ports jammy-security/multiverse i386 Packages
20.83 Ign:27 http://ports.ubuntu.com/ubuntu-ports jammy-security/universe i386 Packages
20.98 Ign:31 http://ports.ubuntu.com/ubuntu-ports jammy-security/restricted i386 Packages
21.14 Ign:32 http://ports.ubuntu.com/ubuntu-ports jammy-security/main i386 Packages
21.44 Err:9 http://ports.ubuntu.com/ubuntu-ports jammy/multiverse i386 Packages
21.44   404  Not Found [IP: 185.125.190.36 80]
21.57 Ign:10 http://ports.ubuntu.com/ubuntu-ports jammy/main i386 Packages
21.72 Ign:11 http://ports.ubuntu.com/ubuntu-ports jammy/universe i386 Packages
21.86 Ign:13 http://ports.ubuntu.com/ubuntu-ports jammy/restricted i386 Packages
22.00 Err:14 http://ports.ubuntu.com/ubuntu-ports jammy-updates/main i386 Packages
22.00   404  Not Found [IP: 185.125.190.39 80]
24.32 Ign:16 http://ports.ubuntu.com/ubuntu-ports jammy-updates/universe i386 Packages
24.47 Ign:20 http://ports.ubuntu.com/ubuntu-ports jammy-updates/restricted i386 Packages
24.61 Ign:21 http://ports.ubuntu.com/ubuntu-ports jammy-updates/multiverse i386 Packages
24.76 Err:23 http://ports.ubuntu.com/ubuntu-ports jammy-backports/universe i386 Packages
24.76   404  Not Found [IP: 185.125.190.39 80]
24.90 Ign:25 http://ports.ubuntu.com/ubuntu-ports jammy-backports/main i386 Packages
25.05 Err:26 http://ports.ubuntu.com/ubuntu-ports jammy-security/multiverse i386 Packages
25.05   404  Not Found [IP: 185.125.190.36 80]
25.19 Ign:27 http://ports.ubuntu.com/ubuntu-ports jammy-security/universe i386 Packages
25.34 Ign:31 http://ports.ubuntu.com/ubuntu-ports jammy-security/restricted i386 Packages
25.48 Ign:32 http://ports.ubuntu.com/ubuntu-ports jammy-security/main i386 Packages
25.49 Fetched 33.4 MB in 25s (1326 kB/s)
25.49 Reading package lists...
25.49 W: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa/InRelease  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 23.212.249.207 443]
25.49 E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/jammy/multiverse/binary-i386/Packages  404  Not Found [IP: 185.125.190.36 80]
25.49 E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/jammy-updates/main/binary-i386/Packages  404  Not Found [IP: 185.125.190.39 80]
25.49 E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/jammy-backports/universe/binary-i386/Packages  404  Not Found [IP: 185.125.190.39 80]
25.49 E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/dists/jammy-security/multiverse/binary-i386/Packages  404  Not Found [IP: 185.125.190.36 80]
25.49 W: Some index files failed to download. They have been ignored, or old ones used instead.
------
Dockerfile:21
--------------------
  20 |
  21 | >>> RUN dpkg --add-architecture i386 && \
  22 | >>>     apt-get update && apt-get install -y --no-install-recommends \
  23 | >>>         apt-utils \
  24 | >>>         build-essential \
  25 | >>>         ca-certificates \
  26 | >>>         curl \
  27 | >>>         kmod \
  28 | >>>         file \
  29 | >>>         libelf-dev \
  30 | >>>         libglvnd-dev \
  31 | >>>         pkg-config && \
  32 | >>>     rm -rf /var/lib/apt/lists/*
  33 |
--------------------
ERROR: failed to solve: process "/bin/sh -c dpkg --add-architecture i386 &&     apt-get update && apt-get install -y --no-install-recommends         apt-utils         build-essential         ca-certificates         curl         kmod         file         libelf-dev         libglvnd-dev         pkg-config &&     rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100

Here it is clearly visible that several packages/repos are returning 404 error. I had to manually remove this line: dpkg --add-architecture i386 to make it work.

Even after fixing the above, Certificate authority for https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa is Unknown. I can see that ca-certificates package gets installed, but still the CA for https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa is unknown.

Need help fixing this. We are trying to upgrade our EKS cluster to k8s v1.31 and have been running into roadblocks due to nvidia gpu drivers. Refer to this issue for another problem we had: NVIDIA/gpu-operator#1220