What's Changed

NOTE: This release is a unified release of the NVIDIA Container Toolkit that consists of the following packages:

• libnvidia-container 1.20.1
• nvidia-container-toolkit 1.20.1

The packages for this release are published to the libnvidia-container package repositories.

Fixes and Features

• CDI specifications and JIT-CDI mode now inject MIG management capability devices into containers. Set the NVIDIA_MIG_CONFIG_DEVICES or NVIDIA_MIG_MONITOR_DEVICES environment variable to all in a container with CAP_SYS_ADMIN to inject the /dev/nvidia-caps/ device nodes for MIG partition management. For supported values and constraints, refer to MIG Management Devices. For more information, refer to issue #1740 and PR #1947.
• Containers can now enforce soft and hard per-GPU CUDA memory limits. Set the NVIDIA_GPU_MEMORY_REQUEST and NVIDIA_GPU_MEMORY_LIMIT environment variables in MiB to control the limits. This feature requires an R615 or later driver. For more information, refer to PR #2075.
• Containers running with an R615 or later driver now receive the GPU firmware files that the driver requires at runtime. CDI specifications include the ucode_*.bin files automatically. For more information, refer to PR #2095.
• The nvidia-ctk runtime configure command now preserves existing Docker feature flags when enabling CDI. Previously, running the command could overwrite flags such as containerd-snapshotter. For more information, refer to PR #2049.
• Fixed an intermittent EBADF error in the create-symlinks hook that caused container creation to fail on busy hosts. The error occurred when the garbage collector closed a file descriptor while a system call was in progress. For more information, refer to PR #2082.
• Fixed a regression in the update-ldcache hook that caused container creation to fail under gVisor. The hook now uses a sealed memfd to execute ldconfig, which avoids mounting /proc inside the container root. For more information, refer to PR #2059.
• On WSL2, setting NVIDIA_VISIBLE_DEVICES=none now correctly prevents GPU device injection. For more information, refer to PR #2025.
• Fixed NVIDIA X.Org driver module mount paths so that containers render correctly when the host uses a custom ModulePath in xorg.conf.d. For more information, refer to PR #1980.
• GPU devices are now available in the CDI specification on first boot. The nvidia-cdi-refresh systemd service creates NVIDIA control device nodes before generating the specification. For more information, refer to PR #1979.
• GPU containers that start at boot no longer fail due to missing device nodes. The nvidia-cdi-refresh service is now ordered before Docker, containerd, and CRI-O. For more information, refer to PR #2019.
• Containers on read-only filesystems no longer fail to start when the application profile cannot be updated. The update-application-profile hook now ignores read-only filesystem errors. For more information, refer to PR #2028.
• Fixed malformed hook scratch directory names that could prevent hooks from running correctly. For more information, refer to PR #2042.
• Containers running under crun with user namespaces no longer encounter permission errors when resolving the rootfs path. The OCI state parser now uses the non-standard root field from crun. For more information, refer to PR #1971.

Full Changelog: v1.20.0...v1.20.1

What's Changed

NOTE: This release is a unified release of the NVIDIA Container Toolkit that consists of the following packages:

• libnvidia-container 1.20.0
• nvidia-container-toolkit 1.20.0

The packages for this release are published to the libnvidia-container package repositories.

Fixes and Features

• CDI specifications can now include an application-profile hook that limits EGL and Vulkan visibility to the GPUs assigned to the container. Graphics applications no longer see unassigned host GPUs through these APIs. For more information, refer to issue #1899 and PR #1939.
• CUDA compatibility handling now uses libcuda.so ELF metadata whenever it is available. This improves CUDA minor-version compatibility by selecting the container’s compatibility libraries only when they are appropriate for the installed driver.
• Driver file discovery now supports libraries spread across multiple directories, as occurs on distributions such as Debian, and matches graphics libraries against the exact installed driver version. This fix avoids both missing required libraries and injecting libraries from another installed driver version. For information about multiple-directory discovery, refer to issue #1559 and PR #1820. For information about exact-version matching, refer to PR #1948.
• CDI specifications now include the NVIDIA OpenCL ICD file and the legacy libnvidia-nvvm70.so library when present. OpenCL loaders can locate the NVIDIA implementation, and workloads that depend on the legacy NVVM library receive it automatically. For information about OpenCL support, refer to issue #682 and PR #1893. For information about legacy NVVM support, refer to issue #1875 and PR #1876.
• On WSL2, CDI discovery now includes additional .so, .bin, and .dll files from the NVIDIA driver store instead of relying only on a fixed file list. This fix enables containers to receive driver components introduced by newer Windows driver releases without waiting for a toolkit-specific allowlist update. For more information, refer to issue #1864 and PR #1890.
• IMEX channel requests in CDI and JIT-CDI mode are now validated for both the supported ID range and the presence of the corresponding host device. Invalid requests fail with a clear error. For more information, refer to issue #1309 and PR #1913.
• JIT-CDI mode now honors the nvidia-container-runtime.modes.jit-cdi.nvcdi-disable-hooks configuration option. You can disable individual CDI hooks for environments where a generated hook is unnecessary or incompatible.
• Updating config.toml no longer reverts previously modified options to their defaults.
• CDI generation no longer adds an update-ldcache hook when it discovers no driver libraries. This fix prevents containers from running an unnecessary hook and avoids failures on systems or modes that do not inject libraries. For more information, refer to issue #373 and PR #1894.
• Fixed an issue where NVIDIA runtime handlers in a generated containerd drop-in configuration could omit runtime_type when the base configuration did not define it. The affected containers failed to start with a container.Runtime.Name must be set error. The toolkit now sets the default runtime type when the field is missing or empty. For more information, refer to issue #1956 and PR #1969.

Packaging Changes

• RPMs rebuilt by the toolkit packaging image now use XZ payload compression instead of zstd. The resulting packages can be installed on older Linux distributions that do not support zstd compression, such as Amazon Linux 2.
• Source package builds can use Podman by setting DOCKER=podman. The build handles Podman’s local image naming, SELinux volume labeling, and artifact-directory creation automatically.

Enhancements to container-toolkit Container Image

• The container-toolkit image now uses the non-development distroless base and includes a static BusyBox shell. Init-container wrappers and lifecycle hooks retain the shell commands they need without depending on the development image.
• The NRI plugin can inject ordinary workload CDI devices outside the toolkit namespace. For management devices, you can authorize additional namespaces with the --nri-management-cdi-device-namespaces option or the NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES environment variable, enabling centralized management workloads without granting access cluster-wide.
• The toolkit installer now installs nvidia-cdi-hook instead of wrapping it with a shell script. NRI-based deployments can invoke the hook on hosts that do not provide a shell.

Full Changelog: v1.19.1...v1.20.0

What's Changed

• Validate imex channels for CDI/JIT-CDI mode
• [wsl2] discover all .so, .bin, and .dll files at nvidia driver store path
• [nri plugin] move cdi inject info log to debug level
• [CDI Hooks] add ability to specify OCI hook type
• [cudacompat] use Masterminds/semver for parsing/comparing versions
• Expand cudacompat hook to always inspect libcuda.so ELF header if available
• Add libnvidia-nvvm70.so library to CDI spec
• fix: prevent config option from reverting modified configurations
• [nri-plugin] support injection of management CDI devices in multiple namespaces
• docker.mk: Add Podman support and fix volume/artifact handling
• Only generate update-ldcache hook if libraries are discovered
• Mount nvidia.icd file needed by OpenCL
• Update SECURITY.md
• Allow multiple driver library paths
• import text/template instead of html/template
• feat: drop nvidia-cdi-hook shell shim
• Fix nvidia-cdi-refresh systemd packaging
• Add ability to disable CDI hooks in jit-cdi mode
• [nri] only restrict management CDI devices to the toolkit namespace

Changes in the Toolkit Container

• Ship static busybox shell in container-toolkit image

Changes in libnvidia-container

• refactor: Use version script instead of linker script for library symbols

NOTE: This release is a unified release of the NVIDIA Container Toolkit that consists of the following packages:

• libnvidia-container 1.19.1
• nvidia-container-toolkit 1.19.1

The packages for this release are published to the libnvidia-container package repositories.

What's Changed

• Fix the nvidia-cdi-refresh systemd unit conditions to work on WSL2.
• Remove the dependency on multi-user.target in the nvidia-cdi-refresh systemd service. For more information, see issue #1735.
• The egl-wayland2 library and configuration file are now discovered and added to CDI specifications.
• Fix handling of the CUDA compat header on Orin systems.
• Fix default path used by nvidia-ctk and nvidia-ctk-installer for cri-o’s drop-in configuration directory.
• Add support for injecting /dev/dri* device nodes for MIG devices.
• Add disable-ipc-discoverer feature flag to the nvcdi library for disabling the discovery of IPC sockets.

What's Changed

NOTE: This release is a unified release of the NVIDIA Container Toolkit that consists of the following packages:

• libnvidia-container 1.19.0
• nvidia-container-toolkit 1.19.0

The packages for this release are published to the libnvidia-container package repositories.

This is a feature release and includes the following high-level changes:

• Report errors when unknown OCI runtime spec fields are detected.
• Added support for IGX 2.0 Thor-based systems including those with dGPUs installed.
• Added support for CUDA Forward Compatibility on Tegra-based systems. On Orin-based systems, this requires specific compat libraries in the container.
• Added support for running containers as a user that may not have explicit access to a device node without requiring that additional groups be explicitly specified.
• Improve triggering of systemd service that ensures that CDI specifications are up-to-date.
• Add support for read-only root filesystems such as those on an initramfs.

Changes since v1.19.0-rc.7

• Bump version for v1.19.0 release by @elezar in #1722

v1.19.0-rc.7

• build(deps): Bump actions/download-artifact from 7 to 8 by @dependabot[bot] in #1683
• build(deps): Bump actions/upload-artifact from 6 to 7 by @dependabot[bot] in #1684
• stop marking PRs as stale by @rahulait in #1675
• fix: Ensure that CUDA Compat Container path is set by default by @elezar in #1690
• fix: Use correct version when updating versions.mk by @elezar in #1681
• chore: Run goimports on mocks by @elezar in #1692
• chore: Rename isIntegratedGPUID function to isOrinGPUID by @elezar in #1674
• Use automatic CDI spec generation to generate CDI specs for other modifiers by @elezar in #1667
• fix: Don't use driver version in ELF header for compat check by @elezar in #1697
• fix: Reuse instantiated editsFactory in CDI by @elezar in #1666
• Bump version for v1.19.0-rc.7 release by @elezar in #1698

v1.19.0-rc.6

• Use host CUDA version for all CSV compat checks by @elezar in #1679
• Bump version for v1.19.0-rc.6 release by @elezar in #1680

v1.19.0-rc.5

• Separate nvcdi construction options from runtime options by @elezar in #1659
• build(deps): Bump nvidia/distroless/go from v4.0.1-dev to v4.0.2-dev in /deployments/container by @dependabot[bot] in #1672
• chore: Fix release tooling to support annotated tags by @elezar in #1669
• allow stale cache to be deleted by @rahulait in #1670
• Remove dlopen locator by @elezar in #1676
• Bump version for v1.19.0-rc.5 release by @elezar in #1678

v1.19.0-rc.4

• build(deps): Bump golang from 1.25.6 to 1.25.7 in /deployments/devel by @dependabot[bot] in #1640
• build(deps): Bump golang.org/x/mod from 0.32.0 to 0.33.0 by @dependabot[bot] in #1642
• build(deps): Bump github.com/onsi/ginkgo/v2 from 2.27.5 to 2.28.1 in /tests by @dependabot[bot] in #1619
• build(deps): Bump golang from 1.25.7 to 1.26.0 in /deployments/devel by @dependabot[bot] in #1647
• build(deps): Bump golang.org/x/crypto from 0.47.0 to 0.48.0 in /tests by @dependabot[bot] in #1644
• build(deps): Bump golang.org/x/sys from 0.40.0 to 0.41.0 by @dependabot[bot] in #1641
• Refactor device node tests by @elezar in #1637
• Add debug-level log option for ctk-installer by @jactor-sue in #1551
• build(deps): Bump golang.org/x/mod from 0.32.0 to 0.33.0 in /tests by @dependabot[bot] in #1643
• fix: Correct container compat path for orin by @elezar in #1649
• Refactor ldcache locator by @elezar in #1648
• test: Add to package with Ptr function by @elezar in #1653
• chore: Rename management receiver from m to l by @elezar in #1654
• test: Don't use modifier to add test runtime hook by @elezar in #1656
• refactor: Move update-ldcache argument processing into hook creator by @elezar in #1651
• Add a modifier factory by @elezar in #1655
• refactor: Prefer cdilib methods over functions by @elezar in #1658
• Fix golangci-lint errors by @elezar in #1662
• fix: Fix support for nvswitch mode by @elezar in #1661
• refactor: Move NormalizeSearchPaths into lookup package by @elezar in #1660
• Allow Orin CUDA forward compat root to be specified by @elezar in #1614
• fix: Fix typo due to merge conflict by @elezar in #1664
• refactor: Add AsOptional wrapper for lookup.Locator by @elezar in #1645
• Generate CDI specification including additional GIDs by @elezar in #630
• Add dev root to driver by @elezar in #1652
• Add dlopen library locator by @elezar in #1646
• Bump version for v1.19.0-rc.4 release by @elezar in #1668

v1.19.0-rc.3

• [CI] : add github issue templates by @rahulait in #1613
• test: Adjust e2e test for Docker 29.2.0 by @elezar in #1615
• Use rpmrebuild instead of fpm to rebuild rpm packages by @elezar in #1612
• build(deps): Bump third_party/libnvidia-container from a83ddc0 to fe0d8e5 by @dependabot[bot] in #1622
• build(deps): Bump nvidia/distroless/go from v4.0.0-dev to v4.0.1-dev in /deployments/container by @dependabot[bot] in #1618
• build(deps): Bump third_party/libnvidia-container from fe0d8e5 to 7585946 by @dependabot[bot] in #1623
• build(deps): Bump github.com/onsi/gomega from 1.39.0 to 1.39.1 in /tests by @dependabot[bot] in #1620
• fix: Allow config options to be set to default values by @elezar in #1629
• Expose internal packages for external consumption by @elezar in #1577
• fix: Log actual CDI spec version by @elezar in #1633
• fix: Set device node GID in CDI specs by @elezar in #1631
• Bump version for v1.19.0-rc.3 release by @elezar in #1636
• Remove restart logic from nvidia-cdi-refresh.service by @elezar in #1638

v1.19.0-rc.2

• [no-relnote] Fix typo in release script by @elezar in #1529
• Bump github.com/onsi/gomega from 1.38.2 to 1.38.3 in /tests by @dependabot[bot] in #1533
• trim extraneous whitespaces when parsing comma-separated config sources by @tariq1890 in #1535
• Bump nvidia/distroless/go from v3.2.1-dev to v3.2.2-dev in /deployments/container by @dependabot[bot] in #1547
• fix: Return error on JIT CDI spec generation failure by @elezar in #1538
• Bump tags.cncf.io/container-device-interface from 1.0.2-0.20251114135136-1b24d969689f to 1.1.0 by @dependabot[bot] in https://gith...

What's Changed

• build(deps): Bump actions/download-artifact from 7 to 8 by @dependabot[bot] in #1683
• build(deps): Bump actions/upload-artifact from 6 to 7 by @dependabot[bot] in #1684
• stop marking PRs as stale by @rahulait in #1675
• fix: Ensure that CUDA Compat Container path is set by default by @elezar in #1690
• fix: Use correct version when updating versions.mk by @elezar in #1681
• chore: Run goimports on mocks by @elezar in #1692
• chore: Rename isIntegratedGPUID function to isOrinGPUID by @elezar in #1674
• Use automatic CDI spec generation to generate CDI specs for other modifiers by @elezar in #1667
• fix: Don't use driver version in ELF header for compat check by @elezar in #1697
• fix: Reuse instantiated editsFactory in CDI by @elezar in #1666
• Bump version for v1.19.0-rc.7 release by @elezar in #1698

Full Changelog: v1.19.0-rc.6...v1.19.0-rc.7

What's Changed

• Use host CUDA version for all CSV compat checks by @elezar in #1679
• Bump version for v1.19.0-rc.6 release by @elezar in #1680

Full Changelog: v1.19.0-rc.5...v1.19.0-rc.6

Fixes a critical error in CDI spec generation in v1.19.0-rc.4.

What's Changed

• Separate nvcdi construction options from runtime options by @elezar in #1659
• build(deps): Bump nvidia/distroless/go from v4.0.1-dev to v4.0.2-dev in /deployments/container by @dependabot[bot] in #1672
• chore: Fix release tooling to support annotated tags by @elezar in #1669
• allow stale cache to be deleted by @rahulait in #1670
• Remove dlopen locator by @elezar in #1676
• Bump version for v1.19.0-rc.5 release by @elezar in #1678

Full Changelog: v1.19.0-rc.4...v1.19.0-rc.5

What's Changed

• build(deps): Bump golang from 1.25.6 to 1.25.7 in /deployments/devel by @dependabot[bot] in #1640
• build(deps): Bump golang.org/x/mod from 0.32.0 to 0.33.0 by @dependabot[bot] in #1642
• build(deps): Bump github.com/onsi/ginkgo/v2 from 2.27.5 to 2.28.1 in /tests by @dependabot[bot] in #1619
• build(deps): Bump golang from 1.25.7 to 1.26.0 in /deployments/devel by @dependabot[bot] in #1647
• build(deps): Bump golang.org/x/crypto from 0.47.0 to 0.48.0 in /tests by @dependabot[bot] in #1644
• build(deps): Bump golang.org/x/sys from 0.40.0 to 0.41.0 by @dependabot[bot] in #1641
• Refactor device node tests by @elezar in #1637
• Add debug-level log option for ctk-installer by @jactor-sue in #1551
• build(deps): Bump golang.org/x/mod from 0.32.0 to 0.33.0 in /tests by @dependabot[bot] in #1643
• fix: Correct container compat path for orin by @elezar in #1649
• Refactor ldcache locator by @elezar in #1648
• test: Add to package with Ptr function by @elezar in #1653
• chore: Rename management receiver from m to l by @elezar in #1654
• test: Don't use modifier to add test runtime hook by @elezar in #1656
• refactor: Move update-ldcache argument processing into hook creator by @elezar in #1651
• Add a modifier factory by @elezar in #1655
• refactor: Prefer cdilib methods over functions by @elezar in #1658
• Fix golangci-lint errors by @elezar in #1662
• fix: Fix support for nvswitch mode by @elezar in #1661
• refactor: Move NormalizeSearchPaths into lookup package by @elezar in #1660
• Allow Orin CUDA forward compat root to be specified by @elezar in #1614
• fix: Fix typo due to merge conflict by @elezar in #1664
• refactor: Add AsOptional wrapper for lookup.Locator by @elezar in #1645
• Generate CDI specification including additional GIDs by @elezar in #630
• Add dev root to driver by @elezar in #1652
• Add dlopen library locator by @elezar in #1646
• Bump version for v1.19.0-rc.4 release by @elezar in #1668

New Contributors

• @jactor-sue made their first contribution in #1551

Full Changelog: v1.19.0-rc.3...v1.19.0-rc.4

What's Changed

• [CI] : add github issue templates by @rahulait in #1613
• test: Adjust e2e test for Docker 29.2.0 by @elezar in #1615
• Use rpmrebuild instead of fpm to rebuild rpm packages by @elezar in #1612
• build(deps): Bump third_party/libnvidia-container from a83ddc0 to fe0d8e5 by @dependabot[bot] in #1622
• build(deps): Bump nvidia/distroless/go from v4.0.0-dev to v4.0.1-dev in /deployments/container by @dependabot[bot] in #1618
• build(deps): Bump third_party/libnvidia-container from fe0d8e5 to 7585946 by @dependabot[bot] in #1623
• build(deps): Bump github.com/onsi/gomega from 1.39.0 to 1.39.1 in /tests by @dependabot[bot] in #1620
• fix: Allow config options to be set to default values by @elezar in #1629
• Expose internal packages for external consumption by @elezar in #1577
• fix: Log actual CDI spec version by @elezar in #1633
• fix: Set device node GID in CDI specs by @elezar in #1631
• Bump version for v1.19.0-rc.3 release by @elezar in #1636
• Remove restart logic from nvidia-cdi-refresh.service by @elezar in #1638

New Contributors

• @rahulait made their first contribution in #1613

Full Changelog: v1.19.0-rc.2...v1.19.0-rc.3