NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver (Linux)

Published Sources checked By helps.wiki editorial team

On Linux, this message means the nvidia-smi tool cannot reach the NVIDIA kernel module. A common reason is a kernel update after which the module was not rebuilt for the new kernel. Installing the matching kernel headers, rebuilding with DKMS and loading the module fixes many cases.

Applies to:Linux with the proprietary NVIDIA driverUbuntu, RHEL/Rocky, Arch and other distributions
On this page

What it means

"NVIDIA driver error" covers several different problems. This article is about the Linux one: you run nvidia-smi and it prints NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. For the Windows Device Manager failure, see NVIDIA GPU error Code 43.

The command-line tools are only the front end. They talk to a kernel module, and the message means that conversation is not happening. The module may not be loaded, it may have been built for a different kernel than the one now running, or the kernel may have refused it. In rarer cases the card itself has dropped off the PCIe bus.

The first step below only reads information, and it tells you which cause applies. Package names differ between distributions, so check your distro's own NVIDIA driver documentation for exact names.

Symptoms and causes

You are likely dealing with this problem if:

  • nvidia-smi prints the "couldn't communicate with the NVIDIA driver" message.
  • The screen goes black, or the desktop falls back to a low resolution.
  • lsmod | grep nvidia prints nothing.

The usual causes, most common first:

Kernel updated, module not rebuilt
DKMS did not build the module for the new kernel, or the headers for the running kernel are missing.
Secure Boot blocks the module
With Secure Boot on, the kernel refuses an unsigned NVIDIA module.
Wrong or incomplete driver package
A headless or server package without a kernel module, or the open kernel module (nvidia-open) on a GPU generation it does not support. The Netdata guide says that means Maxwell, Pascal and Volta cards.
nouveau conflict
The open-source nouveau driver is in the way, which NVIDIA's README lists as a problem when installing.
Driver crashed or GPU hung
The module loaded earlier but is now stuck or unusable.
GPU fell off the bus (Xid 79)
A hardware event: the card is no longer reachable over PCIe.
Optimus laptop with the discrete GPU powered off
The dGPU is switched off, so there is nothing for the driver to talk to.

How to fix it

Work through these in order. They start with the quickest, safest checks and move to deeper repairs; stop as soon as the problem is gone.

1.Read the current state

Easy

These commands change nothing and show which cause you are dealing with.

  1. Open a terminal and run the checks.
    Terminal
    nvidia-smi; echo "exit=$?"
lsmod | grep nvidia
lsmod | grep nouveau
uname -r; dkms status
dmesg -T | grep -i nvrm
dmesg -T | grep -i "NVRM: Xid"
lspci -d 10de:
mokutil --sb-state
  2. Read the output and pick the matching route. dkms status lists an nvidia entry, but not for the kernel that uname -r shows: go to Install matching headers and rebuild the module. dkms status shows no nvidia entry at all: go to Install the right driver package. lsmod | grep nouveau prints something: nouveau is loaded, so go to Install the right driver package as well. Secure Boot enabled plus NVRM messages about the module being rejected: go to Deal with Secure Boot. An Xid 79 line: go to Treat Xid 79 as a hardware fault.
  3. If lspci -d 10de: shows no NVIDIA device at all, you are on a laptop with switchable (Optimus) graphics, and the discrete GPU may simply be powered off. Check your laptop's graphics-switching settings before treating it as a hardware fault. If it is a desktop, see Treat Xid 79 as a hardware fault.

2.Install matching headers and rebuild the module

Easy

After a kernel update the module must be built again for the new kernel. NVIDIA's README confirms that matching kernel headers are required and that DKMS handles minor kernel updates.

  1. On Debian or Ubuntu, install the headers for the running kernel, rebuild, and load the module.
    Terminalneeds sudo / root
    sudo apt install linux-headers-$(uname -r)
sudo dkms autoinstall
sudo modprobe nvidia
  2. On RHEL or Fedora, install kernel-devel for the running kernel instead and check the akmods output for build errors.
  3. Run nvidia-smi again. If the build fails, boot the previous kernel from the boot menu as a fallback. After a major kernel jump, a newer driver version may be required.

3.Deal with Secure Boot

AdvancedChanges settings

An unsigned module is refused when Secure Boot is on. The answer is to sign the module, not to turn protection off.

Before you start: Keep the private signing key somewhere safe and private. Do not switch Secure Boot off as a shortcut.

  1. Confirm the state with mokutil --sb-state. If it says enabled and dmesg shows the module being rejected, the module needs a signature your system trusts.
    Terminal
    mokutil --sb-state
  2. Either enrol a Machine Owner Key (MOK) at the next boot, following your distro's documentation, or sign the module yourself. NVIDIA's .run installer accepts --module-signing-secret-key and --module-signing-public-key for this.
  3. Reboot, complete any MOK enrolment prompt, then run nvidia-smi.

4.Install the right driver package

ModerateChanges settings

No DKMS entry, a headless or metapackage without a kernel module, the open variant on an unsupported GPU, or nouveau holding the card all leave nothing for nvidia-smi to talk to.

Before you start: Note which driver branch you currently have installed before swapping packages, and have a way to reach a console in case the display fails.

  1. Install the DKMS package for your driver branch, then build and load it. The command below is for Ubuntu: replace <branch> with the driver series you want (the number in the package name), and use your distro's package name if it is not Ubuntu.
    Terminalneeds sudo / root
    sudo apt install nvidia-dkms-<branch>
sudo dkms autoinstall
sudo modprobe nvidia
  2. If you have an -open package, check your GPU generation. The open kernel module needs a newer card; the Netdata guide says Maxwell, Pascal and Volta GPUs are not supported. On one of those, replace it with the proprietary package of the same branch and reboot.
  3. If lsmod | grep nouveau printed something, nouveau is loaded and must be disabled before you install with the .run file. NVIDIA's Linux README, chapter 4 lists nouveau as a known installation problem; follow the nouveau instructions in that README or your distro's documentation, then check again that lsmod | grep nouveau is empty after a reboot.

5.Reload a stuck module

ModerateChanges settings

If the module is present but the driver stopped responding, reloading it can clear the fault.

Before you start: Unloading the module ends any GPU work in progress and may blank the display. Save your work first.

  1. See what is using the GPU and close those programs.
    Terminalneeds sudo / root
    fuser -v /dev/nvidia*
  2. Unload and load the module again.
    Terminalneeds sudo / root
    sudo modprobe -r nvidia && sudo modprobe nvidia
  3. If the module will not unload, reboot.

6.Treat Xid 79 as a hardware fault

Moderate

Xid 79 means the GPU fell off the bus, which points at the card, its seating or its power rather than at software.

  1. Reboot and check lspci -d 10de: to see whether the card appears again.
  2. If it keeps disappearing, power off, reseat the card and its power cables, and try again. If it still fails, replace the card or start a warranty claim, as the Netdata guide suggests.

Check that it worked

  • nvidia-smi lists your GPU and driver version instead of the error.
  • lsmod | grep nvidia shows the nvidia modules.
  • The result stays the same after a reboot.

If it still fails

  • Save the output of dmesg -T | grep -i nvrm, dkms status and uname -r. These show whether the problem is the build, the signature or the hardware.
  • On an Optimus laptop, check that the discrete GPU is powered on and visible in lspci -d 10de:.
  • If the card repeatedly drops off the bus, contact the card or computer manufacturer about a repair or replacement.

Questions

What if the message is "Failed to initialize NVML: Driver/library version mismatch"?

That is a different error: the loaded kernel module and the userspace libraries are different versions.

Is this the same as Code 43 on Windows?

No. Code 43 is a Windows Device Manager status. See NVIDIA GPU error Code 43.

Sources

  1. NVIDIA-SMI "couldn't communicate with the NVIDIA driver" guide · Netdata
  2. Installing the NVIDIA Driver (Linux README, chapter 4)Official · NVIDIA

Browse