Install CUDA Toolkit for your AI

First thing is to install NVIDIA CUDA Toolkit. but first lets just make sure the packages are up to date.

sudo apt update && sudo apt upgrade

If there were any updates it's a good idea to reboot to make sure there are no errors from updating packages that appear on startup. If you havent before check your startup messages.

sudo dmesg

This will show the startup output and will show drivers being loaded and various minor to severe errors in the computers setup. Next we will install gcc.

sudo apt install gcc

Now we will download our toolkit files, go to https://developer.nvidia.com/cuda-downloads and select your package. For me, I am using Debian 12 so my output would be

wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-9

Now we will install the NVIDIA driver. there are two options, one for the open source version and one the proprietary driver. I tend to use the proprietary

sudo apt-get install -y cuda-drivers

Now we will add the kernel headers, this will activate the driver

sudo apt install linux-headers-$(uname -r)

Now we will add the path to the CUDA Toolkit. This can be done on a per user basis by adding it to your home .bashrc file but I prefer to make it for all profiles so system users have the path. In /etc/profile add a new line at the end

export PATH=${PATH}:/usr/local/cuda-12.9/bin

Now reboot, and we can test our cuda installation

nvcc --version

For my installation version this returns

Copyright (c) 2005-2025 NVIDIA Corporation
Built on Tue_May_27_02:21:03_PDT_2025
Cuda compilation tools, release 12.9, V12.9.86
Build cuda_12.9.r12.9/compiler.36037853_0

Now we verify our NVIDIA GPU

nvidia-smi

Which returns an output of your NVIDIA GPU's available.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.57.08              Driver Version: 575.57.08      CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        On  |   00000000:01:00.0  On |                  N/A |
|  0%   32C    P8             12W /  420W |       9MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

One thing to note for other troubleshooting problems is this tool nvidia-smi will show the load on the card but more importantly the idle voltage and heat when under no load. This can give you information or a warning of problems with the card or power supply as its well known modern video cards can possibly melt the power connectors if they are slighlty bent or weak and cause strange voltage draws and rising heat under no load because of this causing your card to fail.