NVIDIA

Models and offerings #

Who they are

This hub is for anyone whose code touches a GPU, which is nearly everyone on this site, and for people deciding whether to buy, rent or avoid NVIDIA hardware. The company was founded in 1993 to make graphics chips, released CUDA in 2007 so that those chips could run general computation, and became the default platform for deep learning when AlexNet was trained on two GTX 580s in 2012. Every major framework targets CUDA first and other backends second, and the hardware and acceleration topic explains why the memory bandwidth and tensor cores matter more than the clock speed.

Since 2023 NVIDIA has pushed up the stack: inference compilers, packaged microservices, training frameworks and its own open-weight models. It is a software vendor with a hardware moat, and the moat is CUDA.

Tooling and APIs

Most people never install CUDA directly. The PyTorch and JAX wheels bundle the CUDA runtime, and only the driver needs to be on the machine. CUDA Toolkit 13.5 is current at the time of writing (the release notes page gives no date, and 13.4 was September 2026). The 13.x series removed offline compilation for Maxwell, Pascal and Volta, so GTX 10-series and V100 cards need a CUDA 12.x build, and 13.5 deprecates Debian 12. Linux is the primary platform. Windows is supported natively and through WSL2, including Windows on Arm for DGX Station. There is no CUDA on macOS.

The inference layer: TensorRT compiles a model to an engine for one GPU type, and TensorRT-LLM (Apache-2.0, 1.4 release candidates at the time of writing, built against CUDA 13.4 and PyTorch 2.14) adds the LLM-specific kernels, paged attention and quantisation. NVIDIA also contributes those kernels into vLLM and SGLang, so you can get much of the benefit without adopting the product. Triton Inference Server (BSD-3) serves engines over HTTP and gRPC. NIM wraps a model, TensorRT-LLM or vLLM and an OpenAI-compatible API in one container:

# needs-api
import os
from openai import OpenAI

client = OpenAI(api_key=os.environ.get("NIM_API_KEY", "none"), base_url="http://127.0.0.1:8000/v1")
chat = client.chat.completions.create(
    model=os.environ["NIM_MODEL"],
    messages=[{"role": "user", "content": "In one sentence, what does a learning rate control?"}],
)
print(chat.choices[0].message.content)

NIM containers are licensed as a runtime under NVIDIA AI Enterprise, which is priced per GPU per year, and the models inside carry their own licences. We could not load the AI Enterprise pricing page at the time of writing, so see NVIDIA’s product page. Hosted endpoints on build.nvidia.com offer a trial quota.

Models: the Nemotron line is NVIDIA’s open-weight family. Nemotron 3.5 Lightning (30 B total, 3 B active, hybrid Mamba-2 and MoE with some attention layers, 1 M context, OpenMDW-1.1, 11 August 2026) is the one that fits a single GPU. Nemotron 3 Ultra is 550 B total and 55 B active with 1 M context, and Nemotron 3.5 Super VL (120 B total, 12 B active) is in early access. The model card for Lightning says layer counts, expert counts and the data mixture are undisclosed. OpenMDW-1.1 is a permissive model licence that the card describes as allowing commercial use, which makes these open weights with a permissive licence rather than restricted ones.

Pain points: driver and toolkit version matrices, engines that must be rebuilt per GPU and per TensorRT version, and container images that run to tens of gigabytes.

Migration notes

Away from NVIDIA: AMD ROCm is the nearest equivalent and PyTorch 2.15 ships ROCm 10.0 wheels, so training code usually runs with a different wheel, but custom CUDA kernels, Triton kernels tuned for one architecture and anything using cuDNN-specific features need work. Apple MLX and PyTorch MPS cover Macs with no CUDA at all. Google TPUs need JAX or PyTorch/XLA. For inference, vLLM and llama.cpp run on ROCm, Metal, Vulkan and CPU, which is a cheaper path off NVIDIA than porting a TensorRT pipeline.

Between NVIDIA generations: engines built with TensorRT do not carry over between architectures (Hopper to Blackwell) and must be recompiled. NVFP4 is Blackwell-only and FP8 needs Hopper or later. The quantisation formats reference lists which format needs which hardware.

Model formats: NIM and TensorRT-LLM read Hugging Face checkpoints and build engines on first run. Nemotron checkpoints are standard safetensors and load in Transformers and vLLM, and the Mamba layers need recent versions of both.

Watch for

CUDA is the lock-in. The EULA permits free use and redistribution of the runtime but not running CUDA through translation layers on other vendors’ hardware, and the lead NVIDIA holds in kernel maturity is what you pay for. Pricing of the hardware itself is set by supply and by cloud providers rather than by a list price you can rely on.

The NIM licence model is a trap for the unwary: the container is free to try through the developer programme but production use requires an AI Enterprise subscription, and the model inside may have a licence that forbids the use you intend. Check both. Nemotron, Llama Nemotron and the NVIDIA Open Model License variants differ in their terms.

Old GPUs stop getting toolkit support with each major CUDA release, and consumer cards are not licensed for data-centre use under the driver EULA. NVIDIA’s own benchmarks (Lightning is “the fastest in its 30B-A3B class” according to the catalogue page) are the vendor’s. Containers from NGC and the NVIDIA Python packages phone home for update checks unless you disable it, which matters in air-gapped sites.