On-Prem Deployment

GPU:

GPU	GPU Count (Recommended)	GPU Count (Future Ready)	Total GPU vRAM (Recomended)	Total GPU vRAM (Future Ready)	PCIe
H100 (96 GB)	2	4	192GB	384GB	4
H200 (96 GB)	2	4	192GB	384GB	4
RTX Pro 6000 (96 GB)	2	4	192GB	384GB	5
A100 (80 GB)	2	4	160GB	320GB	4
A6000 (48 GB)	4	8	192GB	384GB	4
L40S (48 GB)	4	8	192GB	384GB	4

Server:

Component	Specification
CPU Cores	Minimum: 16 Cores Recommended: 32 Cores
RAM	Minimum: 64 GB Recommended: 128 GB
Disk	2 TB

Software:

Component	Specification
Environment	A 64-bit Ubuntu Server version 22.04 or later Docker & Docker-Compose NVIDIA-based GPU with CUDA 12 NVIDIA Container Toolkit
License	NCP-Basic NCP-Advanced NCP-Performance
Model	LLAMA 3.3 70B Instruct (Default) GPT-OSS-120B