On-Prem Deployment
Last updated
Last updated
| Item | Description |
|---|---|
Hardware
Minimum Requirements:
CPU - 8 Cores
RAM - 16 GB
Disk - 100 GB
NIC - Management
Example: DELL R740, DELL C4130 etc..
CPU Specification:
Essential: CPU with AVX512 instruction set.
Recommended: Modern multi-core CPU with 3.0 GHz+ clock speed.
RAM: 32GB+ (Our software package includes additional services)
For higher capacity, please consider the following specifications:
128 devices: 256 GB disk
256 devices: 1 TB disk
Compatible GPUs
Essential: NVIDIA-based GPU with compute capability 7.0 or higher.
VRAM:
Minimum VRAM: 16GB+
Mistral model loaded in FP16 (Good balance of performance and accuracy)
Recommended VRAM: 24GB+
Mistral loaded in TF32 (Ampere and newer) or FP32 (Faster inference, minimal accuracy impact)
Supported GPUs
Data Center:
NVIDIA H100
NVIDIA L4, L40
NVIDIA A100, A40, A30, A10, A16, A2
NVIDIA T4
NVIDIA V100
Desktop:
RTX 6000, A6000, A5000, A4000
GeForce:
RTX 4090, 4080, 4070 Ti, 4060 Ti
RTX 3090 Ti, 3090, 3080 Ti, 3080
RTX 3070 Ti, 3070, 3060 Ti, 3060
Environment
A 64-bit Ubuntu 22.04 server with sudo privileges
License
AVIZ Network Copilot Base License