Thales - Dual Dell R650 AI Inference Cluster

Build Plan

Two Dell PowerEdge R650s (2x Xeon Silver 4309Y, 128GB DDR4-3200, PERC H755, dual 25GbE) with one NVIDIA T4 each. Goal: private AI coding endpoints (fast / deep / embed) behind a LiteLLM router, reachable from anywhere via Tailscale.

Naming used throughout: inf1 (fast lane + router), inf2 (deep lane + monitoring).

Phase 0 — Procurement

• Order 2x NVIDIA T4 (prefer Dell OEM / PowerEdge pulls for clean iDRAC thermal profiles)
• 1x SFP28 DAC cable (1-3m) for direct server-to-server link
• Confirm rack location handles noise + heat (garage / basement / vented closet)
• Optional: 2x NVMe SSD for model storage if pulling the SAS drives

Phase 1 — Physical & Firmware (do now, no GPUs needed)

iDRAC

• Reset iDRAC Dell Support
• Assign static LAN IPs to both iDRACs (LAN-only — never on Tailscale, never port-forwarded)
• Change default iDRAC credentials
• Update firmware on both boxes via iDRAC: BIOS, iDRAC, PERC H755, NICs, backplane
• Verify iDRAC Enterprise license active (virtual console + virtual media)

BIOS

• System Profile → Performance
• Verify riser config; confirm one open x16 slot per box for the T4
• Verify both PSUs healthy

Storage (PERC H755)

• RAID1 the 2x 480GB SATA SSDs → boot volume
• Decide SAS drives: pull them (less power/noise) or RAID5 for bulk model storage
• If pulled: note caddies + drives for resale or NAS reuse

Phase 2 — OS & Networking

OS install (both boxes)

• Ubuntu Server 24.04 LTS, minimal install, via iDRAC virtual media
• Hostnames: inf1, inf2
• OpenSSH enabled, key-based auth only, password auth off
• apt update && apt full-upgrade, enable unattended-upgrades for security patches

LAN

• Static DHCP reservations for both boxes’ LAN ports
• Verify DNS + NTP

25GbE direct link

• DAC cable between the PCIe E810 NICs (not the OCP ports)
• Point-to-point subnet: inf1 = 10.10.10.1/30, inf2 = 10.10.10.2/30 (netplan)
• iperf3 between boxes — expect ~23-25 Gbit/s
• Add /etc/hosts entries so each box resolves the other over the fast link

Tailscale

• Install on both servers + laptop + phone; enable tailscaled as a service
• MagicDNS on
• ACLs: client devices → inf1:4000 (LiteLLM) and SSH only; deny everything else
• Confirm SSH to both boxes from off-network (phone hotspot test)
• Confirm iDRAC is NOT reachable via tailnet

Phase 3 — Inference Stack, CPU-Only (validate before GPUs arrive)

llama.cpp

• Build from source on both boxes (no CUDA flags yet)
• Download models (big files — start early):
  • gpt-oss-20b (MXFP4) → inf1
  • Qwen3-Coder-30B-A3B Q4_K_M → inf2
  • Embedding model (e.g. nomic-embed-code) → inf1
• llama-server systemd units on both boxes:
  • Bind to LAN + tailnet interfaces only — never 0.0.0.0
  • Restart=on-failure
  • inf1: gpt-oss-20b, parallel slots (start --parallel 4)
  • inf2: Qwen3-Coder-30B, 1-2 slots, large context
• Smoke test: curl a completion from each server

LiteLLM router (inf1)

• Install LiteLLM proxy, systemd unit
• Model map: fast → inf1 gpt-oss, deep → inf2 Qwen3-Coder, embed → inf1 embeddings
• Master key set; issue per-device virtual keys
• Test all three routes through the proxy from the workstation over Tailscale

Client tooling

• Point aider / Continue.dev at the LiteLLM endpoint
• Configure architect/editor split (deep plans, fast edits)
• Daily-drive for a few days on CPU (expect ~5-10 tok/s) — shake out auth, routing, context settings

Monitoring

• Uptime Kuma (Docker) on inf2 → monitor both llama-server endpoints + LiteLLM; phone notifications
• Optional: Netdata on both boxes
• Verify everything survives a full reboot of both boxes (systemd + Tailscale + static IPs)

Phase 4 — GPU Day (T4s arrive)

Install

• Power down, install one T4 per box (x16 slot), power up
• iDRAC recognizes the card; if generic card and fans ramp: racadm set system.thermalsettings.ThirdPartyPCIFanResponse 0
• NVIDIA driver (server/headless variant via ubuntu-drivers or CUDA repo)
• nvidia-smi shows T4, 16GB, ~70W cap

Rebuild & retune

• Rebuild llama.cpp with -DGGML_CUDA=ON on both boxes
• inf1: gpt-oss-20b fully on GPU (-ngl 999), KV cache q8_0, retune slot count
• inf2: Qwen3-Coder-30B hybrid — attention + KV on GPU, experts in RAM (--n-cpu-moe), KV q8_0
• Benchmark vs CPU baselines (prefill + generation tok/s); record numbers
• Watch T4 thermals under sustained load (nvidia-smi dmon) — passive card, needs chassis airflow

Experiment (optional, one evening)

• llama.cpp RPC mode: pool both T4s over the 25GbE link, run the 30B fully on GPU
• Benchmark vs single-box hybrid; keep whichever wins

Ongoing

• Monthly: apt updates, firmware check, review Tailscale ACLs + devices
• Model refresh cadence: check for new coding-model releases quarterly
• Backups: LiteLLM config, systemd units, netplan — commit to a private git repo
• Power draw check after a full month (validate the electricity cost estimate)

Key decisions locked in