JoeCode

My local model setup

Meet Thales, my local inference model setup powered by 2 surplus Dell EMC R650 1U servers.

The Thales project is a private AI inference cluster built on two Dell PowerEdge R650 servers, each equipped with an NVIDIA T4 GPU. The goal is to provide low‑latency, high‑throughput inference endpoints for a variety of models—fast, deep, and embedding—behind a LiteLLM router. This post summarizes the current status of the build, the steps completed, and the next milestones.

Thales?

Thales is one of the artificial intelligences in Arthur C. Clarke and Stephen Baxter’s A Time Odyssey series. In the story’s near-future setting (2037), Thales is a quietly capable, trustworthy machine intelligence working alongside humans serving humanity’s lunar operations.

After you read Sun Storm, you’ll understand the magnetosphere illustration

Overview

Two Dell R650 servers, each with an NVIDIA T4 GPU, form the core of the Thales cluster. A dedicated 25 GbE link (10.10.10.1/30 ↔ 10.10.10.2/30) connects the machines, while Tailscale provides secure remote access. The stack runs Ubuntu 24.04 LTS, with iDRAC firmware, BIOS, and storage (RAID1 boot, RAID5 bulk) fully updated.

Inference Stack

• llama.cpp built from source (CPU‑only for now). The gpt‑oss‑20b model is on inf1; the Qwen3‑Coder model will live on inf2.
• llama‑server units bind only to LAN and tailnet interfaces.
• LiteLLM router on inf1 exposes three endpoints: fast (gpt‑oss), deep (Qwen3‑Coder), and embed (to be added). Master key is set; database and per‑device keys are pending.

Current Status

All hardware is online, networking is stable, and the CPU‑only inference stack is functional. GPU acceleration will be added once NVIDIA drivers and CUDA are installed. Security hardening is complete; only authorized devices can reach the services.

Next Steps

1. Order and install Nvidia T4 GPU cards.
2. Install NVIDIA drivers and CUDA.
3. Enable GPU inference for both models.

The Thales cluster is now a fully functional, secure inference platform ready for GPU acceleration. With the foundation laid, the next phase will focus on performance tuning, scaling, and integrating the system into our development workflows. Stay tuned for updates as we move from CPU validation to GPU‑powered inference.

See the detailed Dell R650 hardware overview here: Dell R650 Hardware Overview

Thales (GPT-OSS-20B) inference AI provided assistance drafting this post