JoeCode

Project Thales - Dual Dell R650 AI Inference Cluster

Thales?

Thales is one of the artificial intelligences in Arthur C. Clarke and Stephen Baxter’s A Time Odyssey series. In the story’s near-future setting (2037), Thales is a quietly capable, trustworthy machine intelligence working alongside humans serving humanity’s lunar operations.

Build Plan

Two Dell PowerEdge R650s (2x Xeon Silver 4309Y, 128GB DDR4-3200, PERC H755, dual 25GbE) with one NVIDIA T4 each. Goal: private AI coding endpoints (fast / deep / embed) behind a LiteLLM router, reachable from anywhere via Tailscale.

Naming used throughout: inf1 (fast lane + router), inf2 (deep lane + monitoring).

Phase 0 — Procurement

Phase 1 — Physical & Firmware (do now, no GPUs needed)

iDRAC

BIOS

Storage (PERC H755)

Phase 2 — OS & Networking

OS install (both boxes)

LAN

Tailscale

Phase 3 — Inference Stack, CPU-Only (validate before GPUs arrive)

llama.cpp

LiteLLM router (inf1)

Client tooling

Monitoring

Phase 4 — GPU Day (T4s arrive)

Install

Rebuild & retune

Experiment (optional, one evening)

VS Code Copilot Chat custom model

VS Code 1.135 and GitHub Copilot Chat 0.63 use the built-in Custom Endpoint provider with the Chat Completions API. The old OpenAI Compatible provider and github.copilot.chat.customOAIModels setting are deprecated.

LiteLLM model map

The LiteLLM proxy on inference1:4000 exposes the fast and deep aliases:

model_list:
  - model_name: fast
    litellm_params:
      model: openai/<exact-model-id-served-by-llama.cpp-on-inf1>
      api_base: http://127.0.0.1:8080/v1
      api_key: not-needed

  - model_name: deep
    litellm_params:
      model: openai/<exact-model-id-served-by-llama.cpp-on-inf2>
      api_base: http://inference2:8082/v1
      api_key: not-needed

litellm_settings:
  drop_params: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

The upstream api_base ends in /v1; LiteLLM adds /chat/completions. The openai/ prefix marks the upstream as OpenAI-compatible and is not part of the client-facing alias.

llama.cpp context

Copilot Agent requires a 32,768-token context per slot for its system prompt, tools, workspace context, and response. llama.cpp divides --ctx-size across --parallel slots.

Fast on inference1:

--parallel 1
--ctx-size 32768

Deep on inference2:

--parallel 2
--ctx-size 65536

Verify each server reports at least 32,768 tokens per slot:

curl --silent http://127.0.0.1:8080/props | jq '{total_slots, n_ctx: .default_generation_settings.n_ctx}'
curl --silent http://127.0.0.1:8082/props | jq '{total_slots, n_ctx: .default_generation_settings.n_ctx}'

VS Code configuration

  1. Run Chat: Manage Language Models.
  2. Select Add Models, then Custom Endpoint.
  3. Enter Thales for the group and display name.
  4. Enter the LiteLLM master key. Replace it with a restricted virtual key after adding the LiteLLM database.
  5. Select Chat Completions.
  6. Preserve the generated apiKey secret reference and configure both models in chatLanguageModels.json:
{
  "name": "Thales",
  "vendor": "customendpoint",
  "apiKey": "${input:chat.lm.secret.thales}",
  "apiType": "chat-completions",
  "models": [
    {
      "id": "fast",
      "name": "Thales - Fast (gpt-oss-20b)",
      "url": "http://inference1.local:4000/v1/chat/completions",
      "toolCalling": true,
      "vision": false,
      "streaming": true,
      "requestHeaders": {
        "Authorization": "Bearer ${apiKey}"
      },
      "maxInputTokens": 30720,
      "maxOutputTokens": 2048
    },
    {
      "id": "deep",
      "name": "Thales - Deep (Qwen3-Coder-30B)",
      "url": "http://inference1.local:4000/v1/chat/completions",
      "toolCalling": true,
      "vision": false,
      "streaming": true,
      "requestHeaders": {
        "Authorization": "Bearer ${apiKey}"
      },
      "maxInputTokens": 30720,
      "maxOutputTokens": 2048
    }
  ]
}

The input and output limits total 32,768 tokens, matching each server slot. The explicit header sends the VS Code-managed secret as a bearer token. Keep toolCalling enabled so both models remain available in Agent mode.

Run Developer: Reload Window, start a new chat, and verify both models with a short prompt and a read-only workspace request.

References: VS Code language models and Custom Endpoint, LiteLLM proxy quick start, and LiteLLM OpenAI-compatible endpoints.

Ongoing

Key decisions locked in

DecisionChoiceRationale
OSUbuntu Server 24.04 LTSBest NVIDIA/llama.cpp support, matches WSL workflow
Boot storageRAID1 2x480GB SSDRedundancy, PERC-native
Role splitinf1 = fast, inf2 = deepSpecialization > load balancing for one user
RouterLiteLLM on inf1One OpenAI-compatible endpoint, per-key auth
Remote accessTailscaleNo open ports, ACLs, 15-minute setup
InterconnectDirect 25GbE DACNo switch needed, enables RPC experiment
iDRAC exposureLAN onlyBMCs never touch the internet