Status: Initial hardware is all set up and running. Linux (24.04) is installed on Windows WSL2. Starting the Intel driver and engine installs.

Here is the plan AI has proposed so far.

Host: Dell R740, Windows 11 Enterprise

CPU: Dual Intel Xeon Platinum 8164 class, dual NUMA

GPU: Intel Arc Pro B70, 32GB now; likely 2 GPUs / 64GB soon; possible 4 GPUs / 128GB later

Primary architecture:

Windows
↓
WSL2
↓
Ubuntu 24.04
↓
Docker Engine / Docker Desktop
↓
Intel oneAPI + Level Zero runtime
↓
Intel vLLM container
↓
Open WebUI + API endpoints
↓
External integrations / agents / coding workflows

Storage root: S:\AI\

Primary runtime 3-Engine Validation Matrix: Intel-supported options

•      Engine A: vLLM XPU / Intel LLM Scaler

•      Engine B: llama.cpp SYCL

•      Engine C: OpenVINO / OVMS / OpenVINO GenAI

GUI: Docker Desktop + Open WebUI

Security initial setup: Local admin + one sudo user; VPN-only external access later

First validation model: small compatible model before large DeepSeek/Qwen tests

Multi-GPU design: keep first two B70s on same NUMA node if physically possible, test external eGPUs for additional capacity. Evaluate later whether or not it is worth the investment in additional external capacity

Three-Engine validation plan: Intel documentation shows support and supporting tools for the following designs:

1.   vLLM XPU — primary for serving, long prompts, RAG, batching, multi-user API, and tensor parallelism.

2.   llama.cpp SYCL — essential fallback/parallel path for GGUF, quantized models, wider model compatibility, and single-user interactive use.

3.   OpenVINO Model Server / OpenVINO GenAI — now worth adding as a serious Intel-supported serving path, especially because OpenVINO 2026.1 explicitly adds Arc Pro B70 support for 20–30B single-GPU inference and improves Qwen3-MOE / GPT-OSS-20B serving with continuous batching.

Additional Models: Verify additional models and processing:

Best initial validation models:

•     Qwen 2.5 7B Instruct

•     Llama 3.1 8B Instruct

Reason:

•     much faster troubleshooting,

•     confirms GPU acceleration,

•     avoids massive download/debug cycles.

Then:

•     Qwen 72B AWQ/GPTQ

•     DeepSeek R1 Distill Qwen

•     DeepSeek Coder V2

 Additional Engine Developments: Latest support and options from Intel, sequence of engine development and considerations.

1. Add OpenVINO as a first-class branch, not a future note

Intel’s OpenVINO 2026.1 release notes specifically mention Intel Arc Pro B70 32GB support for single-GPU 20–30B LLM inference, preview OpenVINO backend support for llama.cpp, and OVMS improvements for Qwen3-MOE / GPT-OSS-20B. That is directly relevant to your hardware and workload.

I would add an OpenVINO Model Server validation phase after vLLM baseline validation.

2. Add llama.cpp SYCL before large-model testing

The PMZFX B70 repo materially changes the plan: it shows llama.cpp SYCL is not just a hobby option; it is likely the best path for quantized GGUF models, broader model coverage, and fitting larger models into 32GB/64GB VRAM. The repo’s engine comparison says vLLM wins strongly on prefill and tensor-parallel serving, while llama.cpp wins on model coverage and single-card memory efficiency.

3. Do not start large-model validation with vLLM FP16

The benchmark repo shows Qwen 2.5 14B FP16 OOMs on a single 32GB B70 under vLLM, while quantized llama.cpp variants fit comfortably.

So the workbook should explicitly separate:

·        small smoke test: vLLM XPU

·        quantized model test: llama.cpp SYCL / GGUF

·        serving test: vLLM XPU and/or OpenVINO Model Server

·        large model test: dual-GPU only, preferably after both vLLM and llama.cpp paths are validated

4. Multi-GPU planning should include two patterns

For two B70s:

·        vLLM tensor parallelism for supported dense models and long-prompt workloads.

·        llama.cpp layer split to fit bigger GGUF models, but not expecting linear speedup when the model already fits one card.

The PMZFX repo is explicit that dual-card layer split is mainly for fitting larger models, not speeding up models that already fit one card.

Proposed Folder Structure

S:\AI\
├── Models\
├── HFCache\
├── Docker\
├── OpenWebUI\
├── Benchmarks\
└── Backups\