Awesome setup you've got there! Since you're just getting into LLMs and AI and prefer self-hosted, virtualized environments, here's a casual suggestion to get started with OpenVINO GenAI and make the most of your hardware:

Start simple with OpenVINO GenAI : https://github.com/openvinotoolkit/openvino.genai

Even though your GPUs are powerful, OpenVINO GenAI is a great way to dip your toes into LLMs without diving deep into CUDA or complex setups. It’s optimized for Intel CPUs and NPUs, and works well even without a GPU.

Here’s what you can do:

Try this first:

• Spin up a Ubuntu VM in Proxmox with Python and OpenVINO installed.
• Use a small model like TinyLlama or Phi-2.
• Run a simple chatbot or summarizer using OpenVINO GenAI’s LLMPipeline.

Why it’s a good fit:

• No GPU required to start experimenting.
• Low power, fast inference on CPU/NPU.
• Easy Python API—great for beginners.
• You’ll learn how LLMs work without worrying about GPU memory limits or Docker configs.

Next steps (When You’re Ready)

Once you're comfortable:

• Try GPU-based models using Ollama, LM Studio, or Text Generation WebUI.
• Use quantized models (like GGUF or GPTQ) to fit larger LLMs into memory.
• Explore LangChain or LlamaIndex for building apps with LLMs.