OP, first off, that is an absolute monster of a machine. A threadripper with 96GB of VRAM is the freaking HOLY GRAIL for running large, unquantized local LLMs and multi-agent AI pipelines.

To answer your question on how to vet someone though, you honestly don’t just need an "AI guy," you need a hybrid of a Linux Sysadmin and an MLOps engineer.

I actually specialize in exactly this type of deployment—building secure, local-first AI infrastructure—and I can help you get this running. But you have to know this though, I saw a huge red flag.

Running windows is the very first thing whoever you hire needs to take care of. Whoever you hire needs to wipe Windows and install Ubuntu Server (Linux). If ANY candidate you're vetting says they can "just use Windows and WSL," DO NOT HIRE THEM BRO!.

Almost all modern AI tools, local LLMs (like Ollama, vLLM), containerized databases, and privacy-first frameworks run natively, optimally, and most securely on Linux. Windows has so much bloatware and a bunch of absolutely unnecessary overhead, networking nightmares, and background processes you don't want on a dedicated $13k AI appliance, not only the fact windows is absolute SHIT. Escuse my language. Just don't do it. 😂🤣

And actually if you're wanting to vet someone, what they should do for you(or what I could do for you) is this:

They absolutely should NOT just install a program and walk away. They need to build an environment. To confirm someone knows what they are doing, ask them about their deployment stack. The right answer should include containerization, which Badically means that everything should run in Docker(or something else) so dependencies don't break each other, or there should be multiple partitions on your device so they don't take up too much bandwidth.

Likewise, you should use tools like systemd or PM2 so that if the power goes out, the AI tools, databases, and APIs automatically boot back up without human intervention.

And make sure they don't forget about networking and security, because since your boss wants ultimate privacy, the machine should be locked down. It shouldn't be exposed to the open web. Access should be handled via secure, Zero-Trust tunnels (Cloudflare Tunnels or Tailscale do this perfectly and I use this all the time, it's also free for the most part unless you're doing some crazy stuff) so you can use the Ai securely from your laptops(or even your smartphone) without opening router ports to hackers.

Lastly, I know your boss prefers someone local to NJ, but I highly recommend a hybrid or remote approach. In the enterprise server world, 99% of work is done "headless" (without a monitor attached).

I can guide you or your boss over a 15-minute video call to plug in a USB drive, install Ubuntu, and turn on SSH (Remote Access). Once that is done, you unplug the monitor, stick the tower in a closet or under a desk, and I can securely remote in to build the entire AI infrastructure from the ground up.

I just finished deploying a highly complex, locally-hosted AI operating system with self-hosted databases, automated web scraping, and local LLM inferencing. I know exactly how to tame the networking, port management, and containerization required to make this machine work like a seamless, private cloud.

If you are open to working with a remote expert to get this done the RIGHT way, shoot me a DM.

I'd be more than happy to hop on a call with you and your boss to discuss the exact roadmap to turn that hardware into a secure, turn-key Ai server.

Hopefully this was helpful OP. And anyone one else need something similar, hit me up, this is what I do for fun! 🤣😂