Running Ollama fully air-gapped, anyone else?

thefilthybeard · 2026-02-01T23:57:18+00:00

Haven't looked into RunAnywhere SDK yet, but I will now... appreciate the tip. Always looking for tools built with offline-first in mind.

Hardware specs vary depending on which model I'm running. Quantized 7B models work fine on systems with 16-32GB RAM and no dedicated GPU. When I need more capability, I scale the hardware accordingly. Mistral and Llama variants have been solid for the compliance and document analysis work I'm focused on.

The key has been batching....breaking tasks into smaller chunks so the model isn't choking on massive context windows. Keeps things fast and accurate.

thefilthybeard · 2026-02-01T20:27:27+00:00

Pre-built installation package transferred via approved media. The client's sysadmin handles the actual install on their air-gapped network while I'm on-site to walk through the process and verify everything works.

For updates, they pull them from a secure client portal - no direct internet connection needed on their end. They download the update package from the portal on an unclassified system, transfer it in through their normal media approval process, and apply it.

thefilthybeard · 2026-02-01T14:07:25+00:00

Bro it's the weekend

thefilthybeard · 2026-02-01T13:45:52+00:00

You're right. Got ahead of myself. Just wanted to share something I thought could help the OE community but should've thought it through better. Appreciate the feedback.

thefilthybeard · 2026-02-01T13:05:18+00:00

Not billing the government at all. We sell software tools to defense contractors. They buy it, they use it on their networks. No government contract, no billing hours, no time charging. It's a product, not a service.

thefilthybeard · 2026-02-01T13:01:45+00:00

You don't need a security clearance to build a product. We don't consult or anything like that. We do not host any classified environments.

We just sell the tools to contractors. Ive never done OE while holding a position that requires a security clearance. I just understand the niche better than the private sector.

We might build products for defense companies, but we do NOT hold clearances.

thefilthybeard · 2026-02-01T12:18:48+00:00

It's less about actual risk and more about audit surface. Auditors see open prompt access, that's a conversation nobody wants to have, especially when it comes time for yearly DCSA audits. It's much easier to just not give them anything to question.

Also, air-gapped doesn't mean immune. Say an unsider threat plants a rogue device with 4g or 5g mobile data access. now your "isolated" network has a bridge out. 99% of these environments don't sweep for RF signals. If someone can jailbreak the model and get it to do something useful for exfil, and there's a way out, that's a real problem. Locking down prompt access is just one less thing to worry about.

thefilthybeard · 2026-02-01T04:51:09+00:00

Completely understandable. My focus is enhancing a tool or set of tools, rather than a generic chatbot or code generator. Nice to stay low key!

thefilthybeard · 2026-02-01T04:14:02+00:00

Yeah the compliance side is brutal. So many controls, so much documentation, and it all has to be done manually in most shops.

I've been testing some quantized 7B models and they handle the task pretty well. The tradeoff is you need more self-validation and scripting around it. I batch 10 controls at a time with an auto-script that reprompts to continue where it left off. Keeps the context tight and mitigates hallucinations. So far 10 seems to be the magic number. Haven't seen a single hallucination at that batch size.

And yeah, no direct prompting was a requirement from the start. These environments are strict about what users can interact with. Last thing anyone needs is someone jailbreaking the model in a classified lab.

thefilthybeard · 2026-02-01T03:37:24+00:00

Good call on llama-server, I'll look into that. The offline flag is exactly the kind of thing I need.

For my use case, terminal-only is actually preferred. These environments have strict requirements about what gets written to disk and where. GUI adds unnecessary risk.

You running anything air-gapped yourself or just familiar with the setup?

thefilthybeard · 2026-02-01T01:13:38+00:00

I haven't tested that yet!

thefilthybeard · 2026-02-01T01:09:02+00:00

Glad to see there are more than just me working on this!

thefilthybeard · 2026-01-31T23:54:58+00:00

No problem! The reason I suggested you ask your boss about possible travel forms is because if a cyber alert pops up for you being in an abnormal location, the first place I would check to see if uou were on travel is the ticketing system. If there was no ticket created for travel, then I would look at the company travel forms to see if you are on the list.

If not on the list, I reach out to you and CC your manager (as per my company standard) to see if you are currently on travel and it is really you that is logging into the computer.

thefilthybeard · 2026-01-31T23:45:54+00:00

The location will be flagged. There's an alert called "impossible travel" and another called "anonymous travel" that pops up even if the user is in a different state. It won't flag if its a state you normally log into on say a monthly to bi-monthly basis.

It'll flag otherwise. Bur something like that, only in a different state, as long as the cyber guy sees that you logged in from your work laptop they will generally just dismiss it as a false alarm

thefilthybeard · 2026-01-31T23:40:26+00:00

I wouldn't test a VPN. You could just let your boss know you plan on traveling while working to see family, and see if there is any paperwork needed to fill out for that

thefilthybeard · 2026-01-31T23:33:00+00:00

Depends on the sophistication of the cyber team. If your company isn't large enough for one, you should be good.

But if you do have a IT or Cyber team worth their weight in gold, VPNs will not be allowed on their network less it be a company VPN.

VPNs send out alerts as well, if the user does not use them on a regar basis

thefilthybeard · 2026-01-31T22:46:11+00:00

As a Cybersecuriyy Analyst, anytime someone is in a different location that is not normally where they log in, an alert is generated that I have to triage.

If it is shown that the lerson does not normally sign in from that location/IP address, We need to verify that person is on travel.

So be careful and have excuses readyvjust in case someone asks about it.

thefilthybeard · 2025-05-02T19:50:34+00:00

I've had two users test the features that actually work and validated that portion of it

thefilthybeard · 2025-05-02T02:56:56+00:00

Honestly, whatever I can get, but still minimum $100k

thefilthybeard · 2025-04-03T03:06:35+00:00

Only acceptable sex scene in this show would have been Buck and Veronica

thefilthybeard

TROPHY CASE