The AI your data never leaves
I’ve got a gaming PC in my house that spends most of its life doing nothing. Somewhere along the way it stopped being a game machine and started being the most useful box in my lab, because it runs AI models locally. No API key, no subscription, no data going anywhere.
If you’ve been nervous about your team pasting client information into public chatbots (and you should be), this is the version of AI you can actually say yes to. It runs on hardware you might already own, it’s free, and the conversation never leaves your building.
Why bother when ChatGPT exists
Three reasons, and only one of them is privacy.
Your data stays put. Anything you type into a hosted AI service goes to a company you don’t control. For most casual use that’s fine. For a client’s financial records, a patient name, or a contract under negotiation, it’s a problem you can’t undo after the fact. A local model physically cannot leak what it never transmitted.
No per-seat cost. Ten employees on a paid AI plan is a real monthly line item. A model running on hardware you own costs electricity.
It works when the internet doesn’t. Which, during hurricane season down here, is not a hypothetical.
What you need
The honest answer is a graphics card with decent memory. The model has to fit in video memory to run at a usable speed. Rough guide:
- 8GB of VRAM. Runs the small models fine. Good for summarizing, rewriting, drafting emails, quick questions.
- 12 to 16GB. The sweet spot. Mid-size models that are genuinely useful for real work.
- 24GB and up. Larger models, longer documents, faster responses.
No graphics card? It still runs on the CPU, just slowly. Words come out at reading speed instead of instantly. For a lot of tasks that’s perfectly acceptable. An old workstation with 32GB of RAM will do the job.
A Mac with Apple silicon is quietly great at this too, because the memory is shared between the processor and the graphics. A Mac mini with 16GB or more punches above its price here.
Getting it running
The tool is called Ollama, and this is one of the rare pieces of self-hosted software that just works.
- Download Ollama from its official site (bookmark the real one, don’t search for it).
- Install it. On Windows it’s a normal installer. On Linux it’s one command.
- Open a terminal and pull a model. Something like ollama run llama3.1 or ollama run qwen2.5 to start.
- Wait for the download. Models run a few gigabytes each.
- Type a question. That’s it, you’re talking to a local AI.
Total time is about fifteen minutes, most of it downloading. If you want a proper chat window instead of a terminal, install Open WebUI on top. It’s a browser interface that looks and feels like the commercial products, with saved conversations and multiple users.
Making it useful for the whole office
Once it’s running on one machine, you can point other computers on your network at it. That turns a spare PC into an internal AI service everybody can reach from a browser tab.
A few things to get right if you go that far:
- Keep it off the internet. Do not port forward this. Ever. If you need remote access, use a private network overlay like Tailscale or a VPN. An exposed AI endpoint is an invitation.
- Put it on its own network segment if you’ve got VLANs configured. A lab box with an experimental service on it does not belong on the same segment as your accounting machine.
- Turn on authentication in Open WebUI so it’s not open to anyone who finds the address.
What local models are actually good at
Set expectations correctly and you’ll be happy. A model running on your own hardware is not going to match the biggest commercial systems on hard reasoning. That’s fine, because most office work isn’t hard reasoning.
Genuinely good at: summarizing long documents, rewriting a rough email into something professional, pulling structured data out of messy text, drafting job descriptions and SOPs, translating, explaining a technical error message in plain English, first drafts of anything.
Not great at: complicated multi-step logic, current events (the model’s knowledge is frozen at training time), anything where being subtly wrong is expensive.
My rule is simple. Local models for anything touching client data or internal documents. Commercial models for the hard thinking on non-sensitive work. You don’t have to pick one.
The business case, in numbers
Say you’ve got eight people and you’d be paying twenty-something dollars a month per seat for AI access. That’s roughly $2,000 a year. A used workstation with a decent used graphics card runs $400 to $800 one time. It pays for itself before the summer’s out, and you keep the hardware.
That’s the same math that makes homelabs worth it in general. Old hardware plus free software replaces a surprising number of subscriptions.
Start small this weekend
Install Ollama on whatever machine you’ve got. Pull one small model. Ask it to summarize a long email thread. See if the output is good enough for your actual work, because that’s the only test that matters.
If it is, then you’ve got a path to giving your team AI tools without handing your client data to a third party. If you want help setting it up properly, segmented off your business network and locked down, that’s the kind of thing I do for clients around Delray Beach and West Palm. Reach out and let’s talk about it.