This piece is intended to show that it's possible for someone who isn't a tech expert (me) to get started with a local AI setup, but once you are convinced please escalate to more technical documents than you'll find below. This isn't a complete instruction manual, it's more like a primer that shows you one path that worked for me.
So I decided to build my own local AI setup.
Why?
I’m hoping this will support digital sovereignty, privacy and ultimately sustainability. Canada’s government expressed the need for Canada to develop digital sovereignty, meaning roughly that our nation can’t be destabilized by losing access to digital tools from elsewhere. Personally I’ve found the toughest part of the buy-Canadian moment has been digital services.
We’re all aware by now that the “free” software we rely on tends to have hidden costs. Among those is privacy and the monetization of our data. Some will argue that the loss of privacy is worthwhile for convenience offered by these services. I think that view is missing some context. For instance, many people don’t mind Google having their data, but don’t want a scam operation to get it. The irony is if any Google information is hacked it will get spread around by malicious actors very quickly.
I just want you to know you have a choice. So if you want to set up your own local AI, how difficult is it? I would have assumed I couldn’t possibly manage. I have some degree of technical skills, but I’m not a programmer, not a computer analyst, not a technician. I point this out because I genuinely think if you’re reading this you could probably set up your own local AI as well. The open source community has done a phenomenal job of building accessible tools.
What can my setup do?
At the moment I can enter text prompts, do image generation, do video generation, and set up Storytelling environments. I can do the text and storytelling from my phone, away from my home, without relying on Google or Microsoft, and with a secure connection that keeps my data to myself.
How did I set it up?
To be honest, I started with a bit of an advantage: I’m a gamer. I have a relatively new graphics card for that purpose. A graphics card is part of a computer which helps you play videogames and also is the hardware on which AI runs. I don’t know enough to tell you it’s worth buying one for AI purposes, but if you already have a gamer in the house you may already have what you need.
Here are the components I used. I accomplished all of it using prompts in Gemini or Copilot, and then following the instructions.
The path I followed is:1. Set up “Docker”
2. Set up “OpenwebUI” in Docker
3. Install Ollama
4. Download a model that fit my hardware
5. Connect OpenWebUI to the model
6. Install Tailscale on phone and computer.
7. Install/Set up similar components for Image/Video processing (Optional)
8. Install/Set up Sillytavern (Storytelling device, optional)
I will tell you what that means, but there are options for you, as long as you understand the structure.I’ll start by assuming you’ve never deliberately used any kind of AI. You’ll have heard people say “I asked AI” or “I asked chatgpt”. What they mean is that they opened an app on their computer or phone and submitted something like a google search. Like “When was the CN Tower constructed?” The thing they typed into is the “Frontend”. It's the bit you interact with. It takes in your question and it shows you the response, but it does not process it. The processing is done by the LLM or “model”. The model already knows a little bit about the CN Tower and can answer your question.
Docker is a way to simplify other programs. Like moving apartments by picking up your entire room and moving it from A to B without having to pack all your stuff and clean up.
The frontend I use is “openwebui”. It’s the thing that looks and acts like gemini, copilot or chatgpt. The LLM/model is an important choice, but to get started nearly the only important question is how big a model you want. Too big for your computer won’t run. It’s VERY easy to change if you install one too big, too small, or just don’t like what you got.
Tailscale is a secure way for you to run your AI at home and access it remotely. It means if you’re out shopping you can still use your local AI, you don’t have to revert back to Gemini or ChatGPT.
With these pieces set up, you have a functional system. It won’t do everything you want quite yet, but the core is there and you can add to it over time. Everything else is just customization. The biggest part of that is adding new “Tools”. Tools are how an LLM interacts with the world. Doing a web search is a tool. making a picture is a tool.
I’m not sure about why naming has evolved the way it has within the AI industry, but frequently there is confusing naming that makes it easy to not realize that the frontend and model are different.
So now you know what the parts are and what they do. How do you get started? Literally just type “How do I set up a local AI on my computer at home/work” and then following the instructions. Don’t get me wrong, it won’t happen in one afternoon. You’ll have to know how to open a command prompt and powershell. Both are already on your computer. You can find them in the "search" bar on windows. When the AI tells you to do that, you just type “how do I...” into the AI and copy (ctrl-c) and paste (ctrl-v) whatever text it gave you into the box that opens. Sometimes it will be errors. Just give the errors back to your AI for it to fix. It's really good at that. Repeat the process until all the pieces are in place. It took me about a week or two to set up, doing a few hours each day after work.
And now you have your own locally-hosted AI. If you had the graphics card already, this also didn't cost you anything (other than time) to set up. You can add functions from here if you want. You're on solid ground, so enjoy!