JymboAI

Learn

Running AI models on your own computer: a beginner's guide

You don't always need the cloud to use AI. Here is what running a model locally means, what hardware you need, and how to get started safely.

Beginner3 min readBy JymboAI

Most people use AI through a website or app, which means the work happens on someone else's computers. But many capable AI models can also run on your own machine. This is called running a model locally.

Why run AI locally?

  • Privacy: your prompts and files stay on your computer.
  • Works offline: once a model is downloaded, no internet is needed to use it.
  • No per-message fees: after setup, using it costs only electricity.
  • Learning: it's one of the best ways to understand how AI models actually work.

There are trade-offs. The largest, most capable models need far more memory than a typical computer has, so local models are usually smaller. They're often very good, but not always as strong as the biggest cloud models.

What hardware do you need?

The most important number is memory. A model has to fit in memory to run at a reasonable speed.

  • Apple Silicon Macs (M1 and later): the processor and graphics share one pool of memory, so total RAM is what counts. 16 GB is a comfortable starting point for small models.
  • Windows or Linux PCs with a graphics card: the card's own memory (VRAM) matters most. Models that don't fit in VRAM can still run, but more slowly.
  • Computers without a dedicated graphics card: small models will run, just slower.

Model sizes, in plain terms

Model names often include a number followed by "B", such as 3B, 8B or 70B. That's the number of parameters in billions, which roughly measures the model's size.

Model sizeRough memory needed (Q4)Good for
1B–4B1–4 GBQuick tasks, older computers
7B–9B5–7 GBEveryday writing and Q&A
12B–14B8–11 GBStronger reasoning
30B–34B18–24 GBAdvanced use, high-memory machines
70B40+ GBWorkstation-class hardware

These are rough figures for compressed ("Q4") versions of each model. Use our Local Model Memory Estimator for your specific case, and read Quantization explained to understand the "Q" numbers.

Getting started with Ollama

Ollama (opens in a new tab) is a free, open-source tool that makes downloading and running models straightforward on macOS, Windows and Linux. Once it's installed from the official website, running a model is a single command in the terminal:

ollama run llama3.2

The first time, this downloads the model (often a few gigabytes), then opens a chat in your terminal. Type a message and press Enter. To see which models you have:

ollama list

Model names and availability change over time, so check the official Ollama model library for current options.

Staying safe

  • Download only from official sources. Get tools and models from their official websites or libraries.
  • Watch your disk space. Models are large files, and they add up quickly.
  • Local doesn't mean automatically private. If you connect a local model to tools, plugins or the internet, check what those connections can access.
  • Leave memory for everything else. Your operating system and apps need room too.

Note: Commands and model names in this guide reflect common usage at the time of writing. Always check a tool's official documentation for the latest instructions.

Where to go next