Which Machine Runs a Model Locally

You want to run a language model on your own machine — for privacy, for the fun of it, or because you are tired of paying per request — and every guide you open starts with a shopping list. This one starts with the machine you already have. Before you buy anything, you can find out in three short readings what your computer can run today, and exactly which figure to look at if you do decide to buy.

The figure is memory. Not processor speed, not the number of cores, not the benchmark score on the box. A model is a file, and to run it that file has to sit in the memory of the part of the machine that does the work — the graphics card's own memory on a PC, the shared memory on a Mac. If it does not fit, nothing else matters; if it fits, the second figure, memory bandwidth, decides how fast the words come out. That is the whole of this guide, and the rest is reading the numbers.

Everything measured on this page was measured on one machine, with the commands shown: a plain Linux container with 16 GB of memory, four cores and no graphics card — the kind of machine a build server hands out, kept for the length of one run and then thrown away. It is not anybody's computer, and it is not a good machine for running models. It is here because the readings have to come from somewhere real, and a disposable one is the only somewhere this site has. Every other figure — a chip's bandwidth, a card's memory, a model's file size, another machine's speed — is quoted from the maker's own specification page or from a published test, with the source named. This site has no lab and claims none.

The readings you are about to see therefore have a hole in them where a graphics card would be, and the guide says so at each step rather than hiding it. Your own machine is the one that matters, and the three commands below read it.

By the end of this guide you will

• Read how much memory your machine has, and how much of it belongs to the graphics card, with one file you run yourself.
• Say which of eleven common models fit on your machine today, from their published file sizes, and which never will.
• Choose between a Mac with unified memory and a PC with a discrete graphics card using the two numbers that matter, not the ones on the box.
• Explain, in one paragraph, what a "4-bit" or "quantised" model is and why it fits where the full one does not.

Before you start

You need two things, and neither costs anything.

• Node.js 22 or newer. Run node --version. It prints a version such as v24.14.1, which is what the readings on this page used. If it prints an error, install Node.js from the official site.
• A terminal in an empty folder. Run mkdir machine-check, then cd machine-check. The file you write next reads your machine and touches nothing else: no key, no network, no installation.

If you have an NVIDIA graphics card, a command called nvidia-smi usually comes with its driver; the file uses it when it is there and says so when it is not. On a Mac there is no separate card to ask, and the file says that too.

Step 1 — Read how much memory your machine has

Create a file named machine.mjs and put this in it:

import os from 'node:os';

const gb = (bytes) => (bytes / 2 ** 30).toFixed(1);

// What stands where a value is withheld: a name for the value, not a blank. Printed by this
// script, never substituted afterwards.
const CPU_TYPE = '[CPU type]';
const GPU_TYPE = '[GPU type]';
const OS_BUILD = '[OS build number]';

// The platform's family, not its build: `os.release()` on Windows is the exact patch level.
const family = { win32: 'windows', darwin: 'macOS', linux: 'linux' }[os.platform()] ?? os.platform();

console.log(`system: ${family} on ${os.arch()} (build ${OS_BUILD})`);
console.log(`processor: ${CPU_TYPE}, ${os.cpus().length} logical cores`);
console.log(`memory: ${gb(os.totalmem())} GB in total, ${gb(os.freemem())} GB free right now`);

system: linux on x64 (build [OS build number])
processor: [CPU type], 4 logical cores
memory: 15.7 GB in total, 15.1 GB free right now

Run it with node machine.mjs. Three lines come back. Here is what they said on the container these readings were taken on, and how to read yours.

• system: linux on x64 (build [OS build number]) — the operating-system family as Node names it, and the processor architecture. linux is Linux, windows is Windows whatever the version, macOS is a Mac. Where the build number would go, the file prints a name for the value instead of the value. That is deliberate: a build number says exactly which published security fixes a machine has not installed, and this page is on the open web. Your own copy prints your own build if you want it — the file shows you where.
• processor: [CPU type], 4 logical cores — the same arrangement: the number of threads the machine runs at once is printed, the processor's model name is not. Note the count, and then put it aside: for running a model it matters far less than the next line.
• memory: 15.7 GB in total, 15.1 GB free right now — the number this guide is about. os.totalmem() returns bytes; the file divides by 2 to the power of 30 and shows one decimal. A machine sold as "16 GB" shows a little less here, because memory is counted in binary gigabytes and the hardware keeps a slice for itself. The second figure changes minute to minute — on a container that has just started, almost all of it is free — and the first is what you plan with.

Write your first figure down. On a Mac this is the memory the model will use, full stop: the chip's graphics cores share it with everything else, which is what Apple means by unified memory. On a PC with a graphics card it is the fallback, and the card's own memory is the figure you need next.

Step 2 — Read your graphics memory

Add this below what you wrote in Step 1, in the same file:

import { execFileSync } from 'node:child_process';

// Run a command and return what it printed, or null when the command is not there.
const run = (file, args) => {
  try {
    return execFileSync(file, args, { encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }).trim();
  } catch {
    return null;
  }
};

// The card's memory, not its name: `--query-gpu=memory.total` alone, so the model is never read.
// Add `name,` to that query on your own machine and the real name prints beside the memory.
const nvidiaMemory = run('nvidia-smi', ['--query-gpu=memory.total', '--format=csv,noheader']);

if (nvidiaMemory !== null) {
  const cards = nvidiaMemory.split(/\r?\n/);
  console.log(`graphics card: ${GPU_TYPE} (${cards.length} NVIDIA card found)`);
  console.log(`GPU memory: ${cards.join('; ')}`);
} else if (process.platform === 'darwin') {
  console.log('graphics card: none separate — a Mac: the graphics cores use the unified memory counted above');
} else {
  console.log('graphics card: no NVIDIA card found — read a discrete GPU\'s memory in its vendor\'s own tool');
}

graphics card: no NVIDIA card found — read a discrete GPU's memory in its vendor's own tool

Run node machine.mjs again. Step 1's three lines print first; then one more, or two. On the container:

• graphics card: no NVIDIA card found — read a discrete GPU's memory in its vendor's own tool — one line, and the whole of Step 2's answer here. nvidia-smi is not installed, so the file says so instead of guessing. This is the honest shape of the reading on a machine with nothing to ask: there is no second figure, and the memory from Step 1 is the only memory there is.

On a machine with an NVIDIA card, two lines print instead, and you will see them when you run the file yourself: a graphics-card line carrying a name for the card's model and how many cards answered, and a GPU-memory line carrying each card's memory in MiB — the unit nvidia-smi reports in, where 1024 MiB is one of Step 1's GB. Divide that figure by 1024 and you have the number Step 3 needs. The card's model name is withheld on this page for the reason the build number is; on your own machine the file tells you the one word to add to the query to see it.

On a Mac neither line appears. The file says instead that the graphics cores use the unified memory counted in Step 1 — there is no second figure to find, because there is only one pool.

Now you know which of the two figures this guide is about your machine has: the system's memory, and, if there is a card, the memory of the device that will actually hold the model. On a Mac they are the same number. On this container there is only the first: 15.7.

Step 3 — Set the published sizes against your memory

A model's file has to fit in that memory, so the last step is to compare. The sizes below are not measured here: each is what Ollama's library page lists for the size it serves by default when you type ollama run <name>, and the guide cites each page. Add this below Step 2's code:

// The download size of each model at the size Ollama serves by default, as its library page lists
// it on 2026-09-11 (the guide cites each). A file this size has to sit in memory to run.
const models = [
  ['llama3.2:1b', 1.3],
  ['llama3.2:3b', 2.0],
  ['qwen3:4b', 2.5],
  ['gemma3:4b', 3.3],
  ['llama3.1:8b', 4.9],
  ['qwen3:8b', 5.2],
  ['gemma3:12b', 8.1],
  ['qwen3:14b', 9.3],
  ['gemma3:27b', 17],
  ['qwen3:32b', 20],
  ['llama3.1:70b', 43],
];

// This guide's rule of thumb, not a measurement: a model's file should take no more than three
// quarters of the memory it will live in, leaving the rest for the text it holds and the system.
const room = (memoryGb) => memoryGb * 0.75;

const systemGb = os.totalmem() / 2 ** 30;
const gpuMib = nvidiaMemory === null ? null : Number(/(\d+) MiB/.exec(nvidiaMemory)?.[1] ?? NaN);
const gpuGb = gpuMib === null || Number.isNaN(gpuMib) ? null : gpuMib / 1024;

console.log(`memory to fit into: ${systemGb.toFixed(1)} GB system` + (gpuGb === null ? '' : `, ${gpuGb.toFixed(1)} GB on the GPU`));
console.log('model           file      GPU memory  system memory');
for (const [name, sizeGb] of models) {
  const inGpu = gpuGb === null ? 'n/a' : sizeGb <= room(gpuGb) ? 'fits' : 'no';
  const inSystem = sizeGb <= room(systemGb) ? 'fits' : 'no';
  console.log(`${name.padEnd(15)} ${(sizeGb.toFixed(1) + ' GB').padEnd(9)} ${inGpu.padEnd(11)} ${inSystem}`);
}

memory to fit into: 15.7 GB system
model           file      GPU memory  system memory
llama3.2:1b     1.3 GB    n/a         fits
llama3.2:3b     2.0 GB    n/a         fits
qwen3:4b        2.5 GB    n/a         fits
gemma3:4b       3.3 GB    n/a         fits
llama3.1:8b     4.9 GB    n/a         fits
qwen3:8b        5.2 GB    n/a         fits
gemma3:12b      8.1 GB    n/a         fits
qwen3:14b       9.3 GB    n/a         fits
gemma3:27b      17.0 GB   n/a         no
qwen3:32b       20.0 GB   n/a         no
llama3.1:70b    43.0 GB   n/a         no

Run node machine.mjs a third time. After the lines from Steps 1 and 2, a table. On the container it read:

• memory to fit into: 15.7 GB system — the figures from the first two steps, restated. Only one appears, because Step 2 found no card. On a machine with one, this line carries both.
• llama3.2:1b 1.3 GB n/a fits through qwen3:14b 9.3 GB n/a fits — eight models fit in the system's 15.7 GB by this guide's rule. The middle column reads n/a for every row: there is no card, so the question of whether a model fits in a card's memory has no answer here rather than the answer "no". A column that said no would be telling you something false.
• gemma3:27b 17.0 GB n/a no, qwen3:32b 20.0 GB n/a no, llama3.1:70b 43.0 GB n/a no — these do not fit in 15.7 GB, and no setting changes that. The last column is the one that decides here.
• What the middle column says on a machine with a card is fits or no against the card's memory instead, and that is the column to plan with — it is almost always the smaller of the two, and the model runs at the speed of the memory it fits in. On a 6 GB card, for instance, the first four rows read fits and everything from llama3.1:8b down reads no while the system column still reads fits: those models run, but part of the work leaves the fast memory and the words come slower.

The rule the table applies is this guide's, and it says so in the code: a model's file should take no more than three quarters of the memory it will live in. The rest is for the text the model holds while it works and for the system itself. It is a margin, not a measurement; a runner that squeezes a file into 90% of a card's memory may work, and may not.

Where do the sizes come from? Ollama's library lists llama3.2:1b at 1.3GB and llama3.2:3b at 2.0GB. Ollama's library lists qwen3:4b at 2.5GB, qwen3:8b at 5.2GB, qwen3:14b at 9.3GB and qwen3:32b at 20GB. Ollama's library lists gemma3:4b at 3.3GB, gemma3:12b at 8.1GB and gemma3:27b at 17GB. Ollama's library lists llama3.1:8b at 4.9GB and llama3.1:70b at 43GB. Those are download sizes for the compressed form the library serves by default, which is why an 8-billion-parameter model is a 5 GB file rather than a 16 GB one — the next section explains the compression.

Put your own two figures against the same list and you have your answer for today: with 8 GB of graphics memory, the models up to 5.2 GB fit; with 16 GB, everything up to qwen3:14b at 9.3 GB; with 24 GB, gemma3:27b at 17 GB joins them; qwen3:32b at 20 GB wants 32 GB; and llama3.1:70b at 43 GB wants 64 GB of memory that one device can see.

The one rule

The model must fit in the memory of the device that runs it, and then bandwidth sets the speed. Two figures, in that order: first the size of the memory, which decides whether a model runs at all; then how fast that memory can be read, which decides how many words per second you get from a model that fits.

The second figure is why two machines with the same amount of memory can feel completely different. To produce each word the model reads through all of its numbers, so the speed at which memory can be read is, near enough, the speed at which words come out. A published test shows the shape of it. In the llama.cpp maintainer's published table for a 7B model, the M1 with 68 GB/s of memory bandwidth and a 7-core GPU generates 14.19 t/s at Q4_0. In the same table, the M4 with 120 GB/s and a 10-core GPU generates 24.11 t/s at Q4_0. In the same table, the M5 Max with 614 GB/s and a 40-core GPU generates 119.92 t/s at Q4_0. The discussion's own plot sets prompt processing against GPU cores and text generation against bandwidth. Read the three rows as a line: nine times the bandwidth, about eight times the words per second (t/s is tokens per second, a token being a word or part of one).

Mac unified memory against a discrete card

On a Mac the graphics cores and the processor share one pool, so a 32 GB Mac can hold a 20 GB model in the fast memory. Apple's MacBook Pro tech specs list the M5 chip with 153GB/s memory bandwidth. Apple's MacBook Pro tech specs list the M5 Pro chip with 307GB/s memory bandwidth. Apple's MacBook Pro tech specs list the M5 Max with 460GB/s memory bandwidth with the 32-core GPU and 614GB/s with the 40-core GPU. The MacBook Pro's unified memory starts at 16GB, 24GB or 32GB and is configurable up to 128GB on the M5 Max with the 40-core GPU. Apple's Mac mini tech specs list 16GB unified memory, configurable to 24GB or 32GB, at 170GB/s memory bandwidth. So the Mac's strength is size: a large pool the model can live in, at a bandwidth that rises with the chip's tier.

On a PC the card's memory is separate, and its size is the wall. NVIDIA's specifications list the GeForce RTX 5090 with a standard memory configuration of 32 GB GDDR7 on a 512-bit memory interface. NVIDIA's specifications list the GeForce RTX 5080 with a standard memory configuration of 16 GB GDDR7 on a 256-bit memory interface. NVIDIA's specifications list the GeForce RTX 5070 Ti with 16 GB GDDR7 on a 256-bit interface and the RTX 5070 with 12 GB GDDR7 on a 192-bit interface. The memory figure alone tells you the ceiling on the file size; the interface width is the one to compare between cards of one generation, because a wider interface moves more bits per clock. The card's strength is speed on a model that fits; its weakness is the 12 or 16 GB wall, past which a model spills into the slower system memory or does not run.

Quantisation, in one paragraph

Quantization is a technique to reduce the computational and memory costs of running inference by representing the weights and activations with low-precision data types like 8-bit integer (int8) instead of the usual 32-bit floating point (float32). Reducing the number of bits means the resulting model requires less memory storage, consumes less energy (in theory), and operations like matrix multiplication can be performed much faster with integer arithmetic. In plain terms: a model is billions of numbers, and each number can be stored with more or fewer bits. Store each one with 8 bits instead of 16 and the file halves; with 4 bits it quarters, at a cost in accuracy that most people never notice in conversation. That is the arithmetic behind Step 3's table: an 8-billion-parameter model at 4 bits is 8 billion times half a byte, about 4 GB, and the 4.9 GB Ollama lists is close to that. It is also why the labelsQ4_0 and Q8_0 appear in the llama.cpp table — the same model at 4 and at 8 bits, the 4-bit one faster because there are fewer bytes to read for every word.

When to use it, and when not

Terms that came up

• Model file — the model as a file on disk: billions of numbers, stored at some number of bits each. It must fit in memory to run.
• Parameters — those numbers; "8B" means eight billion of them. More parameters, bigger file, usually better answers.
• Unified memory — the Mac's one pool of memory shared by the processor and the graphics cores; the model lives in it directly.
• Discrete graphics card — a separate card with its own memory (GDDR7 on the current NVIDIA cards); on a PC, the model runs in that memory.
• GB and GiB — the file sizes on this page are in gigabytes as the library lists them; the readings divide bytes by 2 to the power of 30, which is why a "16 GB" machine reads 15.7.
• MiB — the unit nvidia-smi prints when there is a card to ask; 1024 MiB is one of Step 1's GB. Nothing printed it on this page's readings, because the machine had no card.
• n/a — what the table's GPU memory column says when Step 2 found no card: not "it does not fit", but "there is nothing here to fit into".
• Memory bandwidth — how many gigabytes per second the memory can be read; the figure behind words per second.
• Memory interface width — the number of bits a card's memory moves per clock; wider is faster, other things equal.
• Quantisation — storing the model's numbers with fewer bits; Q4_0 is 4 bits, Q8_0 is 8. The reason a 16 GB model becomes a 5 GB file.
• t/s — tokens per second, the speed a model produces text; a token is a word or a piece of one.
• Prompt processing and text generation — reading what you gave the model, and writing the answer; the published table measures both, and only the second depends mainly on bandwidth.

In short

Find the memory of the device that will hold the model — the card on a PC, the whole machine on a Mac — and keep the model's file to three quarters of it. Then, if you are choosing hardware, compare memory bandwidth, because that is what turns a model that fits into words per second. Everything else on the box comes after those two numbers.

What changed recently

No source of this guide has changed since the last check.