How to Run an AI Agent Offline (No API Key, No Token Bill)

You can run a capable AI agent offline, and the setup takes about fifteen minutes plus a download. The model weights sit on your disk, inference runs on your own CPU or GPU, and the reasoning loop needs no API key and no vendor account. Atomic Agent talks to a local llama.cpp server on 127.0.0.1 and drives real multi-step tasks with Wi-Fi off.

The practical entry point in 2026 is roughly 16 GB of memory, with 24 to 32 GB the comfortable floor for real work. This guide is the full path for Atomic Agent v0.6.5: install, pull a model, check that the context window fits the agent loop, switch off the three things that reach the network on their own, disconnect, and run a task.

What “fully offline” actually means here

It means the whole reasoning loop runs on your hardware. The weights are a file on disk. Inference happens locally against a llama.cpp server bound to localhost. No inference request crosses the network, and you need no API key for the agent to function.

The agent still drives your files, shell, and git with the network off. It reads and writes code, runs commands, and commits work without contacting a provider. Memory persists in a plain SQLite file on your disk rather than a cloud account, so context carries across runs without anything being uploaded.

Two honest boundaries are worth stating up front. First, a tool you ask for can still use the network: if you tell the agent to search the web, that specific step goes out. Second, out of the box the agent sends anonymous usage analytics, checks for a newer llama.cpp backend when the daemon starts, and checks for a new release when the TUI opens. None of that carries your files or prompts, and Step 7 turns all three off.

If the distinction between a local model and a local agent is new to you, what a local AI agent is covers the definition before you start installing things.

Why run an agent offline at all

People run agents offline for three reasons: a fixed cost instead of a per-token bill that grows with every tool result, work that legally cannot go to a third-party processor, and freedom from rate limits and outages. If you are still deciding whether local is right for you, local agent versus cloud agent weighs both sides, including where cloud wins. This guide assumes you have decided and want it working.

Hardware requirements

Memory is the constraint that decides everything else. The weights have to fit, and then the agent needs headroom on top, because the KV cache grows with every tool result that lands in the context. Budget for both.

A CPU-only machine works. Inference is slower, sometimes by an order of magnitude, but nothing about the agent loop requires a GPU. On Apple Silicon the numbers above apply to unified memory, with the caveat that macOS caps GPU-addressable memory at roughly 75 percent of the total, so a 24 GB Mac gives the model about 18 GB.

The gap between the 4B and the 27B is real and worth understanding before you pick. For the memory math, quant table, and two measured agent runs on the larger model, see how to run Qwen 3.8 27B locally. If you are not sure which side of the line your machine falls on, start with the 4B: it downloads in a minute and tells you quickly whether the workflow suits you.

Beyond memory you need free disk for the GGUF file, the single-file format llama.cpp uses to store quantized weights, plus macOS, Linux, or Windows, and a terminal. No API key, no account, no payment method.

How to run an AI agent offline, step by step

Managed mode is the recommended path. Atomic Agent downloads the llama.cpp binary and a GGUF model for you, picks a GPU if one is available, and supervises the local daemon. You do not wire anything up by hand. Other local agents split these steps differently, and how Atomic Agent, Hermes, and OpenClaw compare lays out where each one draws the line.

Step 1. Install Atomic Agent

curl -fsSL https://atomicagent.io/install | sh

This is one of only two steps that need the internet. The script fetches the binary and puts the atomic-agent command on your path. Confirm it landed:

atomic-agent --version

If the command is not found, open a new terminal window so the shell picks up the updated path.

Or download the desktop app from atomicagent.io/download. This guide uses the terminal commands throughout.

Step 2. See what models are available

atomic-agent models list

The catalog prints each entry with its slug and download size. Two matter for this guide: qwen-3.5-4b at 2.7 GB, which is the default and runs almost anywhere, and qwen-3.8-27b at 17.9 GB, which needs 24 GB of memory and is the one to pick if you have it. The larger entry is a quantized build of the open-weight Qwen3.8-27B release, whose model card lists the licence and the context length the weights were trained for.

Choosing between them is a bandwidth and memory question, not a preference. Read the table above, then decide once. Switching later means another download.

Step 3. Pull the weights

atomic-agent models pull qwen-3.5-4b

This is the second and last step that needs the network. pull downloads the GGUF file to your disk and is idempotent, so an interrupted download resumes rather than starting over when you run it again.

For the larger model, substitute the slug:

atomic-agent models pull qwen-3.8-27b

Expect this to take a while on the 27B. It is 17.9 GB, and nothing else in the setup depends on it finishing, so start it before you read the rest of this page.

Step 4. Make the model active

atomic-agent models use qwen-3.5-4b

use sets which downloaded model the agent runs against. Pulling a model does not select it, which is deliberate: you can keep several on disk and switch between a fast small one and a capable large one depending on the task.

Step 5. Check the context window

An agent needs far more context than a chat client, because its system prompt carries every tool definition before any work begins: file operations, shell, git, browser, memory, and whatever MCP servers you have connected. That is tens of thousands of tokens on an empty conversation.

By default the managed daemon sizes the window to the free memory on your GPU when it starts. On a machine with plenty of VRAM that is enough. On a small card or a CPU-only laptop it can land below what the agent needs, and the agent then refuses to start rather than silently truncating its own instructions. If you are on modest hardware, pin the window now:

atomic-agent config set '{"localModels":{"managed":{"contextSize":65536}}}'

Leave completionMaxTokens at its default of 16384. Lowering it to 8192 cuts long answers off mid-way, and the agent then spends steps retrying. If you skip this step and hit the limit, the exact warning and its fix are in Troubleshooting.

Step 6. Start the local model server

atomic-agent models start

This launches the llama.cpp daemon on localhost. While you are still online, the first start also checks GitHub for a newer llama.cpp build and pulls it if there is one. It then loads the weights into memory, which on the 27B takes a noticeable moment and on the 4B is nearly instant. The daemon keeps running in the background, so you start it once and leave it.

Verify it came up healthy:

atomic-agent models status

A working state reports the mode as managed, names your active model, and shows health as ok. It also prints the context window the daemon actually allocated. If you pinned 65536 in Step 5 and it reports something smaller, the config did not apply: re-run the config set command, then atomic-agent models stop && atomic-agent models start.

Step 7. Switch off the three things that reach the network

With the network off these calls simply fail, so this step is about a clean result rather than a working one. If you want a network monitor to show nothing at all, or you are on an air-gapped machine, switch off all three:

# 1. backend auto-update when the daemon starts
atomic-agent config set '{"localModels":{"managed":{"autoUpdate":false}}}'

# 2. release check when the terminal UI opens
export ATOMIC_AGENT_UPDATE_CHECK_ON_STARTUP=false

The third is anonymous usage analytics and crash reports. They never include your messages, file paths or tool arguments, and you turn them off from inside the terminal UI (atomic-agent tui):

/privacy analytics off

The same switch lives in config.json as "analytics": { "enabled": false }, and /privacy on its own shows everything that can leave the machine.

Step 8. Pull the network plug

Now prove it. Disconnect Wi-Fi, unplug the Ethernet cable, or turn on airplane mode.

The weights are already on disk and the local server is already running, so the agent has everything it needs. This is the step that separates a real local agent from a cloud wrapper: if it were quietly calling an API, it would break here.

Doing this on your first run is worth the thirty seconds. It converts a claim you read on a website into something you watched happen on your own machine.

Step 9. Run a task

Change into the directory you want the agent working in, then start it:

cd ~/projects/my-project
atomic-agent run

run opens an interactive session. It does not take the task as an argument: the agent starts up, prints a you> prompt, and you type the goal there. This catches people out, because most CLI tools expect the instruction inline.

The agent operates in whatever directory you launched it from, which is why the cd comes first. Three flags are worth knowing:

• --cwd sets the working directory explicitly instead of relying on where you launched from
• --max-steps caps how many steps a single run can take, which is useful when you are testing and do not want a task to wander
• --no-approval skips the confirmation prompt before each action

Leave --no-approval off for your first few runs. Watching the agent ask permission for each file write and shell command is the fastest way to build an accurate picture of what it actually does, and it is also the moment you would notice a tool reaching for the network.

Give it something real: rename files across a folder, refactor a function, summarize a local document, stage and commit your changes. It reasons locally and uses local tools, with the network still off.

Verifying that it really runs offline

The airplane-mode test is the honest one, but there are two more checks worth running if you want to be thorough.

Check what the agent is pointed at. The config lives at <stateDir>/config.json, where the default state directory is ~/.atomic-agent:

atomic-agent config get

For local inference you want the mode set to managed, and the URL pointing at localhost. Anything else means the agent is talking to a server you should be able to name.

Watch the connections. On macOS or Linux, run a task with a network monitor open and look at what the process opens. In managed mode, after Step 7, you should see traffic to 127.0.0.1 on the llama-server port and nothing else, unless a tool you invoked went looking for the web.

If your local server runs somewhere other than the default, set it in config.json:

{
  "localModels": {
    "mode": "external",
    "url": "http://localhost:8080"
  }
}

As long as this points at localhost and no cloud key is configured, the inference stays on your machine.

Does a small local model really work well enough

Yes for reliability, with trade-offs in capability and speed. The failure people expect from a quantized model is malformed tool calls, and Atomic Agent prevents it with GBNF grammars: tokens that would break the schema are filtered out during sampling, so the model cannot emit an invalid call. How grammar-constrained tool calling works walks through the mechanism. In our measured runs on the 27B, six steps across two tasks produced zero malformed calls.

Capability and speed are the real trade. A 4B model plans shorter than a frontier cloud model, and on a 32 GB Mac the 27B took roughly three minutes to list a directory and describe three files. That suits work you start and walk away from. For the longer version of that trade-off, see what a local AI agent is.

What an agent can and cannot do offline

Works with the network off:

• Reading, writing, and generating code across your filesystem
• Running shell commands and reading their output
• Navigating directories, renaming and reorganizing files
• Git operations: status, diff, stage, commit, branch
• Summarizing and transforming local documents
• Persistent memory across sessions, stored in local SQLite

Needs a connection, because the task itself does:

• Web search
• Browser automation against live sites
• Fetching a URL
• Any MCP server that talks to a remote API
• Pulling a new model or updating the agent binary

The line is clean: the agent’s reasoning never needs the network, and a tool needs the network only when the task inherently does. If you ask it to find something on the web in airplane mode, that step fails and the agent tells you so. Nothing else degrades.

If you are choosing a coding agent that works with a local model, our list of open-source Claude Code alternatives compares which ones support llama.cpp, Ollama or LM Studio, and what each sends home by default.

Troubleshooting

The agent refuses to start with a context warning

This is the most common first-run failure, and it is not a broken install:

WARN context window too small for the agent prompt
{"contextWindow":16384,"required":72048}

Cause. The agent’s system prompt includes every tool definition before any work starts, so the floor is tens of thousands of tokens on an empty conversation. The daemon sized the window to your free GPU memory and it landed below that floor. This is the single biggest difference between running a model for chat and running it for agent work, and it is why a model that works fine in a chat app fails here.

Fix. Pin the context size and restart the daemon so the new window is allocated:

atomic-agent config set '{"localModels":{"managed":{"contextSize":65536}}}'
atomic-agent models stop && atomic-agent models start

The restart is not optional. The engine allocates the KV cache when it loads the model, so a config change without a restart leaves the old window in place. After restarting, atomic-agent models status should report a context window of 65536.

The daemon fails to load the model, or the machine starts swapping

Symptom. models start hangs, the process is killed, or the whole machine grinds while the model loads.

Cause. The weights plus the KV cache exceed available memory. Raising contextSize to 65536 is not free: it allocates cache up front, and on a machine that barely fit the weights it is what pushes you over.

Fix. Step down to a smaller model with atomic-agent models use qwen-3.5-4b, or lower contextSize to 32768. On the 4B, 32K is usually enough for file and shell work. Close other memory-hungry applications before deciding you need a smaller model, since a browser with many tabs can easily be the difference.

The agent dies partway through a long task

Symptom. The first several steps run fine, then the process exits twenty steps in.

Cause. The KV cache growing. Every tool call returns data that stays in the context, so memory pressure builds as the task goes rather than showing up at load time.

Fix. Cap the run with --max-steps while you are calibrating, and step down one model size or quant if you are near the memory ceiling. Breaking a long task into several shorter runs also works, since memory carries over in SQLite even when the context does not.

The agent starts but cannot reach the model

Symptom. The startup log shows a health check failure against a port you did not expect.

Cause. The config was switched to external mode, so the agent is looking for a llama-server you are supposed to be running yourself, and it ignores its own managed daemon.

Fix. Run atomic-agent config get and check the mode. If it says external, set it back to managed and confirm the port in the managed config matches the one in the startup line.

Everything works, but responses take minutes

Symptom. A single step takes one to two minutes on a machine that is not swapping.

Cause. This is normal for local inference on consumer hardware, not a misconfiguration. A dense model at 4-bit on a laptop generates tokens at a fraction of cloud speed, and an agent generates a lot of them.

Fix. There is no fix, only choices. A smaller model responds faster. A GPU with enough VRAM for full offload is the single largest speedup available. Beyond that, restructure how you use it: give the agent a batch of work and come back, rather than sitting through each step.

Tool calls come back malformed

Symptom. Steps fail with parsing errors on the model’s output.

Cause. The model is generating freely rather than under a grammar. This happens when you point the agent at a third-party runtime that does not apply GBNF constraints.

Fix. Confirm you are running through the managed daemon rather than an external server configured without grammar support. Constraining structure at the decoder is exactly what keeps a quantized model from breaking the schema, and without it a small model will occasionally miss.

Bottom line

Running an agent offline is now a fifteen-minute setup rather than a project. Install, pull a model that fits your memory, check the context window, switch off the three network touchpoints, disconnect, and type your goal at the prompt.

Key takeaways:

• Only two steps need the internet: installing the binary and pulling the weights
• Match the model to your memory: qwen-3.5-4b from 8 GB, qwen-3.8-27b from 24 GB
• Check the context window before your first run, and pin contextSize to 65536 on modest hardware
• Three switches make the machine silent: /privacy analytics off, autoUpdate: false, ATOMIC_AGENT_UPDATE_CHECK_ON_STARTUP=false
• Grammar-constrained decoding is what makes tool calls reliable on a quantized model, and it is not something you configure
• You are trading speed for cost and control, and the trade suits work you can start and walk away from

Ready to try it? Follow the quickstart guide and run your first task offline in a few minutes, or download the desktop app.