23 August 2026

 

A cloud-based AI assistant presents an uncomfortable bargain. In return for access to enormous computing resources, users send their questions, documents and sometimes commercially sensitive information to infrastructure controlled by somebody else.

For everyday requests, that may be acceptable. It becomes more complicated when the material includes unpublished research, customer records, proprietary source code, legal documents or confidential business plans.

The alternative is to bring the AI home. Open-weight large language models can now run on a desktop computer, AI workstation or small office server. Once the software and model files have been downloaded, a local LLM can answer questions, summarise documents and help write code without sending prompts to an external provider.

But “local,” “private” and “air-gapped” are not interchangeable.

 

What is an air-gapped local LLM?

 

A local large language model, also known as a local LLM, LLM on-prem or sLLM (small LLM), performs its calculations on the user’s hardware rather than in a remote data centre. That removes one obvious route through which information can leave an organisation, but the computer may still be connected to the internet.

Other software could transmit telemetry. A browser interface might load external resources, while extensions and integrations may contact cloud services. Malware can steal files regardless of where the AI model runs.

A genuinely air-gapped AI system is physically and logically isolated from external networks. It might consist of one standalone machine or several computers connected to a self-contained internal network. What matters is that there is no route to the public internet or another external network.

This isolation introduces inconvenience by design. The model cannot retrieve current information, install its own updates or consult online services. Air-gapping is less like activating privacy mode and more like establishing a small, self-contained computing environment.

 

How much memory does a local LLM need?

 

The central hardware question is memory.

An LLM consists of billions of parameters whose values must be accessible while the model is running. At 16-bit precision, the weights of a seven-billion-parameter model require roughly 14GB of memory. A 13-billion-parameter model needs approximately 26GB for its weights alone, before allowing for the context cache and other runtime overhead.

That makes a card such as the ASUS ROG Strix GeForce RTX 4090, with 24GB of GDDR6X video memory, a powerful but revealing example. Although it belongs to NVIDIA’s previous GPU generation, its large memory allocation remains useful for running AI models locally.

The RTX 4090 can hold the 16-bit weights of a seven-billion-parameter model, with capacity remaining for operation. An unquantised 13-billion-parameter model will not normally fit entirely within its VRAM, but an eight-bit or four-bit version generally can.

Quantisation stores model weights using lower-precision numbers, substantially reducing the memory requirement. The widely used llama.cpp project supports several quantisation levels, including four-bit and eight-bit formats. Compression can cause some loss of accuracy, but it makes capable local models accessible on consumer hardware.

A GPU normally provides the fastest inference. If part of a model spills into ordinary system memory, it may still run through combined CPU and GPU processing, but usually more slowly.

 

Three ways to build a private AI server

 

An entry-level local AI system

An entry-level system can use an existing computer with at least 16GB of system memory and sufficient solid-state storage.

Small quantised models in roughly the one-billion to four-billion-parameter range can handle basic rewriting, classification and question answering. They will not match the strongest cloud models, but they offer an inexpensive introduction to running AI offline.

A practical local AI workstation

A practical AI workstation should combine at least 32GB of system memory with a GPU carrying around 12GB to 24GB of VRAM.

At the upper end, the RTX 4090 can run many useful seven-billion to 14-billion-parameter models after suitable quantisation. It is a credible platform for private document analysis, coding assistance and editorial experimentation.

An advanced multi-GPU AI server

A serious research server needs more room to expand. This is where a workstation processor such as the AMD Ryzen Threadripper PRO 7955WX becomes relevant.

This previous-generation 16-core processor supports the WRX90 platform, which provides 144 usable PCIe lanes, including up to 128 PCIe 5.0 lanes, as well as eight-channel memory.

That abundance of connectivity allows a workstation to accommodate several GPUs and high-speed storage devices without the restrictions of an ordinary desktop platform. Larger models can then be distributed across multiple GPUs, although the RTX 4090 does not support NVLink. Multi-GPU LLM inference therefore depends on software capable of dividing work across cards connected through PCIe.

Buyers must also consider card spacing, cooling and software compatibility. The ROG Strix RTX 4090 is a particularly large 3.5-slot card, so fitting several into one chassis is not a routine desktop build.

 

Powering a local AI workstation

 

Power becomes part of the architecture too. Matrix calculations can keep GPUs heavily occupied for extended periods.

The be quiet! Dark Power Pro 13 1600W provides 1,600 watts of power with an 80 PLUS Titanium efficiency rating. It complies with the ATX 3.0 standard and includes two 12VHPWR connections for high-power graphics cards, making it a more natural foundation for an expandable local AI workstation than a conventional desktop power supply.

A large power supply does not itself guarantee stability or protect against a mains failure. Builders must calculate the complete system demand, check the power requirements of each GPU and leave sufficient headroom. An uninterruptible power supply is still needed if the server must remain operational during an external power interruption.

 

Which software can run an LLM locally?

 

Ollama provides one of the more accessible routes into local AI. It can download and run compatible models through a relatively simple interface, while llama.cpp offers more direct control and broad hardware support.

The first step is choosing a model for the intended task. A compact general-purpose model may be sufficient for summarising notes. Coding, multilingual work or analysing long documents may require a different model and considerably more memory.

Model licences deserve attention too. “Open weight” does not necessarily mean unrestricted open-source software. Some licences limit commercial uses or impose conditions on redistribution.

An air-gapped installation also requires a separate connected environment. The installer, dependencies, model weights and interface software must first be downloaded, checked and transferred to the isolated system.

NVIDIA describes the same two-stage process in its enterprise guidance: prepare the model assets while connected, then transfer and run them without access to remote model registries.

 

How to keep a local AI system offline

 

Wi-Fi, Bluetooth and wired connections to external networks must be removed or disabled. Downloaded files should come from trusted sources and be verified using published checksums where available.

Transfer drives must be controlled because removable storage becomes the bridge across the air gap—and therefore a possible route for malware. Updates, security patches and new models must follow the same download, verification and transfer procedure.

Isolation also does not make an LLM accurate. The model can still hallucinate, reproduce biases or mishandle documents. Air-gapping protects where the data goes; it does not guarantee the quality of what comes back.

 

Hardware for a private AI build

 

Build	Suitable starting point	Potential components
Starter	Existing computer, 16GB RAM and ample SSD space	Memory upgrade, NVMe SSD and external transfer drive
Workstation	32GB or more RAM and a 12GB–24GB GPU	ASUS ROG Strix RTX 4090, cooling and compatible power supply
Advanced	Multi-GPU workstation with 64GB–128GB RAM	AMD Ryzen Threadripper PRO 7955WX, be quiet! Dark Power Pro 13 1600W

 

The most private AI may ultimately be the one with the fewest impressive features. It cannot search the web, synchronise across devices or summon virtually unlimited computing power. But when it reads a confidential document, that document stays in the room.