Air-Gapped AI Deployment: The Complete Guide for Secure Enterprises

Most guides to "on-premise LLM deployment" assume your servers still talk to the internet. They walk through GPU selection, inference engines, and scaling strategies, all useful, but none of it matters if your compliance framework requires true network isolation. Defense contractors working in SCIFs, healthcare systems processing patient records under HIPAA, and financial institutions handling insider information all need something different: AI that works with zero external connectivity.

This guide covers what an air-gapped AI deployment actually requires, from hardware and software architecture to compliance mapping, update strategies, and vendor evaluation. If your data cannot leave the building, this is the reference you need.

What Air-Gapped AI Actually Means (and Why "On-Premise" Is Not Enough)

The term "on-premise" has become a catch-all that obscures critical security differences. In practice, there are four distinct deployment models, and the gap between them is wider than most vendor marketing suggests:

• Cloud SaaS: Your data travels to the vendor's infrastructure over the public internet. Processing happens on shared or dedicated cloud servers. This is the fastest to deploy and the least controlled.
• Private Cloud / VPC: A dedicated instance runs in your cloud account (AWS, Azure, GCP) or the vendor's cloud as a single-tenant environment. Your data stays within a defined network boundary, but the servers still have internet connectivity for updates, licensing, and telemetry.
• On-Premise (connected): Hardware sits in your data center. You control physical access. But the servers maintain outbound internet connections for software updates, model downloads, license verification, and often vendor support tunnels. Most "on-premise" guides describe this model.
• Air-Gapped (fully disconnected): No internet. No outbound connections. No DNS resolution. No NTP sync to external servers. Every component of the AI stack runs locally, and updates arrive on physical media. This is the model required for classified environments, ITAR-controlled data, and the strictest compliance postures.

The distinction matters because compliance auditors, procurement officers, and security teams evaluate these models differently. An "on-premise" deployment that phones home for license checks or sends anonymized telemetry to the vendor may still violate ITAR controls or fail a CMMC assessment. When your regulatory framework says "no data egress," it means none.

Three Deployment Models Compared: VPC, On-Premise, and Air-Gapped

Choosing between deployment models involves trade-offs across security, operational overhead, compliance coverage, and cost. The right answer depends on what data you process and which regulatory frameworks govern your organization.

For organizations evaluating these options in detail, our self-hosted AI deployment page covers each model with technical specifications, supported operating systems, and delivery methods.

The Compliance Case: Which Regulations Require Air-Gapping

Not every organization needs a fully disconnected deployment. But several regulatory frameworks either mandate or strongly favor air-gapped infrastructure for AI workloads that process controlled or sensitive data.

ITAR (International Traffic in Arms Regulations)

ITAR controls the export of defense-related articles and services. Technical data subject to ITAR cannot be stored on or transmitted through systems accessible to foreign nationals or foreign-hosted infrastructure. For defense contractors using AI to analyze technical manuals, engineering drawings, or maintenance records, an air-gapped deployment ensures that controlled data never traverses a network path that could reach non-U.S. infrastructure. Cloud deployments, even on AWS GovCloud, require careful review of who has administrative access.

CMMC 2.0 (Cybersecurity Maturity Model Certification)

CMMC Level 3 requires implementation of controls from NIST SP 800-171 plus additional enhanced security requirements for Controlled Unclassified Information (CUI). While CMMC does not explicitly require air-gapping, the controls around media protection (MP), system and communications protection (SC), and access control (AC) are significantly easier to satisfy when the AI system has no external network connectivity. Many defense primes treat air-gapping as the default for CUI-processing AI tools.

FedRAMP High / DoD IL4-IL5

FedRAMP High baseline includes 421 controls, many of which address network boundary protection, data flow enforcement, and continuous monitoring. DoD Impact Level 4 (IL4) covers CUI, and IL5 covers CUI and mission-critical data. For AI systems processing documents at these impact levels, air-gapped deployment eliminates entire categories of control requirements related to network security monitoring and boundary defense, because there is no boundary to defend.

HIPAA

HIPAA does not require air-gapping. Many healthcare organizations process PHI with cloud-hosted AI tools under a BAA. However, large health systems, research hospitals, and organizations with particularly sensitive datasets (psychiatric records, substance abuse treatment, genomic data) often choose air-gapped deployment for maximum control. For a detailed analysis of deployment models and BAA requirements, see our HIPAA-compliant AI document processing guide.

CJIS (Criminal Justice Information Services)

The CJIS Security Policy governs access to criminal justice data, including National Crime Information Center (NCIC) records, fingerprint data, and case files. AI systems processing CJIS data must operate within CJIS-compliant enclaves with strict access controls. Air-gapped deployment is not explicitly mandated, but it is the most straightforward path to satisfying the network security, encryption, and access control requirements.

Architecture of an Air-Gapped Document AI Stack

A functional air-gapped AI system is more than an LLM running on a GPU. Document intelligence requires a full pipeline: ingestion, parsing, OCR, chunking, embedding, vector storage, retrieval, and inference. Every component must run locally with no external dependencies.

The Complete Offline Stack

Here is a reference architecture for an air-gapped document AI deployment. Every component runs on local hardware with zero internet connectivity:

• LLM Inference: Llama 3 (8B or 70B) served via vLLM or llama.cpp. Open-source models with no licensing callbacks. Supports chat, summarization, extraction, and question-answering over retrieved context.
• Embedding Generation: Local Voyage or E5 models generate vector representations of document chunks. These models run on the same GPU or a dedicated embedding server, depending on throughput requirements.
• Vector Database: Self-hosted Qdrant or Milvus stores and indexes embeddings for semantic search. Both support single-node deployment suitable for air-gapped environments without requiring distributed cluster management.
• Document Parsing and OCR: Apache Tika handles format detection and text extraction for PDFs, Word documents, spreadsheets, and images. Tesseract OCR processes scanned documents and image-based PDFs. Both are open-source with no external dependencies.
• Orchestration Layer: A containerized application (Docker or Podman) coordinates the pipeline: receives documents, routes them through parsing and OCR, chunks the text, generates embeddings, stores vectors, and serves queries through the LLM with retrieved context.

How Data Flows (Everything Stays Inside)

The critical property of this architecture is that data never leaves the security boundary at any stage:

1. Document upload: User uploads a PDF, Word document, or image through the local web interface or API.
2. Parsing: Tika extracts text. Tesseract handles scanned pages. All processing happens in local memory.
3. Chunking: The extracted text is split into semantically meaningful segments (typically 512-1024 tokens).
4. Embedding: Each chunk is converted to a vector using the local embedding model. No API calls.
5. Storage: Vectors are indexed in the local Qdrant/Milvus instance. Original documents are stored on encrypted local disk.
6. Query: When a user asks a question, the query is embedded locally, relevant chunks are retrieved from the vector database, and the LLM generates a response grounded in the retrieved context.
7. Response: The answer is returned to the user with source citations. Nothing is logged externally.

At no point does any data, whether the original document, extracted text, embeddings, queries, or responses, leave the local system.

Hardware Requirements

Both configurations support Ubuntu 22.04 LTS and Red Hat Enterprise Linux (RHEL). The entire stack runs in Docker containers (or Podman for environments that prohibit the Docker daemon). For detailed hardware specifications and pricing, see our hardware requirements breakdown.

Use Cases: Who Needs Air-Gapped AI

Air-gapped deployment is not for everyone. The operational overhead is real: physical media updates, manual patching, and limited vendor support. But for organizations handling the following data types, it is the only option that fully satisfies their security and compliance requirements.

Defense Contractors and SCIFs

A defense contractor analyzing ITAR-controlled technical data for a weapons system cannot send that data to any external service, period. In a SCIF (Sensitive Compartmented Information Facility), the AI system must run on hardware inside the facility with no network connectivity to the outside. Use cases include analyzing engineering specifications, searching maintenance manuals, extracting data from technical orders, and summarizing test reports. The system arrives pre-loaded on encrypted drives, is installed by cleared personnel, and receives updates through the same physical channel.

Healthcare and Life Sciences

While HIPAA allows cloud processing with a BAA, some healthcare organizations choose air-gapping for their most sensitive workloads: psychiatric records, substance abuse treatment data, genomic research, and clinical trial documents. A hospital system running an air-gapped AI can process patient records, generate clinical note summaries, and extract data from lab reports without any data leaving the hospital network. This eliminates entire categories of breach risk and simplifies the compliance narrative for auditors.

Financial Institutions

Investment banks processing M&A due diligence documents, insider information, and trading strategies face strict information barriers. An air-gapped AI system in the deal room can analyze thousands of documents, surface risk factors, and generate summaries without any data crossing the Chinese wall. Similarly, regulatory compliance teams use offline AI to review years of communications for potential violations, a task where data leakage could trigger SEC enforcement actions.

Critical Infrastructure

Energy companies, water utilities, and transportation operators increasingly use AI to analyze operational documents, maintenance logs, and incident reports. These organizations often maintain strict OT/IT segmentation (operational technology separated from information technology networks). An air-gapped AI deployment that lives on the OT side can process operational documents without creating a network bridge that violates segmentation requirements.

Legal Discovery and Privileged Documents

Law firms handling large-scale discovery or privileged attorney-client communications need AI that processes documents without exposing them to any third party. An air-gapped system can ingest a million-page production set, build a searchable index, and answer questions about the documents, all within the firm's security perimeter. For more on how private AI serves legal teams, see our guide to private AI adoption in the enterprise.

Keeping an Air-Gapped System Current: Updates, Patching, and Model Refresh

The biggest operational challenge with air-gapped deployments is not the initial installation; it is keeping the system current. Without internet connectivity, every update requires deliberate, physical intervention. Organizations that plan for this upfront avoid the common failure mode of deploying a system and letting it slowly become outdated and insecure.

The Physical Media Update Cycle

Updates are packaged on encrypted drives (typically AES-256 encrypted USB thumb-drive or portable NVMe) and delivered through secure courier or hand-carry by cleared personnel. Each update package includes:

• Model weights: New or updated LLM and embedding model files
• Software patches: Container image updates for the application stack
• Security fixes: OS-level CVE patches, library updates, and dependency upgrades
• Configuration updates: Tuned parameters, prompt templates, or new document processing rules

Each package is cryptographically signed by the vendor. The receiving system verifies the signature and computes checksums before applying any updates. If verification fails, the update is rejected and the incident is logged for investigation.

Model Versioning Without Internet

Air-gapped environments cannot download model updates from Hugging Face or pull container images from Docker Hub. Instead, model versions are tracked in a local registry. Best practices include:

• Side-by-side deployment: Install the new model alongside the existing one. Run validation tests against a known document set before switching production traffic.
• Rollback capability: Keep the previous model version available for immediate rollback if the new version introduces regressions.
• Validation suite: Maintain a set of test documents with expected outputs. Run automated comparisons after each model update to catch accuracy changes.

Recommended Update Cadence

• Security patches: Monthly, or as critical CVEs are published
• Model refresh: Quarterly, aligned with open-source model release cycles
• Full platform updates: Semi-annually, including major version upgrades and new capabilities

Evaluating Air-Gapped AI Vendors: 10 Questions to Ask

Not every vendor claiming "on-premise" support can actually deliver a fully air-gapped deployment. These questions separate vendors with genuine offline capability from those offering connected on-premise with a marketing overlay.

1. How is the software delivered? Look for encrypted physical media delivery (USB dongle, portable drives). If the vendor says "download from our portal," the product was not designed for air-gapped environments.
2. Does the software phone home? Any licensing check, telemetry ping, or automatic update mechanism that requires internet connectivity disqualifies the product from air-gapped use. Ask for a network traffic analysis or architecture diagram showing zero outbound connections.
3. Which LLM models are included? The vendor should specify exactly which models ship with the product (e.g., Llama 3 8B, Llama 3 70B, Mistral). Ask about model licensing terms for offline use.
4. What is the complete software stack? You need the full list: inference engine, embedding model, vector database, OCR pipeline, document parser, and orchestration layer. Each component must run locally.
5. What hardware is required? The vendor should provide specific GPU, RAM, and storage requirements for each supported model size. Vague answers like "a modern server" indicate the product has not been validated for air-gapped deployment.
6. How are updates delivered? Expect a defined process: encrypted media, cryptographic signing, checksum verification, and rollback capability. Ask about update frequency and what each update includes.
7. What compliance certifications apply? Look for SOC 2 Type II, and ask specifically about ITAR, CMMC, or FedRAMP applicability. Certifications for the vendor's cloud product do not automatically extend to their air-gapped offering.
8. How is support provided? In a disconnected environment, remote support tunnels do not work. Ask about on-site support procedures, documentation quality, and support hours for cleared facilities.
9. Can cleared personnel install the system? For SCIF and classified deployments, your cleared staff may need to perform the installation. The vendor should provide detailed installation guides that your team can execute independently.
10. What is the pricing model? Air-gapped deployments typically use site licensing (annual or perpetual) rather than per-user or per-token pricing. Clarify what the license covers: software only, software plus updates, or software plus updates plus on-site support.

Zedly's self-hosted AI platform supports all three deployment models: private cloud (VPC), on-premise (Docker/Helm), and fully air-gapped (offline delivery via physical media). The air-gapped configuration includes local Llama 3 inference, local embedding models, self-hosted vector storage, and Apache Tika/Tesseract for document processing, with zero internet dependency.

Getting Started: From Evaluation to Deployment

Most organizations follow a phased approach to air-gapped AI deployment:

1. Pilot on connected infrastructure. Start with a VPC or connected on-premise deployment to validate the platform against your document types and use cases. This proves value before investing in the air-gapped configuration.
2. Validate compliance requirements. Work with your compliance team to confirm which regulatory framework applies and whether true air-gapping is required or recommended. Document the decision for auditors.
3. Procure hardware. Order GPU servers that meet the vendor's specifications. Factor in 8-12 week lead times for enterprise GPU hardware.
4. Plan the update cycle. Before deployment, establish the physical media delivery process, designate cleared personnel for updates, and budget for ongoing maintenance.
5. Deploy and test. Install the air-gapped system, run the validation suite against known documents, and confirm that no network connectivity exists. Have your security team verify the air gap independently.

The organizations that succeed with air-gapped AI treat it as an operational commitment, not a one-time installation. The technology is mature, the open-source model ecosystem is strong, and the compliance benefits are clear. What separates successful deployments from abandoned ones is the planning that happens before the hardware arrives. For a broader look at the commercial landscape and a buyer's checklist covering stack components, vendor red flags, and compliance mapping, see our self-hosted document AI buyer's guide.