Private AI is often described as though it were an installation choice: download an open model, put it on a GPU, start an inference server, add a chat interface — you now have private AI. Technically, perhaps. Operationally, almost certainly not. There is an enormous difference between getting a large language model to produce an answer on a server and operating an AI service that a business can safely depend upon.

As organisations move from experimenting with generative AI to connecting it to internal documents, applications, databases and workflows, the model becomes only one component of a much larger system. Authentication matters. Network architecture matters. Data storage matters. Retrieval matters. Logging, monitoring, backups, patching, capacity planning and security boundaries all matter. Increasingly, organisations need to know not simply which model they are running, but where the entire system runs, which other systems it can communicate with, what information is retained and who can access it.

The UK government’s AI Cyber Security Code of Practice reflects this broader view. Its principles extend well beyond protecting a model: organisations are advised to secure infrastructure and supply chains, document data, models and prompts, conduct testing, monitor system behaviour, maintain security updates and manage data and model disposal. The practical lesson is simple — a model is not an AI platform. This article examines what actually sits around a privately operated model, why each component exists, and where organisations need to make architectural decisions before putting private AI into production.

1. Start with the workload, not the GPU

One of the easiest mistakes in private AI projects is beginning with hardware. “How many GPUs do we need?” sounds like the obvious infrastructure question. It often isn’t. Before selecting hardware, the organisation should be able to describe what it wants the AI system to do.

Consider four superficially similar requirements. An internal assistant for fifty employees needs a chat interface for summarisation, drafting and general assistance. A company knowledge assistant lets those same employees ask questions about thousands of internal documents. An AI-enabled application might need to make several hundred model requests per minute through an API. An automated business workflow receives documents, extracts information, queries internal systems and initiates actions. All four might use the same underlying model family, yet their infrastructure requirements are radically different: the internal assistant might prioritise simplicity, authentication and predictable interactive performance; the knowledge assistant needs a retrieval architecture; the application requires an API layer, concurrency planning, rate controls and potentially horizontal or multi-GPU scaling; and the automated workflow introduces considerably greater security questions because the model may be able to initiate actions rather than merely generate text.

That last distinction is particularly important — OWASP’s work on generative-AI application security identifies risks including prompt injection, sensitive-information disclosure, supply-chain vulnerabilities and excessive agency. Infrastructure design should therefore begin with questions about who will use the system, how many simultaneous users are expected, whether the workload is interactive or batch, what information it will process, whether that information is personal, commercially sensitive or regulated, whether the model only produces text or can initiate actions, what availability the business requires, where the organisation wants data and workloads to reside, and what happens if the service is unavailable. Only then does hardware sizing become meaningful.

2. The model serving layer

At the centre of a private generative-AI environment is normally an inference or model-serving layer, whose job is to take requests, execute inference against a model and return results. A production inference server does considerably more than simply load model weights into GPU memory — it needs to handle requests from multiple users, schedule work efficiently, manage memory and provide an interface through which applications can communicate with the model.

One widely used open-source example is vLLM, whose current documentation provides an OpenAI-compatible HTTP API, allowing applications designed around the familiar OpenAI API format to communicate with privately hosted models. That compatibility can be strategically useful: it separates, at least partially, the application from the underlying inference provider, so instead of an application being written exclusively around one commercial model API, an organisation can place a compatible inference endpoint behind it. That doesn’t make models interchangeable — different models have different capabilities, tokenisation, context limits and behaviour — but it can reduce unnecessary coupling between application code and infrastructure.

Even a relatively simple architecture of business application, AI gateway, inference server, model and GPU infrastructure raises operational questions. What happens when ten requests arrive simultaneously — what about a hundred? Should requests queue, and should different applications receive different priorities? What prevents a runaway application from consuming all available inference capacity? How is latency measured, and how do you know when additional GPU capacity is required? The moment a model becomes a shared business service, inference becomes a capacity-management problem.

3. The AI gateway

As organisations deploy more AI applications, connecting every application directly to every model becomes difficult to manage. An intermediary gateway — sitting between multiple applications (a CRM assistant, internal chat, a document tool, a customer app) and the underlying models — can provide a common control point, offering authentication, request routing, quotas, model selection, observability and policy enforcement, and creating an abstraction between applications and individual models.

That becomes increasingly useful when an organisation operates several models: a smaller model might handle simple classification or extraction tasks, a larger model might be reserved for complex reasoning, a specialist embedding model might support document retrieval, and some requests might even legitimately be routed to an external commercial API while sensitive workloads remain on privately controlled infrastructure. The important architectural principle is not that every AI environment requires a particular gateway product — it is that model access should be deliberately controlled. The alternative, applications and developers connecting directly to inference endpoints with inconsistent credentials and controls, becomes increasingly difficult to govern as usage grows.

4. Identity is part of the AI architecture

A private model does not automatically mean a private system. Imagine an organisation successfully deploys a model entirely within its own environment but exposes a web interface with weak authentication — the model might be privately hosted, but the service isn’t meaningfully private. Production AI therefore needs the same identity fundamentals expected from other business systems: central identity-provider integration, multi-factor authentication, role-based access control, application and service identities, API credentials, credential rotation, session controls, and separation between administrative and ordinary users.

Identity becomes even more important when AI systems can access internal knowledge. Suppose an employee asks the assistant to summarise the latest acquisition documents — the question is not merely whether the AI can find those documents, but whether that employee was authorised to see them in the first place. RAG and AI search systems can accidentally create a new route around existing information boundaries if permissions are not considered in the retrieval architecture. Private AI therefore isn’t simply about keeping information inside a particular data centre — it is also about maintaining appropriate access boundaries inside the organisation.

5. RAG adds an entire data architecture

One of the most compelling business applications for private AI is allowing employees to ask questions about organisational knowledge — policies, technical documentation, contracts, product manuals, research, support material, internal procedures. This is commonly implemented using retrieval-augmented generation, or RAG, which does not usually mean simply feeding every company document into an LLM. Instead, a retrieval pipeline identifies information relevant to the user’s question and supplies appropriate context to the model: documents are parsed, chunked, embedded and placed into a vector-capable datastore, and when a question arrives, it’s embedded, matched against relevant document sections, and used to construct a prompt before reaching the model.

Current cloud reference architectures illustrate how substantial this surrounding infrastructure can become — Google’s own architecture guidance, for example, separates application, model, retrieval and data-store components in production RAG systems. The specific services aren’t what matters here; the general lesson is that RAG is a system, not a feature toggle. The retrieval layer introduces its own operational questions: how are new documents ingested, what happens when documents change, how quickly are deleted documents removed from the index, how are access permissions represented, how are document versions handled, how do you prevent one department retrieving another department’s restricted material, how do you evaluate retrieval quality, how do you back up the source data and indexes, and can the system show users which source material informed an answer? These questions frequently matter more to the usefulness of a business AI deployment than choosing between two models with slightly different benchmark scores.

6. Private does not mean secure by default

There is a dangerous assumption that sometimes appears in AI infrastructure discussions: “we’re self-hosting it, therefore it’s secure.” Self-hosting changes the security boundary — it does not eliminate the security problem. In fact, it transfers more responsibility to the organisation operating the environment. The UK’s AI Cyber Security Code specifically includes principles covering secure infrastructure, supply-chain security, testing, monitoring and regular security updates, and the ICO similarly notes that using AI to process personal data can affect an organisation’s security risk profile, with appropriate measures depending on the nature and level of risk presented by the processing.

A privately hosted AI environment may contain Linux hosts, containers, Kubernetes, GPU drivers, model-serving software, Python packages, model files, databases, object storage, web interfaces, reverse proxies, identity systems, APIs, vector databases and monitoring agents — every one of which creates an operational and potentially security-relevant dependency. Private AI therefore needs ordinary infrastructure security as well as AI-specific controls: patch management, network segmentation, credential management, least privilege, vulnerability management, backup testing, logging, configuration management, asset inventories, change control, incident response, and increasingly, AI-specific threat modelling.

7. Prompt injection changes the threat model

Traditional applications generally distinguish relatively clearly between instructions and data. LLM applications blur that boundary — a model consumes natural-language instructions, but natural-language content can itself contain instructions, creating the possibility of prompt injection. OWASP continues to identify prompt injection as a major generative-AI application risk, and the problem becomes especially important in RAG systems and agents: imagine an AI assistant asked to research information from documents, where one retrieved document contains hidden or malicious text instructing the model to ignore previous instructions and reveal information. The AI application has now consumed untrusted data that can potentially influence model behaviour, and the UK government’s AI Cyber Security Code explicitly identifies indirect prompt injection as one of the risks that distinguishes AI security from conventional software security.

This is one reason why giving an LLM unrestricted access to tools and business systems is dangerous. The infrastructure should assume model outputs are not inherently trustworthy — a model should not normally be able to decide, entirely by itself, that something “looks legitimate” and act on it unprompted by a genuine authorisation check. The more agency an AI system receives, the more important explicit permissions, constrained tools, validation and human approval become.

8. Network architecture still matters

AI has not abolished networking. A production private-AI environment still needs deliberate decisions about which components can communicate — typically, internet traffic passes through a reverse proxy or ingress, into an authenticated application, through an AI gateway, into an inference network and finally to GPU nodes, while a separate path connects the AI application to a retrieval service and internal knowledge datastore. Administrative access may travel through a separate management network or secure remote-access mechanism.

The principle is familiar: do not expose infrastructure unnecessarily. A GPU inference endpoint does not need to be directly accessible from the public internet merely because an application using it is public, and a model does not necessarily need unrestricted outbound internet access. Network controls can reduce the consequences of application or model-layer failures. Private AI should therefore be designed as infrastructure — not as a collection of Docker containers placed on a server and exposed until they work.

9. Data residency is not the same as data protection

Organisations frequently ask whether they can keep the AI in the UK, or run it entirely in the EU. Those are legitimate requirements, but data residency and data protection should not be confused. Hosting an AI workload in a particular jurisdiction tells you where infrastructure and data are located — it does not, by itself, tell you whether access controls are appropriate, whether data is encrypted, who can administer the environment, whether unnecessary personal information is being processed, what gets logged, how backups are handled, whether information is transferred elsewhere, or whether the application itself is secure.

The ICO emphasises both security and data minimisation when organisations use AI to process personal data. A sensible infrastructure conversation therefore distinguishes residency (where is the workload and data hosted?) from security (how is it protected?), privacy (what information is processed and for what purpose?) and governance (who is responsible and what controls apply?). Private hosting can make some of these requirements easier to control. It does not automatically satisfy them.

10. Logging an AI system requires care

Traditional infrastructure advice frequently says “log everything.” With AI, that can be dangerous — a prompt might contain confidential business information, a retrieved context could contain personal data, a response might contain commercially sensitive information. If every prompt and response is copied wholesale into a central logging platform, an organisation may accidentally create another sensitive datastore.

Observability still matters enormously — operators need to understand request volumes, error rates, inference latency, token usage, GPU utilisation, queue depth, application performance, failed authentication, unusual behaviour and system health. But content logging should be intentional: do we need complete prompts, how long should they be retained, who can access them, could metadata provide adequate operational insight without retaining content, and can sensitive fields be redacted? This is a good example of why conventional infrastructure practices sometimes need modification when applied to AI systems.

11. Monitoring needs to extend above the server

A GPU at 80% utilisation tells you something useful. It does not tell you whether your AI application is working properly. Private AI needs monitoring at several layers: infrastructure (CPU, memory, GPU memory, GPU utilisation, storage, networking and host health), platform (containers, inference servers, databases, queues, gateways and application availability), service (request latency, throughput, errors and model availability), and — depending on the use case — AI behaviour itself, including output quality, unexpected responses, retrieval failures and abnormal usage.

The UK AI Cyber Security Code explicitly recommends monitoring system behaviour and inputs to identify potential compromise and unexpected behaviour. That is an important shift: for conventional hosting, “the server is up” can sometimes be an acceptable first approximation of service health. For AI, a server that’s up but producing unusable responses is still an outage from the user’s perspective.

12. Backups are more complicated than model weights

Open models can often be downloaded again. Your organisation’s AI environment cannot necessarily be recreated so easily. What needs protecting may include application configuration, prompts and templates, gateway configuration, user configuration, databases, source documents, vector indexes, metadata, access-control mappings, fine-tuning data, adapters, custom models, automation workflows and infrastructure configuration.

This suggests a useful distinction between replaceable assets and unique organisational assets — a publicly available base model may be replaceable, while a year’s worth of curated knowledge, permissions and application configuration may not be. Recovery planning should prioritise accordingly, and as with conventional infrastructure, a backup that has never been restored is an assumption rather than a tested recovery capability.

13. Updates become an operational discipline

AI infrastructure evolves extremely quickly. Inference frameworks change, model versions change, CUDA and GPU software stacks change, containers change, Python dependencies change, security fixes appear, and models themselves may be superseded. The UK government’s AI Cyber Security Code recommends maintaining security updates, patches and mitigations throughout an AI system’s lifecycle — for production environments, this implies that “we got it working” is not an operational strategy.

Updates need testing. Model changes may need evaluation, because a newer model can behave differently even when the API remains compatible. Infrastructure updates can affect performance, and dependency changes can break applications. A sensible production environment therefore needs controlled change rather than continuous experimentation against the live service.

14. Capacity planning is different for AI

AI workloads introduce an unfamiliar resource constraint for many organisations: GPU memory. A model has to fit within the available memory architecture, but model size is only the beginning — concurrency, context length, quantisation, batching and cache requirements can all affect capacity and performance. That is why “which GPU runs model X?” is often the wrong question; the useful question is what infrastructure delivers the required latency and throughput for the expected workload at an acceptable cost. A system supporting five internal users is different from an API serving thousands of requests, and peak usage, context length, response length, service-level expectations and workload testing all matter. The objective should not necessarily be maximum benchmark performance — it should be adequate and predictable application performance at a sustainable cost.

15. Kubernetes is not automatically the answer

AI infrastructure discussions often progress quickly from “we need to host a model” to “we need Kubernetes.” Sometimes you do; sometimes you really don’t. Kubernetes can be extremely useful where organisations need orchestration, resilience, multiple services, deployment automation, scaling and consistent operational patterns — Google’s own current reference architecture demonstrates a RAG implementation using Kubernetes alongside open-source AI tooling — but that does not mean every private AI environment should begin there.

For a modest internal workload, a simpler containerised deployment on dedicated infrastructure may be easier to understand and operate. Complexity should solve a problem; it should not be adopted merely because it appears in enterprise reference architectures. A useful rule: use the simplest architecture that satisfies the actual availability, security and scalability requirements. The same principle applies to almost every component of private AI.

How Polar approaches it

At Polar Networks, we think private AI infrastructure should begin with the workload rather than a catalogue of GPU instances. Tell us what you are trying to run, what information it needs to work with, where it needs to be hosted, who will use it and what availability you need — the infrastructure specification comes afterwards. You build the use case. We can run the infrastructure underneath it.

References

1. UK Government, Department for Science, Innovation and Technology — AI Cyber Security Code of Practice 2. OWASP GenAI Security Project — Top 10 Risks for LLM and Generative AI Applications 3. vLLM Project — Documentation 4. Google Cloud — RAG reference architecture 5. Information Commissioner’s Office — How should we assess security and data minimisation in AI? 6. Google Cloud — Kubernetes RAG reference architecture