For many organisations, the journey into artificial intelligence begins with a browser. An employee opens a public AI service, enters a question and receives an answer — there is no infrastructure to provision, no model server to operate and, apparently, very little architecture to think about. Then the experiment becomes useful: the business wants the AI system to search internal documents, a development team wants to expose a model through an API, customer-support staff want it to work with a knowledge base, someone proposes connecting it to the CRM, another team wants to process confidential reports.
At that point the question changes. It is no longer simply “which AI model should we use?” — it becomes “where should this system run, what should it be allowed to access, who should be allowed to use it, and how do we operate it safely?” This is the point at which organisations often start talking about “private AI.” Unfortunately, the term is frequently used to describe several quite different things. Running an open-weight language model on a GPU does not, by itself, create a private AI platform — a production environment involves considerably more than the model. This article explains the infrastructure behind one.
1. Start with the workload, not the GPU
A common mistake in private AI projects is beginning with hardware. “How many GPUs do we need?” sounds like an infrastructure question, but it is usually several questions disguised as one. Before selecting compute, an organisation needs to understand which model or models it expects to run, how large those models are, expected context lengths, how many simultaneous users or applications will access them, acceptable response latency, expected request volume, whether workloads are interactive or batch-based, availability requirements, whether fine-tuning is required, whether retrieval-augmented generation is involved, what other applications must communicate with the model, and how usage is expected to grow.
Inference infrastructure is therefore better designed backwards from the workload. A 7–8 billion parameter model serving a small internal team is a very different infrastructure problem from a large model exposed to hundreds of concurrent users. Even apparently simple metrics such as “requests per minute” can hide important differences — a request containing several hundred input tokens and requesting a short response places a different demand on the system from a request carrying tens of thousands of tokens of context. The first design principle is consequently simple: size the infrastructure for the workload rather than selecting hardware and hoping the workload fits it.
2. The model server
Once a model is selected, something has to serve it. In production environments, organisations generally do not want every application implementing its own model-loading logic — instead, models are commonly exposed through an inference service. One widely used open-source example is vLLM, which includes mechanisms intended to improve serving efficiency, including PagedAttention-based memory management, continuous batching, prefix caching, chunked prefill and support for several quantisation formats.
These are not merely performance curiosities. They illustrate why production inference is different from loading a model into memory on a workstation: a serving system must deal with multiple requests arriving at different times, different context lengths, competing memory requirements and the need to use expensive accelerator capacity efficiently. Continuous batching, for example, allows a serving engine to combine work from multiple requests rather than treating each request as an entirely isolated batch, which can significantly change the economics of shared inference. The model server effectively becomes the engine room of the AI service — but it should not necessarily be exposed directly to every application. (vLLM documentation)
3. The API layer
A mature AI environment may eventually contain multiple models — perhaps one appropriate for general conversation, another for code, another for embeddings and another for a specialised workflow. Applications should not necessarily need to know where every model lives. An API or gateway layer can provide a consistent interface between applications and the underlying inference services, offering authentication, model routing, quotas, usage accounting, rate limiting, request logging, fallback behaviour and centralised policy.
The architecture begins to look less like “application → model” and more like “application → AI gateway → inference service → model.” That distinction becomes important when an organisation has more than one AI application — without a common service layer, each development team can end up building its own miniature AI platform.
4. Retrieval-augmented generation changes the architecture
One of the most useful enterprise applications of language models is allowing them to work with organisational knowledge. A model’s pretrained knowledge does not automatically include a company’s latest policies, contracts, engineering documents, internal procedures or customer records. Retrieval-augmented generation addresses this by retrieving relevant information from an external knowledge source and supplying that information as context to the model: documents are parsed, chunked, embedded, indexed, retrieved and finally supplied as context alongside the user’s question.
This creates several additional infrastructure components, and it also creates additional security questions. Who is allowed to retrieve which document? Does a user who can access the AI interface automatically gain access to everything indexed by the RAG system? How are deleted documents removed from the retrieval index? How are permissions propagated when a document’s access controls change? Can retrieved documents themselves contain malicious instructions? These are not theoretical concerns — OWASP’s current guidance identifies vector and embedding weaknesses as a distinct class of security risk for systems using RAG, while its prompt-injection guidance discusses retrieval poisoning, in which malicious content is introduced into sources used by the AI system. (OWASP GenAI Top 10, OWASP Prompt Injection Prevention Cheat Sheet)
5. Authentication is not optional
An internal AI system should generally know who is using it. This sounds obvious, yet prototype systems frequently begin as an unauthenticated web interface or API because the initial objective is simply to demonstrate that the model works. Production changes the requirement — identity enables the system to answer who made this request, which models this person may use, which documents they may retrieve, which tools they may invoke, how much capacity they may consume, and what should be recorded in an audit log. Ideally, AI services should integrate with an organisation’s existing identity system rather than establishing another disconnected set of usernames and passwords.
This becomes particularly important with RAG. Authentication answers “who are you?”; authorisation answers “what are you allowed to access?” A RAG system that authenticates everybody but retrieves information from an unrestricted corporate document index has solved only half the problem.
6. Prompt injection changes the traditional trust boundary
Traditional applications generally distinguish between code and data. Generative AI complicates that distinction because natural-language data can influence model behaviour. An AI application may ingest a user’s prompt, a web page, an email, a PDF, a retrieved document, API results, or another model’s output — any of which may contain instructions that influence the model. OWASP currently places prompt injection at the top of its LLM application risk list and distinguishes both direct and indirect forms of attack; an indirect prompt injection occurs when malicious instructions are contained in material the model later reads rather than entered directly by the user.
That has major implications for RAG and agentic systems. A document retrieved from a knowledge base should be treated as untrusted content, not as an extension of the system prompt — particularly important if the AI has access to tools. A model that can merely produce text has one risk profile; a model that can send email, query databases, modify tickets, call APIs, create files or initiate business processes has another. OWASP describes excessive agency as a separate risk because granting models excessive functionality, permissions or autonomy can amplify the consequences of manipulation or mistakes. Private hosting does not remove these risks — it changes where the system runs. Application security still matters. (OWASP GenAI Security Project)
7. Sensitive information remains sensitive
Another misconception is that privately hosting a model automatically solves information-governance problems. It can provide significantly greater control over where processing occurs and how infrastructure is operated, but organisations still need to decide what information the system should process. The UK’s Information Commissioner’s Office states that where AI systems process personal data, data-protection requirements apply, and its AI guidance covers lawfulness, transparency, fairness, security, data minimisation and individual rights — specifically noting that AI can exacerbate existing security risks and make them harder to manage.
A private AI project should still ask whether it needs the personal data it’s processing, what the lawful basis for processing it is, how long prompts and responses should be retained, whether logs are capturing information they don’t need, who can access conversation histories, what data enters embeddings or vector indexes, whether information can be deleted where required, and whether development and test environments are using production information unnecessarily. Private infrastructure can provide the technical environment in which these controls are implemented; it does not replace the controls themselves. The ICO’s AI guidance is currently under review following changes introduced by the Data (Use and Access) Act 2025, so organisations should check the current regulatory position when designing or reviewing systems rather than relying on an old implementation checklist. (ICO — Artificial Intelligence, ICO — Security and data minimisation in AI)
8. Containers are useful. Kubernetes is not compulsory.
AI infrastructure discussions have a habit of becoming Kubernetes discussions, and Kubernetes can absolutely be useful — it provides mechanisms for scheduling workloads, service discovery, scaling, configuration, resilience and managing containerised applications, and it supports scheduling specialised hardware such as GPUs through device plugins, which Kubernetes documents as stable support for schedulable resources exposed by vendor device plugins.
But Kubernetes introduces operational complexity of its own. For a small deployment consisting of one model server, one interface and a handful of supporting services, a well-designed container environment may be entirely sufficient. For a larger environment involving multiple GPU nodes, several inference services, development and production environments, multiple applications, rolling deployments, complex availability requirements and dynamic scheduling, Kubernetes becomes much more compelling. The correct principle is not “AI should run on Kubernetes” — it is to use an orchestration layer proportional to the system being operated. Adding Kubernetes to a three-container application purely so the architecture diagram looks sophisticated is not good engineering. (Kubernetes — Scheduling GPUs)
9. Observability matters more than people expect
When a traditional web application becomes slow, engineers inspect CPU, memory, database performance and request latency. AI workloads introduce additional dimensions worth tracking: requests per second, time to first token, tokens generated per second, prompt length, output length, queue depth, GPU utilisation, accelerator memory utilisation, cache utilisation, request failures, model-loading failures, retrieval latency, vector-search latency, API latency and application-level success rates.
Cost should also be observable — an AI system can remain technically healthy while becoming economically inefficient. Unnecessarily long context windows, poorly controlled agent loops or inefficient routing to expensive models can increase resource consumption substantially, and OWASP’s current LLM risk taxonomy includes “unbounded consumption,” reflecting the potential for uncontrolled model interactions to cause excessive resource use or denial of service. Production AI therefore needs both technical and economic observability.
10. Backups still matter
AI can make infrastructure feel new. Most infrastructure principles are not. Databases still need backups, configuration still needs version control, secrets still need protection, recovery still needs testing, RAG knowledge stores may need rebuilding or restoring, and application state still matters. Infrastructure definitions should ideally be reproducible: a model can often be downloaded again, but the organisation’s configuration, permissions, prompts, application code, retrieval pipeline and business data may be much harder to reconstruct. Disaster recovery should therefore distinguish between replaceable artefacts and irreplaceable state.
11. Model supply chains need attention
Open models provide extraordinary flexibility, but downloading a model from the internet is still a supply-chain decision. AI environments can depend on model weights, Python packages, container images, CUDA libraries, inference frameworks, embedding models, plugins, extensions and external datasets, and OWASP identifies supply-chain risk as one of the major security categories for generative-AI applications. Normal software supply-chain practices therefore remain relevant: controlled sources, dependency management, image scanning, vulnerability management, version pinning where appropriate, change control and maintaining an inventory of deployed components. The fact that the largest file in the stack happens to contain model weights does not make conventional security engineering obsolete.
12. A practical private AI architecture
A moderately sophisticated private AI environment therefore contains a request path — users and business applications, through an identity provider, a reverse proxy or API gateway, an AI gateway, inference services and finally GPU compute — running alongside a separate retrieval path, where document sources feed an ingestion pipeline, an embedding service, a vector or search store and a RAG service. Surrounding all of it: network controls, secrets management, monitoring, logging, backups, vulnerability management, access controls, deployment automation and operational support. That is why “we installed an LLM on a GPU” and “we operate private AI infrastructure” are not the same statement.
13. Private AI is an architectural choice, not a product category
There is also no reason every organisation must choose between completely public and completely private AI — hybrid architectures are entirely legitimate. A company might use a commercial AI API for low-risk general tasks, operate an internal model for sensitive document retrieval, use a specialist external model for coding, and maintain a private embedding and retrieval environment for proprietary information. The correct architecture depends on the workload, information and risk. NIST’s Generative AI Profile similarly treats generative AI as a risk-management problem across the lifecycle rather than implying that one deployment model eliminates the risks. (NIST AI RMF, NIST Generative AI Profile)
14. The important question is not “can we self-host AI?”
Of course you can — the open-source ecosystem has made it remarkably easy to demonstrate. The harder question is: can we operate it reliably? That means understanding capacity, security, identity, data access, model lifecycle, monitoring, backups, upgrades, vulnerability management, recovery, cost and user support. For some organisations, operating that platform internally makes complete sense. Others will conclude that they want control over their AI environment without creating another infrastructure discipline for their internal team to own. That is the gap managed private AI infrastructure is intended to fill.
How Polar approaches it
At Polar Networks, we think private AI infrastructure should begin with the workload rather than a catalogue of GPU instances. Tell us what you are trying to run, what information it needs to work with, where it needs to be hosted, who will use it and what availability you need — the infrastructure specification comes afterwards. Polar provides managed hosting and infrastructure engineering with UK and EU hosting options, and where appropriate that can include the compute, networking, model-serving and supporting infrastructure required to operate private AI applications. You build the use case. We can run the infrastructure underneath it.
References
1. OWASP GenAI Security Project — Top 10 Risks for LLM and Generative AI Applications 2. OWASP Cheat Sheet Series — LLM Prompt Injection Prevention 3. Information Commissioner’s Office — Artificial Intelligence: UK GDPR Guidance and Resources 4. Information Commissioner’s Office — How should we assess security and data minimisation in AI? 5. Kubernetes — Schedule GPUs 6. vLLM Project — Documentation 7. NIST — AI Risk Management Framework: Generative AI Profile