Use case · AI & GPU compute
LLM hosting
Hosting an LLM means running a very large model in production, which is dominated by memory — the model weights plus the KV cache that grows with context length and concurrency. Large models often need high-memory or multiple GPUs. Self-hosting keeps prompts and outputs private and, in the EU, under European jurisdiction.
Key points
- LLM hosting is memory-dominated: model weights plus a KV cache that grows with context and concurrency.
- Large models often exceed one GPU, needing high-memory GPUs or splitting the model across several.
- Serving is measured in tokens per second and time to first token, improved by continuous batching.
- Self-hosting keeps prompts and outputs on hardware you control, rather than sending them to an API.
- Prompts are often an organisation's most sensitive data, which makes EU jurisdiction matter.
What does hosting an LLM actually require?
Large language models are among the most resource-hungry workloads in production computing, and hosting one to serve requests is a substantial undertaking. The defining characteristic is scale: LLMs have very many parameters, so the model itself occupies a great deal of memory — often tens or hundreds of gigabytes — before any requests are served. On top of that, serving the model to real users adds further memory and compute demands that grow with how much text the model handles and how many requests it serves at once. Hosting an LLM therefore means providing hardware with the memory to hold a large model and the capacity to serve it, which is a higher bar than most workloads.
Because of this, LLM hosting is dominated by memory more than almost any other consideration. Whether a given GPU or server can host a particular model depends first on whether the model, plus the memory needed to serve it, fits; if it does not, the model must be shrunk, split across multiple GPUs, or run on larger hardware. The rest of the requirements — throughput, latency, availability, cost — matter greatly, but they come after the fundamental question of fitting the model and its serving memory into the hardware. Understanding LLM hosting starts with understanding this memory demand and what drives it.
Memory: model weights and the KV cache
LLM memory use has two main parts. The first is the model weights — the parameters of the model — which occupy a fixed, often large amount of memory whenever the model is loaded, and which set a floor on the hardware needed. The second, less obvious part is the KV cache: as the model processes and generates text, it keeps intermediate state for the tokens in context, and this state grows with the length of the context and with the number of requests being served at once. The KV cache is a significant and variable memory cost on top of the weights, and it is often what determines how many concurrent requests, or how long a context, a given amount of memory can support.
This matters because it means memory planning for an LLM cannot stop at the model's size. A model whose weights fit comfortably can still run out of memory under many concurrent requests or long contexts, because the KV cache for all those requests and tokens consumes memory beyond the weights. Longer context lengths and higher concurrency both increase KV-cache memory, so hosting an LLM to serve many users with long contexts needs more memory than the weights alone suggest. Getting LLM hosting right means sizing memory for the weights and the expected KV cache together, since the KV cache is frequently the constraint on how much the hosted model can actually serve.
Fitting large models: quantisation and multi-GPU serving
When a model is too large to fit on a single GPU, there are two main ways to host it. The first is quantisation — representing the model at lower numerical precision — which reduces its memory footprint, often substantially, and can let a model that would not otherwise fit run on a given GPU, or run more efficiently, at some cost in precision that is acceptable for many uses. Quantisation is a common and effective way to make large models more tractable to host, and it directly affects which hardware a given model needs.
The second is splitting the model across multiple GPUs, so that a model too large for one GPU's memory is divided among several that together hold it. This model or tensor parallelism lets very large models be served on multi-GPU servers, with the GPUs cooperating to run the single model — which is why LLM hosting often uses servers with several high-memory GPUs and fast interconnect between them, so the GPUs can work together without the communication between them becoming a bottleneck. Between quantisation and multi-GPU serving, even very large models can be hosted, but both shape the hardware required: quantisation reduces the memory needed, and multi-GPU serving demands servers built with the right number of GPUs connected well enough to cooperate.
Throughput and latency for LLM serving
LLM serving performance is usually described in terms of tokens: how many tokens per second the model generates, and how quickly it produces the first token of a response. For interactive uses, the time to the first token shapes how responsive the model feels, and the generation speed determines how quickly a full answer appears; for bulk processing, the total tokens per second across all requests matters most. These measures capture the throughput and latency of LLM serving, and hosting an LLM well means delivering acceptable values for whichever matters to the application.
A large influence on serving efficiency is how requests are batched. Continuous batching — dynamically grouping the ongoing generation of many requests so the GPU processes them together — greatly improves the throughput of LLM serving by keeping the GPU busy across concurrent requests, rather than handling them one at a time with the GPU underused. Serving frameworks built for LLMs implement this and related techniques, such as efficient management of the KV cache, to raise the number of requests a given GPU can serve. The practical result is that how an LLM is served, not just the raw hardware, has a large effect on the throughput achieved, so effective LLM hosting pairs capable hardware with a serving approach that uses it well.
Self-hosting an LLM versus using a hosted API
A fundamental choice in LLM deployment is between self-hosting an open model on your own infrastructure and calling a hosted LLM API run by a provider. A hosted API is convenient: someone else runs the model, and you send requests and receive responses without operating any GPUs. But that convenience carries trade-offs. You send your prompts — and receive outputs — to and from a third party, placing your data and the interaction in their hands; you depend on their availability, pricing, and terms; and you are limited to the models they offer. Self-hosting, by contrast, means running an open model on hardware you control, which requires operating the infrastructure but keeps the model, the data, and the terms in your hands.
The honest way to choose is by what you value. If low operational effort and access to a provider's proprietary models matter most, and sending your data to that provider is acceptable, a hosted API may suit you well, and we will not pretend otherwise. If control over the model, predictable cost at scale, and — above all — keeping your prompts and outputs on infrastructure you control matter more, self-hosting an open model is the way to get them, and it is what we host. Many organisations reach for self-hosting precisely when their prompts become sensitive enough that sending them to an external API is no longer acceptable. That is the case we are built for, though we will say plainly that for casual use or access to specific proprietary models, an API may be the more sensible choice.
Data sovereignty and privacy for prompts and outputs
With LLMs, the prompts sent to the model and the outputs it returns are frequently among the most sensitive data an organisation handles. Prompts can contain confidential documents, personal data, internal information, customer data, and more, because people put whatever they are working on into the model; the outputs derived from them can be equally sensitive. Where these prompts and outputs are processed, and who can access them, is therefore a serious data-protection and confidentiality question — arguably more acute for LLMs than for many other workloads, because of how much sensitive material passes through them.
This is where self-hosting in the EU offers something a hosted API cannot. When you self-host an open model, the prompts and outputs stay on your infrastructure rather than being sent to an external provider; and when that infrastructure is in the EU, the data stays under European jurisdiction. VV Internet Hosting is incorporated in the Netherlands, within the EU, so an LLM self-hosted with us processes its prompts and outputs under European law and outside the direct reach of the US CLOUD Act. For organisations whose prompts are sensitive or regulated — which is many of them, given what people put into these models — self-hosting an LLM in the EU keeps that data private and sovereign in a way that sending it to an external API does not. This is one of the strongest reasons to self-host, and it is central to what we offer.
Cost and utilisation for LLM serving
Hosting an LLM uses expensive, high-memory GPUs, so the cost of LLM hosting is significant, and keeping that cost reasonable depends heavily on utilisation. A GPU serving a steady stream of LLM requests is doing valuable work for its cost; a GPU held for occasional requests, mostly idle, is expensive for what it delivers. Because LLM-capable GPUs are costly, the economics of LLM hosting reward keeping them well-utilised — serving enough requests to justify the hardware — and punish provisioning large GPUs that then sit idle.
For steady production LLM serving, this favours dedicated GPU servers, on the same logic as other sustained GPU workloads. An LLM service with consistent, ongoing demand keeps its GPUs busy, and dedicated hardware at a fixed cost, well-utilised, is more economical than on-demand rates for GPUs that are busy most of the time. Where demand is spiky, elastic capacity has a role for the variable portion; but the expensive, memory-heavy GPUs that LLMs need are exactly the hardware where paying an on-demand premium for continuous use adds up fastest, so a steady baseline is usually best served by dedicated hardware. We will size an LLM deployment to its actual demand and keep the costly GPUs working, rather than provisioning more than the load justifies.
Reliability and availability for production LLM services
An LLM serving production traffic is a user-facing service, so its availability matters as much as its speed. If the LLM service is unreachable, whatever depends on it stops working, and users are affected immediately — the same availability requirement that applies to any production inference applies to hosted LLMs, with the added complication that the hardware involved is expensive and heavily loaded. Production LLM hosting therefore has to be built to keep serving despite failures, with redundancy and failover so that a component failure does not take the service down.
Achieving this means applying resilient-infrastructure practices to the demanding case of LLM serving: redundant capacity so a failure is covered, traffic routed to healthy capacity via health checks and distribution, and hosting on reliable hardware in a datacenter with dependable power and cooling — the last especially relevant given how much power LLM GPUs draw. The goal is an LLM service that stays available and responsive under real production conditions, because a capable hosted model that is often unreachable serves users poorly. When we host LLM infrastructure, availability is treated as part of the deliverable alongside the raw serving performance, since production LLM services are judged on both.
Where VV Internet Hosting fits — and where it does not
We host dedicated infrastructure for self-hosting LLMs: high-memory GPU servers, single or multi-GPU with fast interconnect for models that must be split, sized to the model, its context length, and its concurrency, in EU datacenters under European jurisdiction. This suits organisations self-hosting open models under steady production load, and — especially — those for whom the privacy and sovereignty of prompts and outputs are decisive, since keeping that data on EU infrastructure you control is the core reason to self-host. For those cases we are a strong fit, and we will size the GPUs and memory to the real workload, including the KV cache the concurrency and context demand.
We are candid about where we are not the answer. If you want the convenience of a hosted API and access to a provider's proprietary models, and sending prompts to that provider is acceptable to you, an API may suit you better than self-hosting, and we will say so. If your LLM use is occasional and bursty enough that keeping expensive GPUs on standby makes no sense, on-demand capacity may fit better. And if you need a hyperscaler's managed model-serving platforms rather than infrastructure you operate, that is a different kind of provider. We are for self-hosted, sovereignty-aware LLM infrastructure on hardware you control, under steady load — which is exactly right for organisations that need their prompts kept private and European, and not the right tool for every LLM use.
Related use cases
Questions
LLM hosting, answered plainly
Common questions about hosting for LLM hosting.
Why is LLM hosting so memory-intensive?
Two reasons. The model weights occupy a large, fixed amount of memory — often tens or hundreds of gigabytes. On top of that, the KV cache, the intermediate state kept while processing and generating text, grows with context length and the number of concurrent requests. A model whose weights fit can still run out of memory under long contexts or high concurrency, so both must be sized for.
Should I self-host an LLM or use a hosted API?
It depends on what you value. A hosted API is convenient and gives access to proprietary models, but you send prompts and outputs to a third party. Self-hosting an open model keeps the model and data on infrastructure you control, at the cost of operating it. Organisations often self-host when their prompts become too sensitive to send to an external API.
How does self-hosting an LLM help with data privacy?
Prompts and outputs are often an organisation's most sensitive data, since people put confidential and personal material into the model. Self-hosting keeps that data on your own infrastructure rather than sending it to an external provider — and self-hosting in the EU keeps it under European jurisdiction, outside the direct reach of the US CLOUD Act. This is a leading reason to self-host.
Planning LLM hosting infrastructure?
We host dedicated, EU-sovereign infrastructure sized to your workload — and we will tell you plainly when something else fits better. Tell us what you're building.