Use case · AI & GPU compute
AI inference
AI inference — serving trained models in production — is bound by latency and throughput rather than raw training power. It needs enough memory to hold the model, the ability to scale with demand, and high availability. Steady production load favours dedicated servers, and EU hosting keeps live user data under European jurisdiction.
Key points
- Inference serves a trained model in production, so latency and throughput matter more than training power.
- The hardware must hold the whole model in memory; model size often decides the GPU.
- Real-time inference optimises for latency; batch inference optimises for throughput.
- Production inference is user-facing, so availability matters more than it does in training.
- Inference processes live user data, which makes EU jurisdiction a real consideration.
How is inference infrastructure different from training?
Inference is what happens after a model is trained: the trained model is put to work, taking inputs and producing outputs — predictions, classifications, generated text, and so on. This is a different job from training, and it stresses infrastructure differently. Training is a heavy, sustained batch computation that runs for hours or weeks; inference is an ongoing production service that responds to requests, often continuously and often user-facing. Where training is judged by how quickly a run completes, inference is judged by how quickly it responds, how many requests it can handle, and how reliably it stays available.
These differences change what the infrastructure has to prioritise. Inference does not usually need the enormous multi-GPU interconnect that large training runs demand, because each inference is a smaller computation. Instead, inference infrastructure is shaped by the latency and throughput it must deliver, the memory needed to hold the model, the way load varies with demand, and the availability required of a production service. A machine ideal for training is not automatically ideal for inference, and specifying inference infrastructure means attending to this distinct set of constraints rather than assuming training hardware carries over.
Latency versus throughput: the core inference trade-off
The central tension in inference is between latency — how quickly a single request is answered — and throughput — how many requests are handled per unit of time. Interactive, real-time inference, where a user waits on the response, is dominated by latency: the model must answer quickly enough that the experience feels responsive, so the infrastructure is tuned to minimise the time each request takes. Batch inference, where many inputs are processed together without anyone waiting on each one, is instead about throughput: the goal is to get through as much work as possible, and grouping inputs to use the hardware efficiently is worthwhile even if each individual result takes a little longer.
Recognising which of these an inference workload needs is the first step in sizing it, because they pull in different directions. Optimising hard for latency can mean processing requests individually or in small groups, which uses the hardware less efficiently but responds fast; optimising for throughput often means batching requests together to keep the hardware busy, which is efficient but adds a little delay per request. Dynamic batching techniques try to balance the two, grouping requests that arrive close together to improve efficiency without adding much latency. The right configuration follows from whether users are waiting on each response or whether the work is bulk processing, and much of tuning inference infrastructure is about placing the workload correctly on this latency–throughput spectrum.
Choosing hardware for inference — and whether you need a GPU
As with training, the memory needed to hold the model is often the first constraint in inference: the model must fit in the memory of whatever runs it, so a large model needs hardware with enough memory, and the model's size frequently decides the hardware. But inference has more room than training to reduce these requirements. Techniques such as quantisation — running the model at lower numerical precision — can shrink a model's memory footprint and speed up inference, sometimes letting a model run on smaller, cheaper hardware than training it required, with an acceptable trade in accuracy for many uses.
Whether inference even needs a GPU depends on the model and the load. Large models, and workloads needing high throughput or low latency at scale, benefit from GPUs, which run the computation far faster. Smaller models, or lower request volumes where response time is less critical, can run acceptably on CPUs, avoiding the cost of GPUs entirely. The honest approach is to match the hardware to the model and the load rather than assuming inference must run on the same class of GPU used for training: some inference genuinely needs powerful GPUs, some runs happily on modest hardware, and right-sizing here has a direct effect on the cost of running the service.
Scaling inference with demand
Inference load usually tracks user demand, which varies — over the day, with usage patterns, with growth — so inference infrastructure has to handle a changing request rate. When demand rises, more capacity is needed to keep responses fast and avoid a backlog; when it falls, that capacity sits underused. How to provision for this depends on how predictable and how spiky the demand is. Steady, predictable load can be served efficiently by dedicated capacity sized to it; highly variable or unpredictable load raises the question of how to have enough capacity for peaks without paying for it during troughs.
The two broad approaches are provisioning dedicated capacity for the expected load and scaling capacity up and down with demand. Dedicated capacity, sized to steady production load, is efficient and predictable in cost when the load is fairly constant, which much production inference is. Elastic scaling, adding and removing capacity as demand changes, suits spiky or unpredictable load, trading some cost efficiency for the ability to follow demand. Many real deployments combine the two — a baseline of dedicated capacity for the steady load, with additional capacity for peaks. The right mix follows from the shape of the demand, and we will help work out where a workload sits rather than pushing a single model of provisioning.
Availability and reliability for production inference
Inference is typically a production, user-facing service, which makes its availability far more important than a training run's. When training is interrupted, work is delayed or lost, which is costly but internal; when inference goes down, users cannot get responses, features stop working, and the impact is immediate and visible. This means production inference has to be built for availability — able to keep serving despite failures — in a way that batch training does not, with redundancy and failover so that the loss of a component does not take the service down.
Building this involves the familiar tools of resilient infrastructure applied to inference: redundant capacity so a failure is covered, health checks and load distribution so traffic is routed to working capacity, and hosting on reliable hardware in a datacenter with dependable power and cooling. The aim is that the inference service stays available and responsive even when individual parts fail, because for a user-facing service, availability is part of the product. When we host inference infrastructure, this availability is a first-class concern, since a fast model that is frequently unreachable serves users poorly. Reliability is as much a requirement of production inference as speed is.
Cost efficiency when inference runs continuously
Unlike a training run that ends, production inference runs continuously, so the cost of running it accumulates for as long as the service operates — which makes the per-inference cost, and the utilisation of the hardware, matter a great deal. Inference hardware that is well-utilised, handling a steady stream of requests, is cost-effective; hardware that sits mostly idle, provisioned for a peak that rarely comes, wastes money continuously. Keeping the cost of inference reasonable is therefore largely about matching capacity to load and keeping that capacity busy, so that you are paying for work being done rather than for idle headroom.
For steady, ongoing inference load, this is where dedicated servers tend to win on cost. Because production inference often runs continuously at a fairly predictable rate, dedicated hardware at a fixed cost, kept well-utilised, is usually more economical than paying on-demand rates for capacity that is busy most of the time — the same logic that favours dedicated hardware for sustained training applies to sustained inference. On-demand and elastic capacity remain valuable for the variable portion of load, for peaks, and for genuinely spiky services. The economical design usually pairs dedicated capacity for the steady baseline with elastic capacity for the peaks, and we will size that honestly rather than defaulting to the most expensive option.
Data sovereignty for inference on live user data
Inference is where a model meets real user data: the inputs it processes — queries, documents, images, prompts — are live data from users, and that data is often personal, confidential, or regulated. Because inference processes this data continuously in production, the jurisdiction in which inference runs governs a stream of real user data, not just a static training set. For services handling European users' data, or data subject to European protection, where inference runs is directly a data-protection question, since the inputs being processed are exactly the kind of data those protections cover.
This makes EU-hosted inference a meaningful choice where user data is sensitive. VV Internet Hosting is incorporated in the Netherlands, within the EU, so inference hosted with us runs under European jurisdiction and outside the direct reach of the US CLOUD Act. For a production service processing European or sensitive user data, running the inference in the EU keeps that live data under European law rather than exposing it to foreign legal access — a consideration that applies with particular force to inference precisely because it handles real user inputs in real time. Where the data your model processes is sensitive or regulated, the sovereignty of the inference infrastructure is a real part of the decision.
Where VV Internet Hosting fits — and where it does not
We host dedicated infrastructure for AI inference: GPU servers for large or high-throughput models and CPU capacity for lighter inference, sized to the model and the load, with the availability that production services need, in EU datacenters under European jurisdiction. This suits production inference with steady or predictable load, where dedicated capacity is economical and where the live user data being processed makes European sovereignty matter. For those cases we are a strong fit, and we will help right-size the hardware — including telling you when a smaller GPU, or a CPU, will serve — rather than overselling.
Where we are less suited, we will say so. Inference with extremely spiky, unpredictable demand that needs to scale to zero and back rapidly can be better served by elastic on-demand platforms for the variable portion of the load. If you need a hyperscaler's managed inference platforms, serverless inference that bills per request with no baseline, or a specific proprietary model-serving ecosystem, that is a different kind of provider. We are for dedicated, sovereignty-aware inference infrastructure run on hardware you control, ideally with a steady baseline of load — which describes a great deal of production inference, but not all of it. If that is your situation, we can host it well; if it is not, we would rather tell you than take on a poor fit.
Related use cases
Questions
AI inference, answered plainly
Common questions about hosting for AI inference.
Does AI inference need the same hardware as training?
Not necessarily. Inference serves a trained model rather than training one, so it usually doesn't need the large multi-GPU interconnect training demands. The model must still fit in memory, but techniques like quantisation can shrink it, sometimes letting inference run on smaller GPUs or even CPUs. Match the hardware to the model and load rather than assuming training-class GPUs.
Should I optimise inference for latency or throughput?
It depends on the workload. Interactive, real-time inference where a user waits on the response optimises for latency — answering each request quickly. Batch inference, processing many inputs without anyone waiting, optimises for throughput — getting through as much work as possible, often by batching requests. Dynamic batching balances the two for load that arrives continuously.
Is dedicated or cloud better for production inference?
Steady, predictable production load usually favours dedicated servers: kept well-utilised at a fixed cost, they're more economical than on-demand rates for capacity that's busy most of the time. Spiky or unpredictable load favours elastic capacity. Many deployments combine a dedicated baseline with elastic capacity for peaks — sized to the shape of the demand.
Planning AI inference infrastructure?
We host dedicated, EU-sovereign infrastructure sized to your workload — and we will tell you plainly when something else fits better. Tell us what you're building.