Use case · Big data & analytics
Apache Spark clusters
Apache Spark is a distributed processing engine that runs analytics, batch, streaming, and machine-learning workloads fast across a cluster by processing data in memory. Hosting it well means ample memory — its speed depends on it — many cores across executors, and fast networking for shuffles. Dedicated bare metal suits sustained Spark work, and EU hosting keeps the data sovereign.
Key points
- Apache Spark is a distributed processing engine that runs work fast across a cluster.
- Its speed comes from in-memory processing, so ample memory is critical.
- It runs as a driver coordinating executors spread across the cluster's nodes.
- Shuffles move data between nodes, so fast networking matters for many workloads.
- It's versatile — batch, streaming, SQL, and machine learning on one engine.
What is Apache Spark, and why host it?
Apache Spark is an open-source distributed processing engine, used to run analytical, batch, streaming, and machine-learning workloads quickly across a cluster of machines. It is one of the most widely used engines for big data processing, valued for its speed and its versatility — the same engine handles several kinds of workload. Spark processes large data by distributing the work across a cluster and, importantly, by working with data in memory, which makes it much faster than earlier disk-based approaches for many workloads. Organisations use Spark to process large datasets for analytics, to build data pipelines, to run machine learning at scale, and to handle streaming data.
Hosting Spark well means providing the cluster it runs on with infrastructure suited to how it works. Spark's in-memory processing makes memory a central resource; it distributes work across cores, so processing power across the cluster matters; it moves data between nodes during processing, so networking matters; and it needs storage for data and intermediate results. This page focuses on Spark specifically; the broader practice of big data analytics and related tools are covered on their own pages, while what follows attends to what is particular about running Spark clusters.
In-memory processing: why Spark is fast and memory-hungry
Spark's defining characteristic, and the source of much of its speed, is that it processes data in memory rather than repeatedly reading and writing it to storage between steps. Earlier big data processing wrote intermediate results to disk between stages, which was slow; Spark keeps data in memory across stages where it can, avoiding much of that slow disk access and greatly speeding up multi-stage processing. This in-memory approach is why Spark is fast for the iterative and multi-step workloads common in analytics and machine learning, where the same data is processed repeatedly.
This makes memory a critical resource for Spark, and often the one that most affects its performance and capacity. Because Spark holds data in memory to process it quickly, having enough memory across the cluster to hold the data being worked on lets Spark run at its fastest; when memory is insufficient, Spark must spill data to storage, which slows it toward the disk-based performance it aims to avoid. Providing ample memory across the cluster's nodes is therefore central to Spark performing well. We provide the memory Spark's in-memory processing needs, since memory is where much of Spark's speed advantage lives, and under-provisioning it undercuts the reason to use Spark.
Spark's architecture: driver and executors
Spark runs as a distributed application across a cluster, with a driver program that coordinates the work and executors that perform it across the cluster's nodes. The driver plans the work and coordinates it; the executors, running on the cluster's worker nodes, carry out the actual processing on their portions of the data, in parallel. A cluster manager allocates the cluster's resources to the executors. This architecture is how Spark distributes a job across many nodes — the driver directs, and the executors process in parallel across the cluster, combining to handle large data.
Understanding this architecture clarifies what a Spark cluster's infrastructure must provide: nodes to run executors, with the memory and cores each executor needs, coordinated so they work together. The number of executors and the resources each has determine the cluster's processing capacity, so sizing a Spark cluster means providing enough nodes with enough memory and cores for the executors the workload needs. The driver also needs resources to coordinate. We provide the cluster infrastructure Spark's driver-and-executor model requires — nodes sized for executors, arranged to work together — so that Spark can distribute processing across the cluster as its architecture intends.
Memory and cores for Spark executors
Spark performance depends on the memory and cores given to its executors, since these determine how much data each executor can hold in memory and how much processing it can do in parallel. Memory per executor governs how much data it can process in memory before spilling to storage, and total memory across executors bounds how much data the cluster can hold in memory at once; cores per executor and across the cluster govern the parallelism, how many tasks run at once. Both matter, and sizing them to the workload is central to a Spark cluster performing well.
Balancing memory and cores across the cluster is part of configuring Spark for a workload. A cluster of nodes with ample memory and many cores — as high-core-count processors with generous memory provide — gives Spark the resources to hold data in memory and process it in parallel at scale. The right amount depends on the data volume and the processing, and how memory-intensive the workload is. We provide nodes with the memory and cores Spark executors need, sized to the workload, so that the cluster has both the memory to keep data in memory and the cores to process it in parallel, which together determine how well and how fast Spark handles the work.
Networking: the shuffle
Many Spark workloads involve a shuffle — redistributing data across the cluster's nodes so that related data is brought together for the next stage of processing — and the shuffle moves large amounts of data between nodes, making networking important. Operations that group or join data require data to be moved so that records belonging together end up on the same node, and this shuffle of data across the network can be substantial. When the network between nodes is slow, shuffles become a bottleneck, slowing the whole job, so fast networking between the cluster's nodes matters for shuffle-heavy workloads.
This makes the interconnect between a Spark cluster's nodes a real consideration, alongside the memory and cores. A cluster with fast networking handles shuffles efficiently, keeping data moving between nodes without stalling the processing; slow networking drags on the shuffle-intensive operations common in analytics. The importance of networking depends on how much a workload shuffles, but for many Spark jobs it is significant. We provide fast networking between the nodes of a Spark cluster, so that the shuffles many workloads perform do not become a bottleneck, and the cluster's memory and cores are not left waiting on data moving between nodes.
Spark's versatility: batch, streaming, SQL, and ML
A large part of Spark's appeal is its versatility: one engine handles batch processing, streaming, structured queries, and machine learning, so an organisation can use it across many kinds of data work. Spark provides components for querying structured data, for processing streaming data continuously, and for machine learning at scale, all on the same underlying engine. This lets a Spark cluster serve varied workloads — a batch analytical job, a streaming pipeline, a machine-learning task — rather than needing a different system for each, which simplifies an organisation's data infrastructure.
This versatility has implications for hosting Spark, since a Spark cluster may serve several kinds of workload with differing resource patterns. Batch jobs may run heavily when scheduled; streaming runs continuously; machine learning may be memory- and compute-intensive. A Spark cluster hosting a mix needs infrastructure that accommodates the range, and understanding which workloads a cluster serves informs how it is sized. We host Spark clusters for the varied workloads Spark supports, sizing the infrastructure to the mix of batch, streaming, SQL, and machine-learning work a cluster runs, so that the versatility that makes Spark valuable is matched by infrastructure suited to the range of work it does.
Cost and bare metal for Spark clusters
Spark clusters can be substantial infrastructure, so cost and utilisation matter, and sustained Spark work tends to favour dedicated bare metal. A Spark cluster kept busy — running regular jobs, continuous streaming, or steady processing — is well-utilised, and for such sustained use, dedicated hardware at a fixed cost is usually more economical than on-demand rates for capacity busy most of the time. Because Spark clusters can be large and memory-heavy, the cost of running them matters, and keeping them well-utilised on dedicated hardware controls that cost for sustained workloads.
Bare metal also suits Spark for performance, since Spark's in-memory processing benefits from the full memory and performance of dedicated hardware without virtualisation overhead. On-demand capacity retains a role for intermittent Spark work — occasional large jobs that do not justify a standing cluster — where paying only for the time used fits the sporadic pattern. The economical choice depends on utilisation: sustained Spark work on dedicated bare metal, intermittent work on on-demand capacity. We provide dedicated bare-metal Spark clusters for sustained work, where the utilisation and Spark's memory demands make dedicated hardware economical, and will say so honestly where on-demand capacity would suit intermittent Spark work better.
Sovereignty, and where VV Internet Hosting fits
Spark processes data that is often large and sensitive — the datasets an organisation analyses, which can include personal, behavioural, and regulated data — so where a Spark cluster runs governs that data, making sovereignty a consideration. VV Internet Hosting is incorporated in the Netherlands, within the EU, so a Spark cluster hosted with us runs under European jurisdiction and outside the direct reach of the US CLOUD Act. For processing data subject to European protection, keeping the Spark cluster and its data in the EU keeps them under European law, which matters for the sensitive data Spark often processes.
We host dedicated bare-metal Spark clusters: nodes with ample memory for Spark's in-memory processing, many cores for parallel executors, fast networking for shuffles, and storage sized to the data — in EU datacenters under European jurisdiction. This suits organisations running sustained Spark workloads that want performance, economical dedicated hardware, and European sovereignty. We are clear about our limits: we provide the dedicated infrastructure you run Spark on, not a fully managed Spark service that operates it for you; and intermittent Spark work may suit on-demand capacity better. If you want a managed big data platform, that is a different offering. We are the dedicated, sovereign infrastructure for self-hosting Spark clusters — and if that fits your work, we can host it well.
Questions
Spark clusters, answered plainly
Common questions about hosting for Spark clusters.
Why is Apache Spark fast, and why does it need so much memory?
Spark processes data in memory rather than repeatedly reading and writing it to storage between steps, which avoids slow disk access and speeds up multi-stage and iterative workloads. This in-memory approach is the source of its speed — and it makes memory critical: enough memory across the cluster to hold the data being worked on lets Spark run fastest, while insufficient memory forces it to spill to storage.
How is a Spark cluster structured?
Spark runs as a distributed application with a driver program that coordinates the work and executors that perform it across the cluster's worker nodes, in parallel. A cluster manager allocates resources to the executors. The number of executors and the memory and cores each has determine the cluster's processing capacity, so sizing a Spark cluster means providing enough of both.
Is dedicated or on-demand better for Spark?
It depends on utilisation. Sustained Spark work — regular jobs, continuous streaming, steady processing — keeps clusters busy, favouring dedicated bare metal at a fixed cost, which also benefits Spark's in-memory processing with full memory and performance. Intermittent Spark work — occasional large jobs that don't justify a standing cluster — can suit on-demand capacity, where you pay only for the time used.
Planning Spark clusters infrastructure?
We host dedicated, EU-sovereign infrastructure sized to your workload — and we will tell you plainly when something else fits better. Tell us what you're building.