Use case · Big data & analytics
Hadoop clusters
Apache Hadoop is the foundational big data ecosystem — HDFS for distributed storage across a cluster, plus MapReduce and YARN for processing and resource management. Its HDFS storage remains widely used for huge datasets, while Spark often replaces MapReduce for processing. Storage-heavy clusters suit dedicated bare metal, and EU hosting keeps the data sovereign.
Key points
- Hadoop is the foundational big data ecosystem: HDFS storage, MapReduce processing, YARN scheduling.
- HDFS distributes huge datasets across a cluster with replication for fault tolerance.
- Hadoop clusters are storage-heavy, using the local disks of many nodes for HDFS.
- MapReduce processing is now often replaced by Spark, while HDFS storage endures.
- Storage-heavy sustained clusters suit dedicated bare metal, hosted in the EU for sovereignty.
What is Apache Hadoop, and what is it for?
Apache Hadoop is an open-source ecosystem for the distributed storage and processing of big data across clusters of ordinary hardware, and it was foundational to the whole field of big data. It brought together several parts: a distributed file system, HDFS, for storing huge datasets across many machines; a processing model, MapReduce, for computing over that data in a distributed way; and a resource manager, YARN, for coordinating the cluster's resources among jobs. Around these grew an ecosystem of related tools. Hadoop made it practical to store and process datasets far larger than a single machine could handle, using clusters of commodity servers.
Hosting a Hadoop cluster means providing the cluster on which these components run — many nodes with substantial storage for HDFS, compute for processing, and networking to tie them together. Hadoop is storage-heavy by design, using the disks of its many nodes to store data, so its infrastructure emphasises storage capacity across the cluster alongside compute. This page focuses on Hadoop; Spark, which often runs alongside or in place of parts of Hadoop, and the broader practice of big data analytics, are covered on their own pages, while what follows attends to what is particular about Hadoop clusters, including their place today.
HDFS: distributed storage for huge data
At the heart of Hadoop is HDFS, the Hadoop Distributed File System, which stores very large datasets across the many nodes of a cluster, distributing the data and replicating it for fault tolerance. HDFS spreads a large dataset across the storage of many machines, so that together they hold far more than one machine could, and it keeps multiple copies of the data across different nodes, so that the failure of a node does not lose data — the copies on other nodes remain. This distribution and replication make HDFS able to store huge datasets reliably on clusters of ordinary hardware.
HDFS is designed for the access patterns of big data: large files, high-throughput sequential access, and processing data where it is stored. It suits storing and reading large datasets in bulk, as big data processing does, rather than the small random access of a transactional system. Its fault tolerance through replication means data survives hardware failures, which matters at the scale of many nodes where failures are expected. HDFS remains a widely-used way to store big data reliably at scale, and it is a core reason Hadoop clusters are storage-heavy — the cluster's nodes contribute their storage to HDFS, together holding the large datasets.
MapReduce and YARN: processing and resource management
Hadoop's original processing model is MapReduce, a way of expressing distributed computation over data in HDFS, and YARN is the resource manager that coordinates the cluster's resources among jobs. MapReduce processes data in a distributed, parallel way across the cluster, working on the data where it is stored, and it was the classic means of computing over big data in Hadoop. YARN allocates the cluster's compute resources to jobs and schedules them, so that multiple jobs and frameworks can share the cluster's resources. Together, they provided the processing and coordination that turned HDFS storage into a platform for big data computation.
MapReduce processes data largely by reading and writing to storage between steps, which is reliable but slower than in-memory approaches for many workloads. This disk-based processing was foundational, but it is where newer engines have improved on Hadoop's original model, as the next section discusses. YARN, by contrast, remains useful as a resource manager, and can schedule other frameworks — including Spark — on a Hadoop cluster, as well as MapReduce. Understanding MapReduce and YARN clarifies Hadoop's original processing approach and why its resource management endures even as its processing model has been superseded for many uses.
Hadoop today: HDFS endures, Spark often replaces MapReduce
The honest picture of Hadoop today is that its storage layer, HDFS, remains widely used, while its original processing model, MapReduce, has largely been superseded by faster engines, chiefly Spark, for many workloads. Spark's in-memory processing is much faster than MapReduce's disk-based approach for the iterative and multi-step work common in analytics and machine learning, so many organisations that once processed data with MapReduce now use Spark instead. But HDFS remains a solid, widely-used way to store big data at scale, and Spark and other engines can read from and write to HDFS and run on a Hadoop cluster via YARN.
This means a Hadoop cluster today often serves as a storage and resource-management foundation on which faster engines run, rather than relying on MapReduce for processing. A common pattern is HDFS for storage, YARN for resource management, and Spark for processing — combining Hadoop's enduring storage with a faster processing engine. We are candid about this: Hadoop remains valuable, especially its HDFS storage, and it need not mean using MapReduce for processing. When we host Hadoop clusters, we host them for what they are genuinely good for today — distributed storage and resource management, often alongside Spark for processing — rather than pretending the original MapReduce model is the best choice for workloads where Spark has surpassed it.
Infrastructure for a Hadoop cluster
A Hadoop cluster is storage-heavy, using the local disks of its many nodes for HDFS, so its infrastructure emphasises storage capacity across the cluster, alongside compute and networking. Because HDFS distributes data across the storage of the cluster's nodes, each node contributes storage, and the cluster's total storage capacity comes from the disks across all its nodes. This makes storage capacity a primary consideration, with nodes provisioned to hold their share of HDFS data, and the cluster's total capacity sized to the datasets it must store.
Compute and networking round out the requirements. The nodes need compute to run the processing — whether MapReduce, Spark, or other engines — over the data they hold; and networking ties the cluster together, carrying data between nodes when processing requires it and supporting HDFS's replication across nodes. The balance of storage and compute depends on how storage-heavy or compute-heavy the workload is. We provide Hadoop cluster infrastructure sized for its storage-heavy nature — nodes with the storage HDFS needs — together with the compute for processing and the networking to tie the cluster together, matched to whether the cluster emphasises storage, processing, or both.
Cost and bare metal for Hadoop clusters
Hadoop clusters are substantial, storage-heavy infrastructure, and sustained use favours dedicated bare metal on both cost and suitability. A Hadoop cluster storing large datasets and running regular processing is used continuously — the stored data persists and the processing recurs — so it is well-utilised, and dedicated hardware at a fixed cost suits this steady, storage-heavy use better than on-demand rates for capacity held continuously. The large storage HDFS requires is especially suited to dedicated hardware, where the storage is provisioned once and used continuously, rather than paid for on demand.
Bare metal also suits Hadoop for the direct access to local storage that HDFS uses and the full performance dedicated hardware provides. HDFS is designed around nodes' local storage and data locality, which fits dedicated servers with their own disks well. On-demand capacity has less of a role for the storage foundation of a Hadoop cluster, which is inherently persistent, though it may suit occasional extra processing. The economical and suitable choice for a sustained, storage-heavy Hadoop cluster is usually dedicated bare metal. We provide dedicated bare-metal Hadoop clusters sized for their storage and processing, which suits the persistent, storage-heavy nature of Hadoop.
Sovereignty, and where VV Internet Hosting fits
A Hadoop cluster's HDFS holds large datasets that are often sensitive — the big data an organisation stores and analyses, which can include personal, behavioural, and regulated data at scale — so where the cluster runs governs a large volume of such data, making sovereignty a real consideration. VV Internet Hosting is incorporated in the Netherlands, within the EU, so a Hadoop cluster hosted with us runs under European jurisdiction and outside the direct reach of the US CLOUD Act. For storing and processing data subject to European protection, keeping the Hadoop cluster and its HDFS data in the EU keeps them under European law, at the scale big data involves.
We host dedicated bare-metal Hadoop clusters: nodes with the storage HDFS needs, compute for processing, and networking to tie the cluster together — able to run HDFS and YARN with Spark or other engines for processing — in EU datacenters under European jurisdiction. This suits organisations running sustained Hadoop clusters for distributed storage and processing that want performance, economical dedicated hardware, and European sovereignty. We are candid about our limits and about Hadoop itself: we host it for what it is good for today, especially HDFS storage often paired with Spark, rather than pushing MapReduce where Spark suits better; we provide the dedicated infrastructure, not a fully managed Hadoop platform. If self-hosting a Hadoop cluster on infrastructure you control fits your needs, we can host it well.
Questions
Hadoop clusters, answered plainly
Common questions about hosting for Hadoop clusters.
What are the main parts of Hadoop?
HDFS, the Hadoop Distributed File System, which stores huge datasets across a cluster with replication for fault tolerance; MapReduce, the original distributed processing model; and YARN, the resource manager that coordinates the cluster's resources among jobs. Around these grew an ecosystem of related tools. Together they made storing and processing datasets far larger than one machine practical.
Is Hadoop still relevant, given Spark?
Partly. HDFS, Hadoop's storage layer, remains widely used as a solid way to store big data at scale, and YARN remains useful for resource management. But MapReduce, Hadoop's original processing model, has largely been superseded by Spark's faster in-memory processing for many workloads. A common pattern today is HDFS and YARN for storage and scheduling, with Spark for processing.
Why are Hadoop clusters storage-heavy?
Because HDFS distributes data across the local disks of the cluster's many nodes, so each node contributes storage and the cluster's total capacity comes from the disks across all nodes. HDFS is designed around nodes' local storage and processing data where it's stored, which makes storage capacity a primary consideration and fits dedicated servers with their own disks well.
Planning Hadoop clusters infrastructure?
We host dedicated, EU-sovereign infrastructure sized to your workload — and we will tell you plainly when something else fits better. Tell us what you're building.