Hadoop is undoubtedly one of the most important software built in recent times. At its peak, Hadoop was so massive that it was synonymous with the term ‘big data’. A lot has changed since the days when batch processing was a novel idea that every business needed to get on board with, and with that change, Hadoop’s importance greatly diminished.
Hadoop is an extremely powerful piece of software with numerous benefits for anyone looking to get into big data. However, it faces significant issues such as an overly-complex distribution process and inefficiency when processing both structured and unstructured data.
Fortunately enough, there are additional big data platforms that take advantage of the significant technological advancement of the last decade or so. These offer large speed increases, more efficiency or improved data processing capabilities.
Apache Spark
Spark is widely considered the most widespread replacement for Hadoop. It was first created as a batch-processing system that can be attached to Hadoop but quickly grew out of that shell. Today, Spark is more commonly used on its own rather than as an attachment to Hadoop. At this point, almost every developer has an idea of what is Hadoop and Spark.
The most obvious difference between Spark and Hadoop is the speed. Spark can be as much as 100x faster at processing data than Hadoop and was designed from the ground up to have a much simpler API.
Its speed is largely due to support for in-memory computing, but it also depends on a different file access paradigm than Hadoop’s two-step method. This way, repeated access to the same data is faster
Its reliance on in-memory processing allows it to support stream processing rather than Hadoop’s batch processing. Stream processing enables a number of applications, the most significant of which is real-time data.
The biggest downside is that Spark requires a lot of memory since all data that needs processing is loaded there by default. Spark excels at iterative computations that happen over the same set of data but isn’t so good at ETL jobs that involve a single pass.
Google BigQuery
BigQuery is a fully-managed big data platform that allows users to rely on SQL without being bothered about the database being used or maintaining any hardware. It’s a cloud-based service that relies heavily on Google services underneath to provide users with interactive analysis of data.
It succeeds as a platform because it takes away the difficulty of managing your own server and having to scale everything on your own if the need arises. Additionally, it often outperforms Hadoop when it comes to discovering specific patterns in raw data.
Hydra
Along with Spark, Hydra is another task processing system designed to deal with the real-time analytics that Hadoop falls flat on its face attempting. It relies on a configured that allows it to support both stream and batch processing across clusters with thousands of nodes.
It’s different from Hadoop because it takes in streams of data and builds trees which contain various transformations of the data. This allows for it to find use in exploring small queries, building systems that rely on large queries and scales well for large queries called hundreds of times within a short period of time.
Additional features include disk-based fault tolerance, a management system for distributing data between nodes and balancing existing jobs.
Ceph
Ceph differentiates itself from other big data platforms by offering support for object, block, and file-level storage, rather than a single large data lake, like that supported by Hadoop. It doesn’t rely on a single NameNode like Hadoop, either, getting rid of a major single point of failure.
Data stored on Ceph is fault-tolerant because the system replicates it on the disk-based memory. Since this process is automatic, Ceph takes away a huge chunk of the problems that would otherwise have to be dealt with in Hadoop. This system is self-maintaining and self-healing.
Additional functionality is granted by CephFS, a filesystem that uses recovery tools such as backups, snapshots and metadata servers to keep data safe..
Presto
Presto is a distributed open-source SQL query engine designed to run analytic queries against large data sources. It can query data from both non-relational sources such as Amazon S3, and HDFS, and relational ones such as Postgres.
This software gains a lot of its power from being able to query data where it’s stored, removing the need to move data to an entirely different analytics system to perform.
Since it allows parallel query execution over a pure memory-based architecture, most results will return in seconds. Its ability to combine data from multiple sources also makes scaling across the whole organization easier.
When it comes to big data software, Presto represents a break from the notion that you either have to have fast analytics on expensive commercial hardware or a ‘free’ solution that’s very slow or needs specialized hardware. Analytics data will always be returned in seconds, or minutes if the data is extremely large and the queries complex.
DataTorrent RTS
DataTorrent is an open-source solution that also provides an interface for both real-time and batch processing of data. It was designed specifically with an improvement of the inner workings of Hadoop’s MapReduce environment and does an even better job by improving the performance of tools like Spark and Storm.
It can process billions of events per second and replicates data stores in-memory to the disk, granting it fault-tolerance across nodes. If any node fails, it’s able to kick off data recovery processes on its own – no need for human intervention.
This big data solution also provides the ability to take in data from a multitude of different sources including structured SQL databases and unstructured files. This is achieved through connectors that exist for ingesting data from sources such as databases but also goes as far as to offer support for social media networks such as Twitter. Potentially anything that generates data can be attached to it.
