A couple of years back, I wrote my first blog on Big data: –Getting in Shape with Big Data. While almost two years have gone by – almost an eternity for the Information age standards – I keep hearing the same question from my customer visits: are we there yet? Reality is that Big Data is just starting to deliver – Yes the value is indeed real and tangible - but the technology is on the early stages and it takes a lot of vision, expertise and sheer hard work to make a Big Data solution work. Hadoop has become the least expensive file system in the world. Many organizations are starting to use it as a cheaper Data Archival alternative with the promise that if the data needs to be retrieved the cost of storage and access will be cheaper than tape itself. While data archiving might sound like a modest use case it has created quite a revolution in the BI ecosystem. If you think about it, many large organizations kept a lot more data in the DWHSE because of the perceived loss of the data once it was moved to tape. However, with the data now being kept on Hard Drives and available with a simple Java map-reduce programs, many organizations are literally cleaning house and moving old data out of the Data Warehouse thus delaying the upgrade of the infrastructure and effectively putting some projects on hold.
From an analytics perspective adoption has been slower, while open source brings some good tools into the ecosystem – including R (for advanced analytics) and Graphite (for advanced graphics and plotting), we have yet to identify how to leverage of the power of the data stored in Hadoop for the masses. While it is true that everybody with Data and Java skills can write Map-Reduce programs, let us face it the majority of the analysts are Excel and Access gurus with the advanced population being proficient on SQL, only a very small percentage actually know how to code in Java. This lack of Java knowledge among the analysts has truly become the bottle neck for Hadoop to be an effective replacement of the Data Warehouse. However, the innovation continues and some open source projects like HIVE & PIGS continue to evolve to enhance the data consumption experience. While still in the early stages, the technology is very promising and will eventually mature to the point that Java Map-Reduce will be an exception rather than the norm for accessing data stored in Hadoop. In fact, there are technologies today, albeit commercial such as Teradata Aster, which can insulate the analysts from writing Map-Reduce and will simplify access to data stored in HDFS (Hadoop File System).
So are we there yet? Maybe not, but the speed of change is accelerating. As of now, probably most of the Fortune 500 companies are defining a big data strategy which in turn will push software, hardware and services vendors to innovate at a faster rate to close the gap against the expectations (or hype) that Big Data has created with the business stakeholders. My prediction is that in two more years (mid 2015) we will be able to access Hadoop in different ways that put the technology much closer (from a data perspective) to the relational SQL databases that we know today.