On vacation with Big Data

According to recent statistics more than 90% of people in the USA spend their days within 30 miles of their home location. They define their daily circles around things they know and are familiar to them: their neighborhood, schools, shops, etc. We are all creatures of habit: chances are that what you do on a typically Monday will be repeated week after week with little variation until you go on vacation. Vacations are truly events of consequence because it is one of the few occasions where people disrupt their routine, it is quite common while on vacation for people to travel more than 30 miles from their home location, either by car, planes, boat or any other transportation method. While on vacation a person is away from the familiar and he/she is probably more open to visit new places, try new food, meet new people or just wander around.

If you think about, this is the perfect opportunity for an external party to influence you, your guard is relaxed and you want to have fun and experience new things. Hospitality companies are aware of this and are starting to use Big Data technologies to make sure you have the “best experience away from home” (alas it does not hurt if this “best experience” puts more money in the hospitality company pocket). Let us explore how Big Data is directing your experience.

Big data influence in your trip starts way before you book your trip. In fact, most hospitality companies today use advanced analytics to drive their marketing campaigns to match their offering/properties to customers/prospects interests and preferences. There is a probably a digital trail that you have created in any (and maybe all) of your previous trips. This digital cookie gives enough information to the hospitality companies to target you on a way where you are most likely to respond.

Once you have decided to stay at a particular property and book the trip, Big Data technologies are used to present you with “bundles” or “packages” that enhance your experience. It can be a combination of car plus your room, or include meals or events. The beauty of Big Data is that regardless if you select, ignore or reject the offering the hospitality industry keeps learning more and more about you and how you react to specific offerings.

When you check-in into the property, the Big Data analytics engine is right there with you; depending on your history with the company, management might offer you an upgrade or just recommend things for you to do around. Keep in mind that the hospitality company will try to entice you to stay longer at the property and consume their services rather than leaving the premises and risk you spending your hard earned dollars with the competition.

So next time you go on vacation, don’t be surprised if you find yourself trying a new experience that you would have never tried at home. After all you are on vacation with Big Data and it is up to you if you bring it back home with you.

Big Data and HR: when analytics become personal

We live in the Code Halos age where everything that we do, every interaction that we have- positive or negative - generates data that lives somewhere in the “cloud”. No matter your age, there is probably a digital trail that you have left behind, data about you starts generating even before you are born. In fact there are many studies that capture data about the development of a baby while still in her mother’s uterus. But what does this mean for us? If it is a good or a bad thing that all this data exists and more importantly persists as we are born and grow up? More importantly, how does this affect our ability to get a job, create a company or participate in public events? This blog will attempt to answer this question by putting it in the context of a real scenario given to me by one of my customers.

This customer is a national retail chain that operates over 8,500 stores and employs over 185,000 associates. This company HR department probably interviews hundreds of candidates a day which costs the company thousands of dollars in people’s time; not to mention that while they have been refining and maturing their interview process it is still not a 100% accurate and bad hires still get through sometimes. What if rather than an elaborate interview process which astute candidates can manipulate, the company implemented a big data system that could process all the information in existence for a particular candidate. All the records since that person got in the “system” would be made available to an engine that would recommend the right candidate(s) for a particular role. Not only that the system would be able to predict (with a high degree of accuracy), how well that person would do on the role, when that person would get promoted, identify the highest position that person would probably occupy through their career in the company and more importantly identify the likelihood of retaining that person in the company the right time to achieve his/her maximum potential.

Is this real or a product of a very imaginative science fiction mind? Before we decide to answer this, let us explore other areas that deal with the very core of what is to be human: spouse selection. Did you know that about 25% of all marriages in the USA started online? In fact there have been scientific studies that concluded that people who met online are happier than people than met a traditional way (through friends, work, etc.). I myself did not believe this until I attended an analytics conference where the chief data scientist from eHarmony gave a presentation and he explained that when you meet in person for the first time, our instincts hone-in primarily in looks which while might lead to temporary satisfaction of being with an attractive person from the opposite sex. However this physical attraction does not necessarily lead to a relationship success in the long run because of potential misalignment in key areas (e.g. career goals, way to raise the kids, etc.). He explained that the online matching industry has developed a set of personality tests that are extremely difficult for someone to fake and once this information is in their systems, they can effectively use it to populate the analytical models that will look for long term compatibility as the primary criteria for matching.

So going back to our original question if big data can provide better results in HR than a face-2-face interview, I argue that not only it can but it will. If something as complex and personal as meeting the right person that will become your spouse is now in the hands of a big data algorithm, it is just a matter of time before big data does the same for your next job interview.

Why is the understanding of the consumer relevant for a Business Intelligence solution?

I was recently surprised when one of my customers asked me if I thought that understanding the mind and behavior of the customer was relevant when designing a Business Intelligence solution. My gut was telling me that the answer was absolutely yes, but I did not have any elements to qualify my response and in all honesty typically it is the other way around, we design Business Intelligence solutions to get insights into the mind and behavior of consumers.

Presented with this conundrum, I really had to go back to my college days and pull out some material on psychology and examine it in the light of my professional experience and role as a consumer. The findings were really interesting as they helped me to put many variables in perspective. I will attempt to document my conclusions on this blog with the hope that others will find them useful as well.

My investigation began while searching for an answer to a basic question: what are our motivations behind consumption? If you think about it, at a basic level, we all have the same physical needs: food, sleep, etc. but our consumption habits are extremely different, even with physical identical twins they might choose different styles, colors or just entirely different products when given an opportunity.

I used Maslow’s hierarchy of needs (Google it if you don’t know what I am talking about) as a base to create my own interpretation of a consumer acquisition patterns (Figure 1)

While it is true that a basic level we all need to satisfy the same physical needs to survive, once these needs are covered, we immediately tend to start looking to satisfy higher level needs that truly vary from person to person according to personality, experience, emotional intelligence, etc. At the end of the other spectrum we have a purely intellectual need that is a reflection of the self where each of us IS truly different.

Definitively an interesting finding, but it and on by itself it does not tell us anything truly new, however it does set the principle that a base level we all need to satisfy the same physical needs. How we decide to satisfy these needs that is entirely up to the self.

Retailers understand this principle extremely well; in fact most of the retail definitions (e.g. stores and brands) exist as the intersection of three variables: Product, Price and Place. It is all about what we want to consume, at what price, where.
So, if it is all about the self and given the choice each of us will select something different, how can we influence what the consumers will pick?
Going back to our retail example, the “easiest” variable to play with it is typically “Price”, as the location of the stores and the item assortments cannot be changed by the minute (while pricing literally can).

Using “Price” to motivate consumption is a science by itself; however, for the purpose of this blog I will attempt to simplify the concept within a continuum as depicted in Figure 2 below.

Most of the consumers will be motivated to acquire a good/service only if it falls within the value zone (the ratio of satisfaction received is proportional to the price), if the price goes too high or even too low we will find extreme behaviors driving the purchase. For example, who can remember when the first Application Store opened for the iPhone there was an application that did not do anything but display the image of a Gem? The developer wanted to charge 1000 USD for the application which only purpose in life was to showcase your friends you were wealthy and had a significant position or social status. Most of the iPhone users ignored the application, but surprisingly 5 people bought the app before Apple pulled it out from the store. The other extreme is an item which is priced too low and we immediately start associating the low price with low quality, for example who would buy a 50 inch LED TV for 250 USD when we know the average price for those TVs is much higher? Again, during 2013 Black Friday there were long lines outside of Target to buy this TV mostly because people enjoyed the trill of getting a very good deal or truly because they did not have money to buy anything better.

During this journey of consumer discovery, we cannot forget that people are social entities that do not live in isolation, but rather in a very complex network of relationships. Our ability to perceive the world is many times tied to the people that we interact more closely, not to mention our emotional and sometimes hormonal fluctuations. In fact, a typical person’s mood might vary significantly during a day in respond to these stimuli as depicted in Figure 3 above.

Retailers or service providers who understand this principle are able to position their offerings in such a way that they connect emotionally with the consumers. For good or bad an emotional connection is many times stronger than logical connection thus once an emotional connection is made the consumer is extremely likely to remain most loyal to the brand and truly go out of their way to acquire the provider’s product or services (e.g. who can forget the long lines at Apple stores with the launch of the early generations iPhones?)

However, it is also worth noticing that while emotion can influence a purchase, the reverse is also true.  For example, for years retailers have been able to determine the shopping occasion by the number and type of items in a shoppers market basket, however using signage, promotions and key interactions with sales associates, the retailer is able to influence and sometimes convert the initial purpose of shopping trip to a completely different outcome. Again, taking our example of Black Friday, many shopper do the lines with the expectation to buy the door buster item and go after the next bargain, but once in the store they can be influenced to stay and shop for other items that were not originally on their list.

I hope that this exploration of the consumer motivations and behavior has shed some light on our original question to determine if this insight is relevant when designing a Business Intelligence solution? A Business Intelligence solution is all about using data facts to measure specific variables against established baselines. However, the key value of the solution is in the understanding of what to measure and more importantly against what baseline. These two variables are invariable linked to the outcome that one wants to optimize and most of the BI implementations are deployed with the objective of improving consumer sales.  

The final retail battle: Brick and Mortar vs. eTailers

For many years retail was a simple art: stand-up a store, fill it with merchandise and let the shoppers take home what they like. The advent of the internet and on-line retailers like Amazon changed all that. This blog will attempt to explain how the electronic retailers changed customer shopping habits and what “hurdles” remain for electronic retailers or eTailers, for short, to completely disrupt what was once the beloved brick and mortar model. This blog will also comment on what opportunities and actions are available to brick and mortar retailers to restore their competitive advantage against the threat of complete domination of eTailers.

I-                    Traditional Shopping model

In the traditional brick and mortar shopping model, a merchant decides to build a store in a community taking into consideration the people within a particular distance radius of where the store will be located. It is assumed that the store’s merchandise will cater to those people and as such the assortment is carefully selected – within the constraints of space at the store – to maximize the product turnaround. Careful analysis of the store demographics is performed to fine tune the assortment, and pricing is determined both by demographics, and other competitive options that people might have available within the store radius of influence.  This model worked well for many years, allowing the creation, consolidation and growth of national, regional and local retailers that followed pretty much these same principles when opening new stores.

This model had a very good advantages, the customers knew what to expect and how to operate within the model and It was very straightforward: 1) look for something that you like on the shelf, 2) take to the register where it will be bagged and 3) pay for it and take it home. This model was also defined by some as “Cash and Carry”, as you would pay for things with cash (or equivalent) and take the merchandise home with you. This model provides an immediate gratification to the buyer and it is fairly simple for the retailer to operate as most of the goods were delivered to the store from a distribution center (Figure 2)

In order for the traditional model to work, both the Distribution Centers and the customers should be close to the store. Physical distance was of paramount of importance because both the cost of delivering goods to the store and the hassle for customers to get to the store increase proportionally to the distance it takes them to reach to the store (Figure 3)

So while people could potential drive a longer distance to get to their favorite store, the reality is that if the store is too far, they would probably not do it, and they would probably just settle for a comparable alternative that was convenient to them.

II-                  The Internet Shopping model

The internet changed the entire paradigm by opening a virtual store that is always a click away, taking the shopping convenience to an entirely new level where you can shop from anywhere you have a connected device without worrying about distance, gas or how you look.

The virtual store itself brought significant advantages to shoppers given them access to an expanded assortment – not limited by physical shelve space constraints, lower prices – giving that the ecommerce retailers could pass on the savings in store personnel, store maintenance, parking lots, etc – and in many instances lower taxes –typically companies are not required to withhold local sales taxes for out of state shoppers.

Many old time retailers were skeptical of the new model as it require access to a computer, internet and the ability to buy based on a limited description of the item rather than having access to the item itself, however time proved them wrong as the internet expanded into everybody’s pockets with the advent of the smartphone and tablets. Further, the benefits of price and selection outweighed any potential concerns of not being able to touch the item, especially for commodity such as books, electronics and many others categories.

The new model was not perfect; it had a small chin in the armor, the wait time. Typically shoppers would need to wait between 3 to 5 business days to get their item after submitting the order. (Figure 4).

The battle was on, with location and immediate gratification as its only weapons, brick and mortar retailers tried to wage war on their ecommerce counterparts with limited success. The ecommerce retailers counterattacked by building additional distribution centers closer to their customers and by introducing special programs that allowed members to receive their orders in about 2 business days. Having lost the battle on price and selection to the internet retailers (Figure 5), brick and mortar retailers could not afford to let them eliminate the location hurdle to wipe them over, so a new strategy was needed.

III-                Transforming Location into a competitive advantage

The strategy for the brick and mortar retailers needed to incorporate elements of the eCommerce counterparts by adding an internet channel but rather than replacing the traditional store channel creating a hybrid strategy that allowed pushing flexible and valuing add services, such as:

a)      Inventory origination flexibility: If the inventory for the item was not available at a particular store, you could find availability of that product within the “distribution cloud” – e.g any distribution center or even other stores that had the item(s) in stock could function as the shipping center, either sending the product to a nearby store where the customers could pick-up the items or directly sending the purchased items to the customer’s home

b)      Personalization: You could order the item on-line with your specific instructions, e.g. A birthday cake with a particular design and/or theme and pick it at the store,  or perhaps an engraving line for that new tablet

c)       Value added-services: like remembering your past purchases and offering you the ability to repeat the same purchase. Especially useful for prescription like items that you need to refill every month, or even for seasonal items that you bought last year and would like to complete the set or maybe for remembering the wine and cheese pairing that your significant other bought that was delicious and you cannot longer remember

d)      Social: you could immediate share what are you buying, your experience with the retailer and your experience with the product to your inner circle or to the masses with a few clicks

The most complex and daunting initiative was perhaps the establishment of the distribution cloud (Figure 6). This distribution cloud would become the new virtual hub to the retailer delivery operations, coordinating across stores and distribution centers to find the most optimal location to ship the inventory from; depending on availability, distance to destination and handling speed/cost.

 IV-               Enabling visibility within the new distribution cloud
Retailers quickly realized that the implementation of this new distribution cloud was a game changer, but they also realized that in order to achieved the promised competitive advantages they needed a way to get visibility into what was happening within the distribution cloud as their existing information systems were not design to provide the visibility required by the additional, smaller transactions being generated by the on-line channel. The new system required the implementation and sometimes definition of the new capabilities, including reverse logistics (e.g. if the customer returns the item to whom would it be shipped back - probably not the original inventory location of the item) and the ability to detect potentially fraudulent transactions, including money laundering. Further, additional business rules needed to be captured that enabled the proper routing of orders to the best suited distribution points, including equipment and labor available on a quasi-real time basis.

While many existing data warehouses already had some of these elements and were producing reports that measured some of these metrics, it was required to retrofit them with the new business rules of a hybrid, order on-line pick in store/receive at home system. New capabilities were required for an end-2-end Business Intelligence distribution platform that could provide the visibility and thus corroborate the effectives, savings and the acquisition/retention of key customers through this system (Figure 7)

V-                 Summary
In summary, in order to survive the brick and mortar retailers are being forced to adopt some of the same strategies that they are competing against. However, if they want to win the race, they need to leverage the competitive advantage their physical locations provide them over pure ecommerce retailers to implement a hybrid strategy that leverages the physical assets that in close proximity to the customers and integrate well with an on-line ordering channel that enables value add services. 

The brick and mortar retailers also need to be prepared to invest in expanding & enhancing their supply chain and distribution cloud business intelligence systems to enable the proper measuring of the new key metrics, thus accounting for the intelligence needed to fine-tune the system and make it delivers the game changer results they need.

How much is loyalty worth?

Have you ever wondered how much is loyalty worth? Ever since the beginning of business history, business owners have tried to reward their best customers to entice them to continue to do business with them. The practice definitely got more popular when American Airlines launched their AAdvantage in 1981, establishing a competitive differentiation for the airline.

Let us explore why loyalty programs were such a game changer in business. If you think about it, most of the items and services you buy are considered commodity (e.g. they can be acquired from more than one service provider or manufacturer with very little differences between the two (or more) companies that provide that service or product). The airline business truly reflects this, if you are traveling from LA to New York, you will get the same if you fly airline A, airline B or airline C. All the planes travel at more or less the same speed, leave and arrive at the same airports and offer similar amenities (or lack of these days). So if three airlines fly the same route, you will always pick the one with the best price right?

Well, here is where things start to get interesting. The best loyalty programs not only give a kickback (or reward) with every purchase, but they are designed in such a way that the more you consume that product or service, the greater the rewards become. Going back to our airline, all the three airlines give you “miles” when you fly with them, but the differences are significant for people who fly them occasionally to people who fly them more often. More frequent flyers start to move up in tiers that offer additional perks such as “priority boarding”, “free bags”, “upgrades”, “better availability of award tickets”, etc.

If you are thinking that these perks hardly matter, think again, these so called perks influence millions of people to acquire products or services from a particular provider just to maintain or attain a particular tier or milestone in the program. In fact, many travelers will choose a particular airline for a route even if the price is higher than competition because of their status in the loyalty. Before you start thinking that these people should be criticized for letting airlines get away with higher fares, let us analyze the reason behind the behavior and what these travelers get in exchange for a few(or a lot) extra dollars: Most people think that if they travel a particular airline often enough they will get upgraded to first class. For good or bad this is true, most (if not all) airlines allocate unsold first class tickets to their most loyal customers often times as a free upgrade. Not to mention that frequent travelers enjoy priority lines and other benefits that do not cost the airline anything extra but they make the traveler feel important and do provide some additional comfort.

So far so good, it seems that users are willing to stick to a particular service provider in exchange for perks even if the price is a little higher for a commodity like service, but what happens when something does not go according to plan? Let us stick to the airline examples and revisit a real life scenario that happened recently in an American Airlines flight from Charlotte to Dallas. The next flight to Dallas had an empty seat that was allocated to the people on stand-by, one executive platinum (the highest American Airlines tier level) got the seat (even though his confirmed flight was four hours later) and boarded the plane early as per the airline policies. When he boarded the plane he noticed that a seat was broken and it was marked with masking tape, because the seat was not assigned seat he did not think much about the issue. Fast forward 20 minutes later when boarding was almost complete and surprise there is a person standing because she got allocated the seat that was broken. Given that no other seats are available the airline personnel needs to make a decision to ask one person to leave the flight.

If the airline truly valued the customer loyalty, who do you think they would asked to deplane? The last person who bought the ticket? The last person who checked-in? The person that got assigned the broken seat? Maybe the person who had the least miles in the airline loyalty program? Well no. By federal law when bumping a passenger from a flight for mechanical reasons (e.g. a broken seat), a confirmed passenger is entitled to compensation, which typically varies from 200 to 400 USD travel voucher for a domestic flight. So in order to minimize the expense the local airline supervisor decided to ask the Executive Platinum passenger who got the stand-by seat to deplane as this passenger was not entitled to any compensation because he did not hold a confirmed seat.

You must be thinking hum, so the loyalty program is good and valid only when the marginal cost to the airline of providing those “benefits” is marginal, when there was a real cost involved the loyalty of this frequent traveler was not worth as much as compensating a confirmed passenger for the next flight… So going back to the original question that this blog posed, how much is loyalty worth? Certainly for American Airlines it was worth much less than a couple of hundred bucks. Do you think the local supervisor made the right decision? What would you have done in his place?

More importantly, while this example applies to a particular airline, what greater lesson can we derive from this experience? My recommendation: be sure that you have a good way to measure the benefits (consumer) and cost (business) of the loyalty program, so the effectiveness can be objectively evaluated. And most of all, keep in mind that even the best loyalty programs will require trade-offs at some point in time and most importantly define that loyalty cannot be taken for granted. It gets renewed with every iteration consumers and business and it gets define over time taking into consideration all the acceptable alternatives.

In conclusion, like it or not, loyalty programs are here to stay. They provide a systematic way to influence consumers towards a particular brand and have proved to be extremely successful in encouraging repeat business. However, we need to be aware that everything comes at a cost for both consumers and business and that as good as the programs might or might not be, at the end is the people element that will make a lasting impression and determine the memory of the experience.

eWallet: The next step in retail evolution

We live the in the “smart” age, everything seems to be connected to the internet these days, your phone, your TV, your car, but what about your wallet? Do you still have those last century old paper bills or maybe those molded plastic credit cards? Reality is that most of us still carry (and god forbid) use Twentieth century tools on a daily basis for buying goods and services in the Twenty first century.

So, how can come the digital revolution has yet to reach us in our pockets?  Don’t worry, it is coming and coming fast. With the new generation of mobile devices is now possible to link the device directly to bank and credit card accounts and many merchants have started to accept payments leveraging either a scan code displayed in the phone screen, or through “contactless” readers using the NFC (Near Field Communication) standards.

If you accept the fact that these technologies are already here, the next logical question is what will it mean for you, will it change what, where and how we shop? Most likely it will, let us explore what possibilities this new technology will bring and how it will impact shopper behavior:

First, let us think about how each of us gets to know about new products and what is happening in the market with our favorite retailers, chances are that you are part of either a mailing list and/or email campaign that periodically targets you with advertising. However, most of us find these kind of unrequested advertising irrelevant, annoying most times and occasionally a nuance. While some of the advanced shoppers would have no doubt signed-up for customized promotions the reality is that the information that we provide (or how these preferences are interpreted by the retailer) can only produce relevant hits once in a while. In my experience there is no better recommendation engine that the one which looks at every customer event: either purchase, on-line visit, or other interaction (e.g. call center), then uses advanced analytics (primarily event cluster – if you were wondering) to create a personalized profile that can distinguish every member of the household (so the dad does not his pregnant daughter specials) and their purchase occasions (e.g. when are you buying for yourself vs. a gift) to build a true historical profile.

In order to achieve this depth of customer knowledge, it is as important for the retailers to understand when you went to the store to buy something (which they can know by identifying you at the POS) and when you went to the store and did not buy anything and when you decided to go to a competitor’s instead -most the brick and mortar lacked the technology to identify you on this scenario. However, with the GPS technology added as a standard to all the “smart phones”, retailers have access to this data in almost real time. They can send promotions that will not only be specific tailored for you, but these promotions will only be delivered when are you physically on a location where you can act upon receiving them.  The possibilities become mind numbing: Imagine the scenario where you spend time in the store trying different outfits and then you decide not to buy anything, on the way out you receive a mobile coupon that is only valid on that visit, or the application could wait and see what other store you visit in the same shopping complex and then provide you an event better promotion that targets you as you enter the “competitor’s store”. Granted these scenarios will require you giving access to your geo-positioning to the retailer, but wouldn’t you do that to save some dollars on your favorite store?

Being able to pinpoint your location is just the beginning, thanks to Big Data, the retailers can play NSA with your purchases history and literally understand what your inventory is on hand, how old it is and how much you paid for it. These opens-up other possibilities, for example, if you are holding an Xbox® game which has strong demand and they know that there is one coming-up that you would definitively like to play, they could offer a higher trade-in amount for that game, which would lure into the store and then market to you to buy the new game at full retail price. You would be delighted and the retailer would have made good profit on your need to trade-up. In a more fashion driven example, imagine that you bought a jacket a month back which went out of fashion, the retailer could send you a reminder of the news trends and urge you to donate your old clothes (e.g. your month old jacket) to your local charity in exchange for a discount on new merchandise.  Better yet, in these scenarios, the credit for the trade-in game and the discount for new merchandise would be literally stored in your mobile device so you do not have to worry about carrying coupons or printed emails. Giving you an added level of convenience and giving the retailer a much closer relationship with you.

It is only logical that Digital Marketing, Mobile and eWallet will converge through Business Intelligence to bring a new level of personalization and convenience to shoppers, in exchange for their personal information and preferences. Reality is that this “Smart Wallet” already exists today and surely and inexorably will eventually replace our dear bills and credit cards. It is an evolution that truly started with Steve Jobs and the first iPhone and will now not stop until it has transformed the whole world. Retailers and Shoppers alike who don’t accept this new reality will be marginalized out of the digital economy, potentially taking the retailers out of business and increasing prices and reducing choices for merchandise (e.g. the shoppers will not receive discount coupons and announcements in real time).

There are still some open questions that will have to be answered, many will fear for their privacy while others will cherish the potential benefits of this deeper retailer-shopper relationship. Which one are you?

Big Data, are we there yet?

A couple of years back, I wrote my first blog on Big data: –Getting in Shape with Big Data. While almost two years have gone by – almost an eternity for the Information age standards – I keep hearing the same question from my customer visits: are we there yet?  Reality is that Big Data is just starting to deliver – Yes the value is indeed real and tangible - but the technology is on the early stages and it takes a lot of vision, expertise and sheer hard work to make a Big Data solution work. Hadoop has become the least expensive file system in the world. Many organizations are starting to use it as a cheaper Data Archival alternative with the promise that if the data needs to be retrieved the cost of storage and access will be cheaper than tape itself. While data archiving might sound like a modest use case it has created quite a revolution in the BI ecosystem. If you think about it, many large organizations kept a lot more data in the DWHSE because of the perceived loss of the data once it was moved to tape. However, with the data now being kept on Hard Drives and available with a simple Java map-reduce programs, many organizations are literally cleaning house and moving old data out of the Data Warehouse thus delaying the upgrade of the infrastructure and effectively putting some projects on hold.

From an analytics perspective adoption has been slower, while open source brings some good tools into the ecosystem – including R (for advanced analytics) and Graphite (for advanced graphics and plotting), we have yet to identify how to leverage of the power of the data stored in Hadoop for the masses. While it is true that everybody with Data and Java skills can write Map-Reduce programs, let us face it the majority of the analysts are Excel and Access gurus with the advanced population being proficient on SQL, only a very small percentage actually know how to code in Java. This lack of Java knowledge among the analysts has truly become the bottle neck for Hadoop to be an effective replacement of the Data Warehouse. However, the innovation continues and some open source projects like HIVE & PIGS continue to evolve to enhance the data consumption experience. While still in the early stages, the technology is very promising and will eventually mature to the point that Java Map-Reduce will be an exception rather than the norm for accessing data stored in Hadoop. In fact, there are technologies today, albeit commercial such as Teradata Aster, which can insulate the analysts from writing Map-Reduce and will simplify access to data stored in HDFS (Hadoop File System).

So are we there yet? Maybe not, but the speed of change is accelerating. As of now, probably most of the Fortune 500 companies are defining a big data strategy which in turn will push software, hardware and services vendors to innovate at a faster rate to close the gap against the expectations (or hype) that Big Data has created with the business stakeholders. My prediction is that in two more years (mid 2015) we will be able to access Hadoop in different ways that put the technology much closer (from a data perspective) to the relational SQL databases that we know today.