Getting in shape to deal with Big Data

Big Data is a big term these days, everybody in the information management industry is talking about it; in fact, along with cloud and mobile, it has become one of the biggest items in 2011 for information professionals. But what makes Big Data so big? Are the numbers getting larger or the hard drives getting heavier? If you are thinking about getting in shape to be able to execute Big Data projects, you are not far from the truth. I am not talking about improving your muscular strength to lift heavier weights, but rethinking how we look at the availability of data for our Business Intelligence implementations. Let me explain, Big Data is a label applied to the collection of structured and un-structured data available on a particular topic, situation or subject. The implications of this are indeed big, think about a well-known company brand, such as Pepsi or Coke and picture the amount of data these companies create on a daily basis within their premises: thousands of sales, inventory, manufacturing and logistics transactions take place each day. This “internal” data might well amount to hundreds of megabytes, not a small amount by any means, but only a fraction of what you could find in blogs, review pages, discussion forums, or Facebook. In all likelihood the volume of data generated by their customers, distributors and consumers will probably be several orders of magnitude larger that the data generated within the company itself. Big Data is all about tapping into this data through all possible channels & means, but more importantly is about making sense out of it. Giving another example, if you are about to buy a product in Amazon, you will probably be tempted to read the reviews provided by previous buyers. While the primary purpose of reading these reviews is to make a buy / non-buy decision, through the reading of these reviews you will start getting a better understanding of the product, getting visibility into its strengths and weakness. This new understanding can (and will surely) shape your expectations on the item that you about to buy and help you to make better use on some features while avoiding others altogether, if you decide to buy it at all. There have been multiple technological efforts to harness Big Data, one of the most prominent, from the open source community, is Hadoop. Hadoop brings a map/reduce approach to the table through which one can explore/analyze multiple streams of data and bring them together to present a summarized result. Think about the US census where the objective is to get the country demographics. With a population of over 3 million it would take a single person a long time to visit everyone’s dwelling .The optimized approach is to break the problem in multiple units that can attack the problem in parallel so every city/county in the country will have a team to process the results for the local area, bringing them together to understand the entire population of the US. Hadoop has evolved into commercial distributions from different vendors that promise to bring a more user friendly approach to the installation and the execution of the technology. As valuable as these commercial distributions are, the real value for an organization will come from making sense out of this data, not in isolation but in combination with the data already within the organization. Getting back to our discussion of Pepsi and Coke, imagine the value that these two companies could drive (and potentially already are) from Big Data, linking what is happening in the organization with the data from the outside world to generate real time insights. While this might sound like a straightforward value proposition there are plenty of challenges on the way, and in order to successfully overcome these challenges we will need a fit, trained, mind that keeps us going without collapsing on the sheer weight of Big Data.

BI in an imperfect world: When organizations let us down

It is probably a well known fact that BI projects have the least successful completion rate in any organization: entire books, whitepapers and Blogs have been written on this topic alone with the hopes that many project teams will recognize, and more importantly avoid these issues. If you are reading this blog, chances are that you are very familiar with them, in fact you might have already experienced and (hopefully solved) some of them yourself. However there is probably one that you are not aware of, even though it maybe “hiding in plain sight”: what if the organization itself (all the way from top management) does not want you to succeed? Impossible, you might say, isn’t the organization paying for your salary and giving you a mandate to complete the program for its own benefit? Although, it might sound like a conspiracy theory, sometimes there are forces coming all the way from the top that have a mandate to protect the status quo at the expense of keeping everyone in the dark. Just recently I had the opportunity to have a good discussion on this topic with a long time friend who works for a top IT firm running a global BI program; he has been working over the last two years to gather performance scorecards on every major area in the organization. Through this process he has discovered things that are working extremely and identified areas that might be consider for improvement, some of them for significant improvement. The results were published and made accessible to all the top lever managers in the organization. You would expect that given the reputation of this global organization, they (management) would have taken the results very seriously and created specific programs to address the short comings or improvements identified in each area. Well, not only this did not happen, but my friend was called into the global CIOs office for a face-2-face meeting. During this meeting he was literally told that his scorecards were creating too much noise and making the company look bad, so he should go back to them and separate the real from other issues and more importantly fix the problems in a couple of days when the Global CIO was meeting with the CEO and the board. If you are still looking for the positive side to this story, the Global CIO offered him to take some sabbatical to evaluate his priorities. I cannot imagine how my friend must have felt after dedicating his efforts to improve the organization through Business Intelligence. I guess it must have been similar to when some people in the dark ages tried to prevent the usage of candles as it brought light to night activities that they would have preferred to keep hidden and people felt they were standing on the way of progress. Going back to our premise, the organization let down my friend, they (management) made the decision that they were better off behaving like an ostrich, hiding from their enemies rather than addressing them head on. However, I side with my friend that he did the right thing by shedding light into difficult situations rather than hiding the truth. My only advice in this situation is even if the organization is letting you down; you should never let your organization down: this is something that you might never forgive yourself as a professional true to Business Intelligence.

Are we prisoners of our own information potential?

When a baby is born all possibilities are there, as the baby matures and grows a series of decisions start limiting how the child, teenager, adult will interact with and perceive the world. If all the babies are born the same way, how is it that some become greater leaders while others struggle to get noticed, how is it that some grow to become rich and famous and others die in poverty? Is every human baby born with innate gifts that will determine who the person will become, or is it the environment that dictates how the baby will grow up? While the question of fate vs. making your own destiny has eluded the biggest minds for centuries, the same paradigm can be applied to Business Intelligence. When a company is incorporated all the possibilities are there, it can become anything: a large corporation, maybe a multinational or maybe just the best company in the block. As it starts maturing a set of decisions are made, contrary to their human counterparts, organizations do not have innate gifs, but rely on the people running them to navigate different alternatives. Once specific decisions are made, like type of product/services that will be offered, market of focus, etc the options start getting more focused, giving the possibility to the enterprise to blossom; data will be an ever present companion in this journey of growth and maturity and will enable the development of the organization from one level to the next. Sooner or later, the organization will face a situation where the analysis of this data will become key in walking the next step in the evolution of the company, at this time the company leadership might decide to foster an open culture that enables collaboration through information or revert to the human nature of relying on key individual’s instincts, thus limiting the information potential. A company, like a human being, can also build their own prison around itself protecting the gifts and weakness that were entrusted to it. While this might seem like the safest alternative, isolation typically leads to not achieving one’s full potential. Human’s personal gifts can then be compared to the information in an organization, both prosper when they are shared and they used to assist others. Breaking this self created information prison requires significant willpower and the ability to establish a vision that freeing the information will enable amazing things to happen. The future is not cast in stone, possibilities open every day before us; the question continues to be: am I responsible for my own decisions, or there is a bigger force driving each of us? The answer lies within each of us, but one thing is certain: we will be better equipped to answer this question if we have all the relevant facts (e.g information) available to us. While not all the babies will grow to be presidents, all companies with an open information culture will become places where ideas can be exchanged and options can be openly discussed, thus leading the way for the company to realize its full information potential.

The league of Information “Justice”

Summer is almost coming to an end, but not before multiple super/anti hero movies have made their way to the big screen this year, including the Green Hornet, Green Lantern, captain America and Thor, among others. At the end “good” defeated “evil” every single time; it seemed so easy that I could not resist applying some of the principles to a Business Intelligence Program. Before you stop reading and call the publisher, let me explain: if you have embarked on a business intelligence project, most likely you intended to do good as well, enlightening your organization by providing visibility they did not have… However along the way, technical limitations, organizational politics or simply unforeseen circumstances might have delayed or canceled your project altogether; If you have been in this situation, I am sure you felt that you were losing the battle against the dark forces of ignorance, your own evil nemesis.
Some battles just might be too big to win on our own, and here is where the concept of the Information justice league comes in. While at times it might seem that you are alone in your organization, and you are the only one “fighting” for the good of the program, I can assure you that there might be more allies in your organization than you realize. Independently they might also be actively fighting their own transformation battles to introduce transparency to the organization through information; in effect each individual becomes its own “vigilante” trying to bring change to their scope of control. Separately of the size and complexity of your organization, you will notice that change is difficult; regardless of the intention or the promised outcomes, the fear of the unknown might paralyze some key decision makers and prevent your project from being implemented, or even launched. On your own, you are a lone vigilante with a charter that even might be questionable from a public light; however, if you were going to join forces with other change agents within your organization, you could establish a league of “Information” justice that would gain immediate credibility given the collective experiences and track record within the company.
Together, this information league could accomplish many things, including facilitating the success of Information projects with the objectives of empowering line level manager to make well informed operational decisions, thus taking the company performance potential to the next level. While the obstacles/enemies would not have gone away, the combined “powers” of this information league would it make easier to defeat declared and subvert foes with the sword of truth and a shield of trust in the battle field of transparency, bringing enlightenment to an enterprise previously in the dark.
You might or might not feel a super hero for your organization, but if you can establish solid Business Intelligence processes and enlist the help of others along the way, you will have effectively created your own league of “Information” justice.

Is BI something you are born with?

Even as the economic conditions in the USA and in the world remain uncertain, Business Intelligence is experiencing high demand. Every day, I get at least a couple of new request for technical Business Intelligence consultants. While our company is quite large and of global nature, we still have difficulty finding the right talent for our engagements. When I talk to my friends outside of the field, they all tell me that I am sure joking, with the unemployment in the USA approaching 10% I should be getting plenty of candidates to choose from. The reality is that there are indeed plenty of candidates in the market, but very few of them are qualified or are willing to do what is required to make the project successful.
So, if people are not qualified, why not just train them? In my experience while it is relatively easy for someone to learn a BI tool, it takes a special mindset to understand BI. I have personally taught entry level BI classes and based on observation, only about 30% of the class really understood what we were trying to achieve (over and above creating a nice report or dashboard). I can assure that I am not a bad teacher, I have been part of the faculty of prestigious universities in Mexico and my teaching scores (provided by the students at the end of the semester are not bad). So, if it not the teacher, what makes BI especially difficult for people to grasp?
While there is no definitive answer, my opinion is that the technical complexity lies with the logical abstraction required to model the business events in a way that these can be stored in a relational database. Further, although the inner workings of BI are indeed technically complex, it is even more difficult to find business leaders who truly can leverage BI at its maximum.
Despite the fact that it is a human quality to be able to consume information to make decisions, if you find (or have) a business leader who understands BI and as such can get the organization to the next level through the right use of information hire or promote this person immediately, you will not regret it.

What is the balance of your personal brand in Business Intelligence?

Why a personal brand? A brand lets other people identify themselves with what we sell or service, in particular a personal brand lets other people identify who we are, what we do and more importantly get insight our track record. No matter if you are in the industry or you are consultant helping other organizations implement a BI solution, your personal brand is the enabler for your continuous growth as a professional. Business Intelligence – a 10+billion USD market – is surprisingly a small community after all. Independently of the vertical (retail, life sciences, telecommunication, etc) that your company is currently doing business at, or the technology stack that you are using, most people who enter this field will remain associated with it throughout their careers. The key questions remain, how do you build a personal brand, does it depend on you an individual, on your firm, or how well the projects that you participate in are perceived? The answer is all of the above. So, if Business Intelligence bolts some of the higher failure rates for projects, the reasons been discussed on previous (and future) blogs, how can you build your personal brand in what it looks to be a highly dangerous professional mine field?
Throughout my professional career I have been associated with many organizations. While as a person I have never significantly change when I transition organizations, I have noticed that the perception of the people in the market (especially those who have not worked with me in the past) does change, sometimes quite drastically, depending on the organization that I was joining to. This is by no means bad, but it might be a challenge that you need to understand and be ready for it. It is always easier and safer to choose to associate yourself with an organization that has already a brand built, this will certainly make things easier and you will be able to do well in projects just by calling the CIO and reminding him/her of the history of the organization. However, this easiness comes at a price, you are piggybacking on the brand of the organization that you joined, which will bring you instant “fame” at the cost of sacrificing your creativity, freedom and potentially your professional values (as you will have to follow the rules of the new organization). Compared this to a situation where you join an organization with no brand, which nobody has heard of. The potential is much more: you will need all your energy, imagination and resources to propel the organization up, you will have much more control of the situation but the effort that you will have to put in will probably be orders of magnitude higher.
While the firm’s brand will certainly define you, this blog proposes that you need to change the equation so your personal brand can influence and define the firm’s. This is easier said than done; at the end of the day no single individual is responsible for the success or failure of any initiative. Granted, you can be a big influencer (either positive or negative) but at the end of the day is team work. Your personal brand is guided and defined by your leadership style. A personal brand leadership does not need to be at CXO levels to matter, but rather is connected to everything that happens in the project: an architect providing technical leadership to a development team, an analyst providing direction to the application builder, a QA lead establishing the parameters that the application will be used to be tested. All these are examples of leadership that define and shape your personal brand.
A personal brand is like a bank account, every success increments the balance, every failure is literally paid. If this is indeed the case, what is your balance today?

The LEGO effect: Re-inventing a BI organization one brick at a time

Who does not remember the iconic LEGO toys? If you have kids (or a kids’s hearth) you probably have some of these toys lying around in the house. When I first got introduce to LEGOs I was fascinated (and somehow intimated) by the infinite possibilities of what could be built with the bricks. The bricks came in different colors, shapes, sizes and forms and yet all of them were required to build a master piece.
I consider a BI organization to be also a master piece within a company; the BI organization is directly responsible for transforming information assets into competitive advantage touching every single business process and operation, with the ability to improve performance by measuring and showcasing light into how things are done. 15 years ago there were no BI organizations as such, this function was carried by a combination of IT and business people who were beginning to explore the potential of relational models in a mainframe dominated world. Today, all Fortune 1000 companies have a group that identifies themselves as “Business Intelligence” and consulting companies have responded accordingly by aligning Information/Transformation management services in a similar fashion. However as companies grow and mature there comes a need for these companies to reinvent themselves; in particular the BI organizations play a key role before, during and after this transformation process.
Discussing this topic with industry experts who have been in the field for a while, we came to shocking yet not surprising revelation: When management is looking at how to re-invent the company they see the existing BI organization not as a uniform entity with a well defined purpose, but rather as a collection of individual bricks that have unique properties in and on themselves: Business Analyst, Data Integration, Reporting, Analytics, Corporate Performance management, Database Management, etc. When you start rationalizing a BI organization like this, it is very easy to get lost in the details. Using the LEGO analogy, when you a look at a finalized LEGO figure (e.g. a Building) you don’t see it or appreciate it from an individual brick perspective, but rather you need to look at all the bricks together to appreciate the end result.
Things get even more complex when you want to create some new shape using the existing blocks. There are different approaches one can take, you could completely disassembled the existing shape, examine each brick independently and then chose the ones you need for the new shape. The problem with this approach is that looking at each brick individually without having the vision for the new shape is not very useful. Each brick is only as good as it enables you to build the new shape you have in mind, if there is no vision, then you cannot possibly build anything other than an amorphous shape that will not survive as it has no purpose. The same is true when a company is trying to re-invent the BI organization. If management starts looking at each components of the group and deciding who they like and who they don’t like without having a firm vision for what the new organization will need to achieve, this will be a very disruptive process which at the end will provide no value to the company, or any of the individuals (bricks) involved.
In Business Intelligence, as in the LEGO world, a brick is just a brick until it is combined with other bricks to create something. It is up to management (aka the organizational architect) to define the vision of what the group needs to achieve, set the goals and strategy and have the organization aligned into that direction. The final design is beyond a simple consolidation of bricks, as it has a reason to exist that transcends the nature of each of its elements to achieve PURPOSE.

Is real time, the right time?

With the advent of the smart phones, it seems that the next wave of BI will come from “mobilizing” the business applications and putting real time information at the ever reach of the executives and decision makers. However, experience dictates that having information available in real time is not always the best option. Early in my career, when we started experimenting with real time data loads, primarily to get around the limitation of fixed time load windows, I found out (the hard way) that sometimes real time data might give more trouble than benefits. I had the opportunity to witness over an upgrade from daily loads to 5 minute batches when I was working for a retailer early in my career. The new system enabled the data warehouse to be updated from all the stores 12 times an hour. From an IT perspective it was a huge success, being able to load data from more than 200 stores in less than 300 seconds from the POS to the Data Warehouse. However, our enthusiasm was short lived when we tried our business users to acknowledge the value of this data. The feedback started coming in that our users could not make any decisions as it was hard for them to tell exactly what was on the data as there were intermittent delays in some of the stores (links/servers down, etc) that prevented from them knowing exactly what they were looking at, and because the data changed every 5 minutes, it became a very troubling exercise to catch if there were any data errors (either introduced at the source or as part of the loading process). It got so bad, that our users asked us to remove the capability and give them access only to the audited data (as of close of business yesterday). For me this was a very valuable lesson that not always faster is better. Our brains are designed to process information when we understand what we are looking at; after analyzing the data you might want to run different slices of analysis which at the end of the day you want to amount to the whole pie.
Imagine that you are driving a next generation vehicle where everything is electronic and it can communicate with your Smartphone in real time. It can give information to the second on gas levels, air pressure, oil life, etc. If the car was sending information to you every 5 minutes with the different metrics speed, avg. fuel consumption, miles left to empty,etc, you would probably start ignoring the data because it becomes non-relevant after a while. However, with this comes the added risk that if you have enough fuel for 2 miles, you will probably miss this tidbit of information among all the other data elements. You would have run out of gas and not for lack of information, but rather excess of it.
Compare this situation with the concept of right time: you get only key notifications with the right amount of notice so you can act on it, for example the car will only SMS you when there are 2 gallons of fuel left, giving you enough time to locate a gas station and refuel. As we witnessed the next mobile evolution (or revolution) in BI, we need to able to separate the fads from long term trends. If you are caught on the real time frenzied, take a deep breath and ask yourself: “Is real time the right time?“

The power of organizational politics in BI

Let us face it, how many times have you read that failure rates in BI projects are sky high and that you have a better chance of getting an upgrade in your next flight than for your BI project to succeed? Reality is that in my professional career I have found this to be a true statement, albeit not for the reasons that most people think. When you think of a failed BI project, most of us conjure an image of a developer who has no clue what he/she is doing and the computer displaying the now legendary blue screen. However, I have never seen a project being shelved because of technical issues. Granted many technical implementations are done by people who have little knowledge and thus typically fail the first time; never the less it is very “easy” to recover from a bad technical implementation: with the right team the project can be turned around in mere weeks. So, if technical glitches are not the reason for permanent failure, what is to blame? You probably guessed the answer by reading the title of this blog: politics. Politics are an inherent quality of the human race and they are always present when you have more than one individual, they permeate everything we do at home, school and work and balance the nature of our interactions with our family, friends and colleagues. So, if politics are everywhere, how come they affect BI projects more than other IT initiatives?
Business Intelligence projects have the potential to significantly upset the balance of power in the organization, both from a project and outcome perspective. Many blogs and white papers have addressed how the outcome of a BI initiative affects the balance of power; this blog however will attempt to prove how the project itself can be an agent of chance, independently of the outcome. In order to understand this hypothesis we need to take a step back and understand why the organization chooses to execute a BI project. BI projects are by nature information projects, organizations realize they need to invest in BI when they are looking for a competitive advantage that is inherent to the organization itself: Business Intelligence enables a company to use its own information (wherever this might be located) as a competitive advantage. They key is not in the what (information you have), but in the how (it will be used). Here, therefore, lies the crux to the situation, it is not the destination that matters (the organization will end up having more or less the same information), but the journey (how this information will be visualized, interpreted and acted upon) which will define success. The process of establishing these new organizational processes will be critical to how the organization will evolve, and more importantly what will it become. Thus, the person that is appointed to lead the project is not only being a project lead, in fact, depending on the scope of the BI initiative, he/she is being effectively designated as the organization architect who will build the foundation layer for a new operating model.
It is not surprising then, that politics start to play well before a BI project gets underway, with every coalition of power in the organization proposing/backing their candidate(s) who will lead the initiative, in a similar fashion to political parties determining who will be the official candidate to run for an election. In addition, given the technical complexity and magnitude of the task, the organization will also likely chose a partner that will supplement/complement the internal staff. The selection of the partner is equally complex, with many variables to be considered, including the quality of the vendor, price, reputation and brand equity. However, the successful criteria for selecting the vendor will always boil down to a matter of trust; do I trust this partner to help me achieve the organizational objectives? Not different from a political candidate choosing their second in command, Barack Obama choosing Joe Biden and John McCain choosing Sarah Paling, in the USA 2008 presidential election. While all the candidates want to win the election, it is important to recognize that the foundation for change is seeded in the campaign, not in the victory. Similarly the foundation for organizational change is established by the project team, as the project executes. The success or failure will not be in the deployment to production, but on how the project team established a clear link between the people who are running the company and the new way of visualizing, interpreting and acting on the information that the project enabled. A successful BI project will empower the people to make effective decisions thus creating a new era in, you guessed right, organizational politics.

What happens in Vegas…

I guess I should finish the title as “stays in Vegas”, but recently I came back from Microstrategy World 2011 where the key themes were dashboards and mobility for BI. The key note sessions went as far as to imply that laptops were a thing of the past and that smart phones and tablets will become the standard for accessing the internet and performing all of our transactions. The CEO of Microstrategy commented he ran the entire company from an iPhone/iPad application enabling him to keep control of the company from any location. In the back of mind, I was thinking that these themes were not new to me, two years ago working for a large CPG manufacturer we discussed the next generation on BI applications on an Iphone, the idea was truly ahead as of time, as nobody saw the device as a business tool but rather as nice “expensive toy. Going further back 10 years, when I was working for a top 10 retailer in San Antonio, we discussed how to enable the field managers by sending information to their mobile devices (back then we were experimenting with SMS and found different incompatibilities across carriers). I was thinking throughout the conference that I was not witnessing a BI vendor trying to change the market but I was seeing a BI vendor truly understanding how the market was shaping and capitalizing on a wave that is yet to reach its full potential. Granted the technology has evolved to the point that you can consume content on a Smartphone or tablet better than you can in paper, the kindle started the trend with text and the iPad popularized electronic/color versions of magazines and newspapers with Android and Apple paving the way for the next generation of consumer & business devices. We are living in exciting and connected times where the key differentiator will not be if you are receiving all the information you can, but rather only receiving the information you need when you need it in the format that is most easily consumable. So going back to the title of this post, let me finish it as follows “What happens in Vegas, came from outside of Vegas and will definitively extend beyond Vegas”