Data Discovery vs. Traditional BI

Surprisingly, it’s not “Either-Or”.

Let’s admit it… we all like to watch a good fight. Whether it’s the world heavyweight boxing championship, the Super Bowl, the Olympics, or the ongoing battle between Data Discovery and Traditional BI, it can be hugely entertaining, a lot of fun, and just plain interesting to watch the competitors duke it out, regardless of whether the arena is athletic or corporate.

Need for Speed

Data Discovery vendors such as Tableau, QlikView, and Tibco Spotfire have garnered a lot of attention and mindshare because of their ability to give business users immediate, flexible, and direct access to a variety of data sources and to create their own reports and dashboards without the need to interface with the corporate IT department.

This faster time-to-insight has proven to be very popular with employees and managers responsible for getting the answers to business questions today – possibly tomorrow – but in many cases even tomorrow won’t cut it, and certainly not next week or next month.

A New Push

Although corporations have spent billions of dollars on their IT systems, including Business Intelligence applications, they are often finding they need to spend even more to equip their business users with the Data Discovery tools which can help them make more effective use of various data assets which exist in a variety of formats and locations – and not always within the well-governed corporate data warehouse.

Large entrenched megavendors such as Microsoft, SAP, IBM, and Oracle have noticed the popularity of these Data Discovery tools and have rushed to provide their own versions of similar products, or in some cases, solutions which the megavendor hopes will provide more advanced technology than the Data Discovery vendors can offer. Traditional BI vendors such as MicroStrategy have done their best to follow suit.

Whats the Big Deal?

One of the primary drivers behind the move to Data Discovery solutions is the relatively inflexible nature of many “Traditional BI” corporate data sources. Although the IT department is often pointed to as a “bottleneck”, it is really the underlying corporate data sources which IT has to deal with which are the root cause of what is often referred to as the “IT Bottleneck”. In addition to the relatively inflexible nature of these data sources, which require time and expertise to implement or modify, another contributing factor to the IT Bottleneck are the controls, governance, accountability, scalability, performance, and security which are absolutely required for corporate data systems and applications.

Combining the inherent inflexibility of corporate data sources and systems with the need to carefully manage and protect these vital and often large-scale data assets, it is not surprising that Data Discovery solutions have provided a desirable alternative to Traditional BI for many business users.

The Data Discovery vendors are known for employing a “land and expand” sales strategy in which their initial goal is simply to “land” in an account, i.e. gain a foothold in the sales department, say, or perhaps marketing. Depending on the account, this may well be a “back door” approach in which the IT department is avoided altogether and the sale made directly to the business users. In terms of the sales strategy, it really doesn’t matter much where the initial foothold is, as the next step is to try to “expand” their presence in that account, by virtue of now having an entree to the rest of the organization via their initial departmental customers who hopefully have been able to adopt the product successfully and provide a good internal reference.

Data Discovery vs. Traditional BI

While there is no denying the success of this approach, as well as the successful implementation of Data Discovery solutions in many companies, the Data Discovery approach to BI is different from Traditional BI which is known for its strengths of governance, scalability, large-scale performance, security, and for providing a trusted “one version of the truth” source of data which can be relied on throughout the company for accurate information. However, “accurate” and “trusted” are not the same as “timely” and “flexible”, and indeed they can often be at odds with each other, forcing business users to choose which is more important. The Data Discovery vendors, to their credit, identified the need for more timely and flexible data sources and BI systems which could provide the speed and agility needed by business users.

Using the SQL Server Analysis Services BI engine, over the years our customers and partners have built thousands of BI applications, both internal and externally-facing BI applications which embody the best elements of Traditional BI: data governance, “one version of the truth”, secure, scalable systems able to handle thousands of users and large data volumes while providing the fast performance users need.

But something was missing. Frankly, from our perspective as a BI software vendor, we did not identify what was missing on our own. Fortunately, we do have a corporate culture of listening very carefully to our customers and valuing their input. We don’t think we know everything. One of the dangers, whether you are in a corporate IT department or a software vendor, is that you can fall into the trap of thinking you know far more than your users or customers do. After all, you are the ones who built the software or systems these people are using every day.

Insights From The Customers

But the truth is, it is the users, it is the customers, who are the ones “in the trenches” every day, trying to use your software to the best of their ability in order to solve real-world business issues or answer real-world business questions. With more data, and more types of data becoming available to users each year, and the world of business becoming ever more complex, interrelated, and integrated each year, it should not surprise us then that the needs of users and the types of questions they are asking have also become more complex.  This additional complexity also means that it is simply not possible for any data architecture or cube design team, no matter how well-funded or talented they may be, to anticipate all the information needs of their business users.

The “A-ha!” moment for us came when we noticed that two of our largest customers were beginning to ask for similar kinds of new functionality that was not in our Analyzer BI solution. Not only was it not in Analyzer, it was not in any other BI products. It is worth noting that both of these two customers use the externally-facing multi-tenant SaaS version of our product, which we quite cleverly call “Analyzer SaaS”.

So What?

The reason this is significant is that it means that both of these customers of ours are each serving thousands of their own external users, which are all paying customers of theirs representing significant revenue streams. Moreover, these paying customers of our customers comprise a veritable “Who’s Who” list of the world’s largest and best-known retailers, food & beverage, consumer packaged goods, pharmaceutical, and healthcare companies.

The significance of this is that these are the types of companies with large volumes of sales in extremely fast-paced and competitive environments. The career of a brand manager at these companies can hinge on even a small market share gain or loss compared to one of their main rivals, as is so memorably described by John Sculley in his book Odyssey: from Pepsi  to Apple. A Journey of Adventure, Ideas, and the Future.

What we found, or rather what our two customers were hearing from the business users at these major companies, was that there was a need for more flexible ways of defining, selecting, grouping, combining, re-combining, displaying, saving, and sharing very specific sets or combinations of data than was ever possible before. And they wanted to do this using the trusted, cleansed, governed, scalable, large-scale, and secure “Traditional BI” data sources they relied on every day for making crucial business decisions.

Can it be Done?

Frankly, when we first heard what they were asking for, our initial response (at least internally!) was “That’s just not possible to do.” It was somewhat akin to someone asking a car dealer for a car that could transform into a motorcycle or an airplane. “Not possible. Never been done before. Not what this car (or data source in our case) was ever designed to do.”

I won’t really try to describe the specifics of what they were asking for in this article, other than to say that they were asking to do things which SQL Server Analysis Services cubes were never designed to do or support, such as creating very granular and flexible sets of data based on a variety of criteria, and sometimes no criteria at all. The trick was they wanted to do all this on the fly as business users, without asking IT to create a new dimension, new named set, new hierarchy, new measure, new cube, etc. And they wanted to be able to immediately save, re-use, and share these newly-created custom data groupings and analytical patterns throughout the organization, extending the value of the work of their most talented, creative, and innovative users and analysts wherever they may be located throughout the world.

Fortunately, after many, many months of incredibly difficult and painstaking work, we discovered that our initial response of “This can’t be done” was not correct. It could be done, and we have done it, to the delight of our customers and their customers.

Recombinant BI

We needed to come up with a new name for this new type of approach to BI, which is really a bridge or hybrid approach, combining the best elements of both Data Discovery and Traditional BI. We called it “Recombinant BI”, named after “Recombinant DNA” in which new combinations of DNA sequences are created which do not normally exist in nature. In the same way, “Recombinant BI” allows business users to independently and easily create new combinations and new representations of data which do not exist in the cube or underlying data sources. The reason this can be done on the fly by business users is that these new custom groups of data and analytical patterns can be created without needing to ask IT to do anything – no changes whatsoever need to be made to the cube or the data sources which feed the cube.

Recombinant BI is enabling major retailers, consumer packaged goods companies, pharmaceutical companies, healthcare organizations, and other customers to ask previously impossible questions and get the answers immediately – even using real-time data — in large-scale, well-governed, scalable, and secure BI environments.

An example would be a CPG organization selling sweetened packaged food products who wants to evaluate whether high-fructose corn syrup sells better or worse than cane sugar in a given product, or even a customized set of products. They could easily add to this analysis other variables such as whether geography, season, or other factors influence consumer preferences for various products containing these ingredients.

Recombinant BI makes this kind of analysis easy, even though the cubes and underlying data sources were never set up with these kinds of questions in mind. As business users ask and answer these types of unexpected real-world questions, whether at their desks or in the field using their mobile devices, they then begin to think of additional ways to ask different kinds of questions and squeeze even more value out of their corporate data assets. Just as in nature, our Recombinant BI continues to evolve in response to these new requirements, and we are happy to be able to provide this powerful new hybrid approach to BI to our customers – and their customers as well.

Bob Abernethy is SVP & GM of Strategy Companion Corporation. A veteran of Oracle Corporation and Siebel Systems, Bob brings over twenty-five years of IT and software industry experience to his discussions with customers and partners about their Business Intelligence implementations. Most recently, Bob has worked closely with many Fortune 500 and mid-sized companies on the deployment of Strategy Companion’s Analyzer BI solution. Bob received his Bachelor of Science degree from Cornell University in New York and his Masters of Management Information Systems from West Coast University in Los Angeles

Selecting The Right BI Vendor:
The Ultimate Guide

Choosing a BI vendor is all about finding the right fit. Our exclusive report will walk you through the process and help you select the perfect solution.Download Now

Speak Your Mind