Can you imagine data being less important in the future? While organizations keep many data success stories out of the general press for competitive reasons, there are plenty of success stories out there and they make for interesting reading, for example the New York Times article How Companies Learn Your Secrets.
Data volumes are increasing, and the types of data we wish to analyze is becoming more varied. On the one hand, we need to process data faster. On the other hand, we have more data to process. What to do?
Ralph Kimball defined “real-time” as “faster than your current ETL architecture can deliver your data”. In the same way, I’d define Big Data as “more than your current analytic architecture can store and process”.
Smart meters, clinical trials, call centers, complex supply chain operations, logistics, changing risk exposure: the desire to be able to visualize important business processes in an up-to-the-minute fashion to make better decisions is becoming more and more important.
So, what great thing would you attempt if you knew you could not fail?
With improvements in technology and architectural approaches, most of us are held back our pre-conceived knowledge of historical limitations. New technologies and approaches are allowing us to solve some old problems, but everything has limitations. Where are the current limitations and how are those limitations changing? Where are the current opportunities?
I find that in discussions with many organizations, the difficulty is in imagining the great things that one might try to attempt. If you were able to ask any business question, or have any business knowledge at all, what would that be? Within the realms where data in some form, anywhere, exists: what would you ask? I’ve found that the organizations making the most progress in operational intelligence are the companies using the most imagination, both in terms of the business questions they are asking, and the technically new architectural approaches they are taking.
Every layer of the traditional data warehouse architecture has been affected by improved approaches in the last 10-15 years, allowing us to tackle both operational, tactical and strategic intelligence questions. After 20 years of decision support progress and new tools, real foundations have been laid for how to architect things differently, and how to collaborate differently as between business and IT.
This is the first in a series where I’ll mostly be exploring what those technically new architectural approaches are. If I had an opportunity to re-architect my old data warehouses using newer tools and approaches, with the knowledge of successful patterns I’ve developed over the years, I’d approach decision support and operational intelligence very differently. For instance, I’d populate my ODS differently. I’d use CDC differently. I’d use checksums differently. I’d use metadata, parameterization, templates, etc., differently. The benefit would not ONLY be the ability to have operational reporting and intelligence: I would also be able to do much more better, faster, and cheaper: without compromise.