“Despite certain rumors to the contrary, data warehousing is thriving.”

I couldn’t agree more with Judy Ko in her recent post, in which she points out that predictions that data warehousing was going to be abstracted away — by service-oriented architecture (SOA) and other new approaches — didn’t quite pan out. Instead, if anything, the need for data warehousing solutions only continues to grow. Data volumes are growing, and businesses are demanding ever-more sophisticated business intelligence and analytics to run against that data.

If anything, approaches such as SOA promise to greatly enhance – not replace – data warehousing, “by providing more real-time data and new ways of delivering data where it’s needed,” Judy says.

SOA promises to enable an abstraction layer that can expose key data warehouse and BI application functions as standardized services that can be made accessible to all end-users anywhere across the enterprise. That enterprise reach – both for gathering and dispersing information – is SOA’s key advantage.

Judy is not alone in seeing the advantages SOA can provide to data warehousing and business intelligence.

Anne Marie Smith, director of education at EWSolutions, recently noted that the enterprise focus of SOA could be brought to bear on nascent data warehousing efforts. “To date, most SOA efforts have centered around transaction systems, but data warehousing can benefit from SOA with the ability to join various actions (services) from different areas of the DW to create composite applications or common services,” she points out. In fact, she adds, “services that are part of a data warehouse – data extraction, transformation, loading, querying, updating, etc. – should be part of the SOA from the start. This should make more comprehensive business intelligence possible, and could assist in the development of fully integrated SOA. Why have extraction and loading routines in separate environments if they are essentially similar?”

Others agree that SOAs offer an alternative to the tried-and-true extract, transfer, and load (ETL) approaches. “Implementing a BI solution by using EDA and SOA is superior to using traditional ETL,” according to Arnon Rotem-Gal-Oz. “Not only do we get our basic BI, but we actually get better, real-time BI-not to mention improvement in the overall quality of our SOA.”

Why is SOA a better alternative to ETL? As Rotem-Gal-Oz points out, “ETL is mature and has a proven record as a basis for building successful BI solutions. However, using ETL basically negates most of the benefits that made us pursue SOA in the first place. One of the main problems in the pre-SOA era (which is still the reality in many organizations) is what is known as integration spaghetti.” SOA, if implemented effectively, “promises that the enterprise data would be woven into a cohesive fabric and not some point-to-point integration spaghetti.”

Shrinivas Mandhana and Jayanta Adhikary also connected the dots between data warehousing, SOA, and BI in a recent article. “The success of a BI solution lies in creating a flexible data repository of business data assets, which will provide an integrated view of the enterprise data,” they explain. “This repository will be considered the trusted single version of truth. Once this repository is created, the next thing to focus on is how the data asset can be reused by downstream systems like business intelligence (BI) ad hoc reporting tools, transactional systems like customer relationship management (CRM), enterprise resource planning (ERP) and so on. This is where DW and SOA architectural paradigms converge; DW focuses on the data acquisition part while [data as a service] focuses on the data exploitation part.”

Once a service-enabled data warehouse and BI infrastructure is in place, this opens the door to the next phase of business intelligence and SOA, which is complex event processing (CEP). John Bates of Cambridge University, considered the father of event-driven technology, was quoted as saying CEP – which provides real-time analysis of what’s happening around the organization – is a step up “older BI approaches” that could only capture data and provide reports on an hourly, daily, or weekly basis.

Indeed, SOA has become a valuable approach for data warehouse and business intelligence. Event-Driven Architecture, or EDA, extends SOA capabilities to manage real-time event processing. As organizations move to the next generation of business intelligence, data warehouses – supported by SOA – will lead the way.