What’s missing from business intelligence? We have well-populated data warehouses and metadata repositories, we have robust data integration tools, and we have a variety of front-end BI and analytic tools. However, the challenge is to vastly increase the value of these investments by orienting them to assist people and processes in better decision-making.

I recently caught up with Neil Raden, co-author of Smart (Enough) Systems and founder of Hired Brains Inc., to get more insights on the next steps for business intelligence and analytics. We focused on two areas where Neil is researching and consulting with clients – decision management and process intelligence. Decision management explores the automation of responses based on information analysis, while process intelligence focuses on the ability to capture and analyze the range of events that shape the outcome of a business process.

As co-author of Smart (Enough) Systems, (I posted details of my interview with co-author James Taylor last month here at Informatica Perspectives), Neil has been doing a lot of exploration at the cutting edge of data management and analytics. Smart (Enough) Systems covers opportunities in the emerging decision management space.

Are organizations ready to entrust decisions entirely to machines? Neil says decision services are best left applied against situations in which large volumes of individual decisions are made on a day-to-day basis, but each decision on its own wouldn’t make a dent in the profitability or productivity of the organization. Rather, decision services should be applied against decisions “that add up to something important, but on their own, as individual decisions, really aren’t that critical.” For example, approving or denying credit to one single customer wouldn’t have a material impact on the business, he explains.

In general, the whole area of decision making is poorly understood, even though there has been a considerable amount of research conducted in academia, especially by people such as Herbert Simon, or Daniel Kahneman and Amos Tversky, but very little of it has seeped back into the commercial sector. “Now that BI has reached a point where its application is almost universal and Moore’s Law affords us some sandbox space, it is time to finally examine what it means to ‘make better decisions,’” Neil said.

This is where process intelligence needs to be eventually considered as well. Business Intelligence (BI) is often touted as an approach to gaining a more holistic view of the state of the enterprise, but Neil sees process intelligence as more capable of reading between the lines. “BI is actually very contrived,” he says. “It only looks at the data left behind by events, and certainly not all of them. It won’t look at your entire business and tell you what’s going on. You have to look at the correlations and understand.”

Process intelligence has been employed for years within manufacturing operations, and offers compelling value for the business at large, Neil says. “Data and data warehouses are the ‘footprint’ of what’s happened,” he explains. “But if you really want to know what’s happened, you have to look at the whole distribution of events, and try to understand that. And BI doesn’t do that very well.”

Although log files within data warehouses or data marts do not contain information about the path directly, “the path can be derived from a real-time, on-the-fly examination of the logs,” Neil explains. “In analyzing complex causal flow systems, path is often important. With Process Intelligence, the process trace or flow path for each process instance is known, and becomes a key analytical primitive.” For example, capturing the items purchased at a store is useful, but it doesn’t tell you anything about how the customer got to the checkout counter, what decisions they made, what offers they accepted or rejected.

Unfortunately, software to enable process intelligence isn’t widely available yet, he says – at least, not in commercially available or open formats. Neil says examples of process intelligence have existed for a number of years now, but have been confined, to specialized and proprietary environments, such as the sensor environments in process control plants. “They evaluate the different steps and what’s going on between the steps. That’s the model for Process Intelligence, but BI and data warehousing have yet to see anything like it.”

Such software, if brought to the BI and decision management space, “looks at events and sub-events, and just like decision management, understands the business processes around those events. It’s actually tricky, because business processes aren’t formal.”