According to J.J. Green, WFED’s National Security Correspondent, the front line on fighting terrorism could be a simple matter of data integration that will allow agencies to better share information. He might be onto something:

“Intelligence sharing is not the problem for the US government when it comes to counter terrorism failures. A former Senior intelligence official said the problem is data integration.”

After looking at the facts of the case, I have to agree. Indeed, September 11^th and other attacks could be avoided if there was a common understanding of information between agencies. We seem to be missing patterns among the larger amounts of information collected, patterns not understood because we lack a well understood data sharing and data integration policy and practice between those who own the relevant data.

“The official said the bits and pieces of information on Umar Farouk Abdulmutallab, who tried to blow up a Northwest Airlines flight on Christmas Day existed, but the system to connect the dots at the speed of light when triggered by any one piece of information is not in place. It exists, the official said, because Google does it everyday, but a comparable system is not being used by the US government. [sic]”

The reasons agencies do not share vital data are many, including no working central integration projects that span agencies, which is typically due to separate budgets that are controlled by separate organizations. I suspect that some integration does exist, but it does not seem to be working. We also see this in corporate America, but the end results are much less tragic.

The truth is that data integration these days is easy, although it requires a bit of planning and selection of the right technology. I’ll go a bit beyond simply suggesting that a fundamental data integration plan be put in place, and provide the government with the steps to get there. Here goes:

· Step 0: Have a single entity control the IT budgets until this problem is addressed.

· Step 1: Create a single logical view of the data. What you need to see, and when you need to see it.

· Step 2: Locate the data sources, including data of record. This means aggregating data from many heterogeneous data sources so they adhere to the logical view of the data created in Step 1.

· Step 3: Determine data latency requirements. Or, how often the data needs to arrive from the source. Considering the amount of date we’re dealing with, this is critical.

· Step 4: Create a data quality plan. The data must be 100 percent reliable and accurate. This is hugely important when considering the consequences.

· Step 5: Select and deploy the right data integration company in and between the various agencies that own and control the data. It’s not as hard as it sounds, and the technology is here and proven.

· Step 6: Create complex event processing (CEP) system to monitor and react to changes in the data. Find conditions that should create an alert such as a guy paying cash for a ticket, and getting on a plane with no luggage. Also, somebody who is already known to intelligence agencies purchasing a ticket.

· Step 7: Conduct continuous improvements on the data integration systems. You’re never done with this step. The systems need to be monitored and a program of continuous improvement should be in place.

Sorry to be preachy, but there is no reason we can’t get this working.