This blog post is part two of an ongoing series highlighting the importance of data in a Service-Oriented Architecture (SOA). I look forward to hearing your thoughts and input on the subject.

I’m back. It’s been a little longer than normal, longer than I would have liked. Perhaps that’s because ‘addressing SOA’s data-centric pitfalls’ isn’t easy. (Really it’s because I’ve been working on other things. But let’s get back to the topic at hand.)

One of the benefits of the SOA approach is the ability to think top-down about problems. The usual approach is to work tightly with the business to define your processes from a business perspective, leading to clearly defined services that the business understands and you can implement together.

This is wonderful and has a clarifying symmetry that Software Engineering has been trying to achieve since the days of CASE. But now, here we are in 2008 with the SOA standards defined and the tools available to potentially achieve this vision. Ah, finally, the integration hairball will be contained and life will improve immeasurably for all!

But as I talked about last time, one of the reasons that things aren’t that simple is the data-centric pitfalls. And addressing this problem is not easy if you want to take a long-term, enterprise-oriented approach.

In talking with folks who have walked down this path, struggled with data problems, and are trying to think holistically about a workable longer-term solution, three themes come up again and again:

1. You must work top-down and bottom up at the same time.
2. A Data Governance initiative will help to improve quality and oversight of your key data by getting the business involved with solving data-centric problems.
3. Some sort of Master Data Management initiative can help to provide key data for your services, such that the resulting building blocks you create will be more successfully reusable.

I’ll dive into these more deeply in the next section.

Top-Down Design and Bottom-Up Design: Certainly working with the business to define your business processes and the necessary resulting services that are required to make these processes work is a foundational approach in the SOA methodology that has multiple benefits. Besides documenting how the business works, top-down documentation has the more important benefit of getting IT and the business on the same page, speaking the same language. But time and again, customers have expressed to me the importance of simultaneously working the bottom-up angle with the top-down design.

With reference to my previous blog posting on the data-centric pitfalls, doing the bottom-up investigations means to investigate the sources of the data, to define the semantics of the data in question, to investigate the change history of the meanings of this data, and to understand the data quality of the data you are working with. Certainly, a data profiling tool can be very important to do this efficiently and correctly. But it can also be useful to pull the resulting information into a Business Glossary where you document this information.

Data Governance: The fruits of your bottom-up data investigation with the business should be documented and maintained as an important resource for the company. This provides a simple impetus to getting a Data Governance initiative going, such that you begin working with the business to improve and maintain the quality, definitions, availability and auditability of your data in a more systematic way. In other words, are you doing your bottom-up investigation and losing the knowledge learned into the ether of the enterprise? Or is this knowledge being documented into a Metadata Management repository that can be searched and understood by the business and IT into the future?

It is no coincidence that Data Governance has grown up as a topic just after people started gaining SOA experience. Certainly, whether it was called Data Governance or not, some organizations have been taking cracks at “data governance” for a couple of decades, in some cases. But now, “data governance” has become “Data Governance” and there are conferences and trade shows for this topic alone. Coincidence? I think not.

Master Data Management: In a similar vein, it is also no coincidence that Master Data Management has sprouted in this same epoch in the continuum of software engineering. Master Data Management should be seen as a means to an end, not an end in itself. Managing master data can provide a structure and approach to solving some (but not all) of the data-centric pitfalls outlined in the previous posting.

Certainly, Data Governance and Master Data Management carry lots of baggage and imply large initiatives on their own. The key takeaway is that the data problems and complexities that exist were not created, nor will they be solved overnight. Companies are attempting to institute better governance and controls to create repeatable approaches to getting a handle on these issues, and these controls can greatly help SOA initiatives if all these efforts are viewed as being holistically complementary.

However, DG and MDM do not necessarily have to mean multiple simultaneous ocean-boiling efforts to solve all of an organization’s data problems. Starting small but thinking long-term is always the best approach. And doing something rather than allowing entropy to continue to reign will provide significant benefits.

In this posting, I’ve talked about three high-level approaches to handling the data-centric pitfalls in SOA. Next time I’ll talk about how BPM is a related topic that has also suffered due to similar data issues. Read all about it in the next post.

Next up “BPM is Missing the Data”