Informatica Enterprise Data Management

SOA has nothing to do with technology. It has everything to do with defining and managing the business as a collection of service functions and information exchanges. A business may be viewed as an internal organizational unit within a large company that provides services to other other units and consumes services from yet other groups. Or if you view the business as the company overall at the macro level, then you could define and manage the business as a collection of services consumed from its supply chain and provided to its customers. [Read more]

Many thanks to Loraine Lawson who wrote an insightful article on her blog Classifying Integration Initiatives as Tactical or Strategic Her posting got me thinking. 20 years ago we didn’t need Integration Competency Centers – now we do. What changed?

From my perspective, there are three key drivers:

First, organizations have gotten bigger. The Fortune 500 companies today are much larger in terms of revenue, number of employees and global operations than Fortune 500 companies in the 1980’s. This conclusion is based on personal experience and anecdotal evidence. If readers of this column have seen any studies or statistics on the growth of large enterprises, please speak up.

Second, data diversity and volume has been growing at an exponential rate. We now have data in multiple formats such as video, images, audio, and text to name a few – and in multiple protocol formats and multiple languages. On the volume front, I recall an article from a few years ago that stated that manufacturers of storage devices would ship more than 22 exabytes (22 million trillion bytes) of hard disk capacity in 2005 - which is four times the space needed to store every word ever spoken by every human who has ever lived. In a relatively short number of years we’ve seen data storage capacities move from megabytes, to gigabytes and terabytes – now we’re talking in petabytes and exabytes. And coming soon are zettabytes and yottabytes.

Third, information technology has become more complex. One of the first principles of computer science is that re-use and decoupling of components is enabled by adding an abstraction layer. When I started my career (roughly 30 years ago), I was loading programs into computers using paper tape readers and toggle switches (I was pretty good at reading octal and hexadecimal code in my prime). But with increasing layers of abstraction – macro code to structured programming languages to 4^th generation languages to graphical integrated development tools to cloud computing in a web 2.0 environment – we have increasingly distanced the user from the underlying hardware and micro-code. Computers basically still work the same as they did 30 years (binary logic gates), but they are much more powerful, easy to use, and interchangeable which has been enabled by layer upon layer of abstractions.

The common elements across these drivers is constant change and growing incompatibility and complexity. Business processes are constantly changing with broader global scope adding many complications that are not transparent. Data definitions are constantly morphing and volumes are increasing to massive levels as everything becomes digitized and more regulated. Technology waves are constantly adding new variations and complexities without retiring prior generations.

Ergo, we have seen the emergence of integration as a new discipline which didn’t exist 20 years ago because it wasn’t needed then. In the 1980’s, integration was viewed as a one-time project discipline, not as an ongoing capability that an organization needed to sustain efficient end-to-end operations in a complex pattern of incompatible components that are constantly changing.

We could say that projects require micro-integration techniques while the complex systems-of-systems that emerge over time at the enterprise and supply chain scale require macro-integration best practices such as Integration Competency Centers. The ICC disciplines are a powerful capability, but they are non-trivial and hence those organizations that do it better than others will have a competitive advantage.

Integration is a discipline. That is, the task of making independent and autonomous entities work together in a synergistic fashion requires unique skills and practices within the context of an enterprise strategy. You may well ask "isn't integration just the same thing as management?" Not really. Management disciplines are clearly one aspect of an integration discipline, but they are insufficient because of two key characteristics: incompatibility and independence. The variations in data and process definitions across software packages and across organizational groups in combination with the autonomous nature of each link in the flow of data drives the need for a specialized management practice.

The autonomous nature of each link in the chain means that we have limited control of all the elements that make up an end-to-end operation. Clearly we have limited control when we partner with another business in the supply chain and we certainly can't control all the regulatory demands, but surely we have control within our own organization so management practices alone should suffice - but do they? While this may have been true years ago in vertically integrated organizations with more autocratic cultures, times have changed. For starters, many large organizations today operate more like a federation than a dictatorship. And the people that work in the organizations have a mind of their own, come from all over the world (and even work all over the world) and often aren't permanent employees of the company. Traditional management practices are no longer sufficient in this new environment.

Integration is a distinct discipline that seeks to find the optimal balance between efficiency in a complex chain of responsibility while remaining adaptable to change and without stifling innovation at each link in the chain. Integration demands unique practices that build on traditional management practices. For example, integration techniques are often more of an agreement process rather than an analytical process.

Integration as a process can be done manually or with the assistance of technology. The internet is a good example of highly automated integration. A person can connect to the internet, discover an unused website name, register it, and create the website; within a short time search engines will index the new site so that others can find it. Voila - you are in business. All this is done totally automatically without human intervention because of common standards, protocols and languages.

What are implications for individual enterprises? Effective business integration can drive bottom line results and if done well, integration will:

• Lower ongoing costs and one-time initiative costs by reducingduplication and redundancy in systems, data and business processes.
• Reduce time to market for new product introductions by developing standardized business services layer that decouples agile business processes from the underlying technical implementation.
• Drive revenue by allowing individual functional units to innovatewhile still leveraging common shared functions.
• Improve customer satisfaction by providing an integrated view of the customer and increased reliability resulting from a more simplified technical infrastructure.
• Improve the employee's quality of life by reducing frustrations of "swivel chair integration" and providing systems that provide the information they need, when they need it and where they need it to do their work.

While these benefits sound great, there are in fact significant implications for achieving them. At a business operational level achieving these benefits requires:

• New organizational structures and relationships such ascross-functional teams with authority to make trade-off decisions. For example, providing product bundles where individual product managers may see lower margins, but the enterprise overall benefits from increased wallet share.
• Changes to metrics, goals and rewards. For example, when just about any function can be performed in just about any channel, you may need to change incentives to encourage store sales staff to steer customers to lower-cost channels. And you may need to add metrics that track cross-functional sharing and reuse and reward managers accordingly.
• Revamped governance processes for funding of cross-functionalmulti-year initiatives. Single-project and single-iteration budget decisions need to be supplemented with planning and governance that takes a two-to-three-year view of application portfolio upgrades.
• More effective communications between functional groups and across geographic regions. And just as important, a closer ongoing strategic working relationship between business functions and IT functions.
• Common vocabulary for data and business processes to enable one version of the truth. The key to process integration is data standardization. As standardized data are made available, business owners can effectively integrate their processes.
• A 360 degree view of customers. Simplified and personalized presentation is a powerful combination that leads to greater customer loyalty and retention.

For the IT organization the implications are no less significant.

• An integration group is needed which is a permanent part of theorganization.
• A defined application portfolio is required with clear accountability across business and IT functions.
• Strict standards for process and data interactions across applications. While you can continue to allow significant autonomy of decisions within an application domain, the critical nature of how all systems are connected with each other requires tight controls over their interactions.
• The need for an integration methodology in addition to change management and software implementation processes. Integration initiatives must start with a definition of the end-to-end business processes and be based on a common business language.
• Implementation of new technology and tools to enable end-to-end dataintegration.
• Development of staff skills in formal modeling techniques and tools.The skills required to abstract functionality and generalize business concepts for enterprise-wide use are not always common in technical staff and must be developed.
• Standards compliance as a matter of principle rather than a matter of convenience. Standards that support the principle of loose coupling between systems are enablers for an agile business.

These capabilities are enabled by a fundamental understanding that integration is a distinct discipline. Organizations that view integration as a "strategy" and that focus their people, policies and investments around the strategy will have a clear competitive advantage. They will create an agile business where each link in the chain can change and adapt to meet local needs while the end-to-end chain remains strongly aligned with the overall operating model.

Are BI managers and professionals sometimes too eager to please the business? Are centralized BI efforts slowing down progress? Should BI teams address requirements before the business even asks for them? These questions may seem counter-intuitive, but Wayne Eckerson, director of research for TDWI, says that the best intentions for BI efforts in many organizations may actually result in sluggish projects, duplication of effort, and misaligned priorities between BI teams and the business. [Read more]

Just about everyone has heard about Nicholas Carr’s Harvard Business Review article titled IT Doesn’t Matter. I’ve heard many reactions to the paper presented at conferences and in the blog-o-sphere, so I figured it was time to go to the source and purchased IT Doesn't Matter (HBR OnPoint Enhanced Edition) from Amazon for $6.50 as a downloadable pdf. The paper is excellent in that it presents not only the original paper first written in 2003, but also features no fewer that 14 reactionary letters plus a final response from Carr to the letters [Read more]

You may need to reorganize your company to take advantage of modern IT architectures such as SOA or MDM. Some people would say this is heresy – that this is a case of the tail wagging the dog and that the IT strategy should echo the business strategy and operating model. True enough – but IT is also “part” of the business and also has an obligation to play a role in leading the enterprise into the future. [Read more]

It’s been almost 5 years since Nicolas Carr started a lively (and sometimes hostile) debate with his Harvard Business Review article IT Doesn’t Matter. And now, just this past fall, Cynthia Rettig added fuel to the fire with her MIT Sloan Management Review paper The Trouble with Enterprise Software. There are indeed serious concerns with the perception of IT and many business leaders are fed up. [Read more]

I was asked a question recently that got me thinking – “Why would companies want to spend money on middleware during a down market?” My knee-jerk reaction was to recite all the reasons that companies might want to invest in integration technologies in order to get more life out of every IT budget dollar – but then I considered a deeper question. [Read more]

Nicholas Carr’s 2003 article “IT Doesn’t Matter” in the Harvard Business Review is still getting attention five years later. I for one won’t let it drop because the article is based on an incorrect premise and misrepresents the IT challenges that business executives should be aware of. [Read more]

The recent Informatica Release 8.5 launch highlighted Real-time Integration Competency Centers (ICCs) as the optimal model for successful data integration. I’d like to review the concept of the Real-time ICC and why Release 8.5 supports this advanced operational, organizational and technology model.

As data integration moves beyond the realm of data warehousing into operational integration, real-time and data services use cases have exploded in importance to the business and necessitated stronger, unified infrastructure for IT to meet the challenge. Philip Russom, Senior Manager, TDWI Research captures this trend specifically in his quote on Release 8.5.

"The movement toward real-time data access and delivery has been the most influential trend in data integration this decade. The trend has enabled user organizations to initiate a variety of valuable real-time practices, including operational BI, real-time data warehousing, on-demand computing, performance monitoring, just-in-time inventory, and so on. And the trend has led vendors to extend their data integration products, so that many functions operate in real-time, not just batch. Informatica 8.5 is a great example of this trend, because it’s re-architected to support more real-time and on-demand functions for data integration, changed data capture, and data quality." [Read more]