In order to realize the true benefits of a reduced database, the DBA’s will need to participate in a few cleanup activities – possibly reorganizing the database to reclaim the space.  When data is selectively removed from a database, the storage of the remaining information resembles Swiss cheese.  Reorganizing the database may or may not be a significant effort – regardless, the storage benefit will not be realized unless the space can be released at the file system.   Once the DBA’s clean up the database, the storage team may also need to participate in a series of reorganization tasks within the storage array.    These last two steps cannot be overlooked.  There will be no benefit to an ILM project if the storage cannot be freed and reused for another application or released and credited in a charge-back model.  The initial measured benefit of the project will be the amount of consumed resources before the ILM project minus the amount of consumed resources after the cleanup activities.

An Application ILM project can provide immediate relief when the storage has been released.  However, a more important aspect of an ILM project is to evaluate the behaviors that got you here to begin with.  ILM is not a technology problem, but rather a behavior problem.  Once you can identify the broken processes or find the missing checks and balances in the consumption of IT resources, make the corrections, start rewarding responsible behaviors and integrate ILM activities into your day to day operations.

Julie Lockner  President & Founder  www.CentricInfo.com

By this I mean that 2010 will be the year when the pace of Integration Competency Center implementations picks up speed and we end the year with a majority of large organizations having one or more competency centers (or COE’s).  Gartner predicted this back in 2006, and the industry is on track to prove the prediction right.

The Computerworld article goes on to say that “37% of 312 respondents said that it’s likely that their IT units will become more centralized in the next 12 months.” A recent independent research (1) report notes “…application development professionals and enterprise architects should consolidate their skilled application, process, and data integration resources into a cross-enterprise shared services organization that will deliver faster, more economical integration solutions to support cross-enterprise needs.”

The other reasons that I believe 2010 will be the Year of the ICC include:

1. While the recession is “technically” over, IT organizations are not out of the woods and the pressure to contain costs is as high as ever.
2. The macro trends of globalization, rapid growth through acquisition, increased regulatory requirements, and proactive risk management are continuing unabated.
3. The proliferation (explosion in some cases) of data has reached a critical level in many enterprises whereby the costs and risks of unmanaged information silos demands immediate attention and can’t be deferred to future years.
4. The Lean Integration book (see www.integrationfactory.com) will be published next month. This is yet another example of the growing body of knowledge in the industry for how to effectively, and successfully, implement an ICC.
5. The Informatica 9 Platform provides a step-function increase in capability which enables organizations to dramatically enhance and simplify their integration infrastructure.
6. Professional services organizations such as Informatica, Accenture, Wipro, HP, and Teradata offer a broad range of services based on leading practices.

Items 1-3 explain WHY organizations need an ICC and items 4-6 show HOW to successfully implement an enterprise-wide integration strategy. 2010 is a “perfect storm” of motive and means which is why this will be The Year of the ICC.

Notes: (1)  “Update: Evaluating Integration Alternatives,” Forrester Research, Inc., January, 2010

Problem is, an SOA-enabled infrastructure with bad data flowing through it can be useless and even dangerous. One commentator even compared SOA to a mosquito that can deliver payloads of bad data (”viral data”) at lightning speed all across the enterprise — pandemic style — before it can be stopped.

The recent launch of the SOA Data Integration Architect Community (SDIAC), an online community focused on the value of data integration and data services in agile architectures such as SOA, promises to help bring the data and SOA worlds get closer together. The organization will bring support to SOA proponents struggling to address data quality issues and data architects seeking to service-enable their environments.

But what will the role of the data warehouse be in a data-driven service oriented architecture? Where and how will these behemoths fit? Seeking to address this piece of the architecture puzzle, long-time data warehousing expert Barry Devlin observes that the role of the traditional data warehouse has been supplemented to some degree by initiatives such as SOA and Web/Enterprise 2.0.

More so than data warehouses, SOA and Web/Enterprise 2.0 are re-arranging the relationships and boundaries between operational, informational, and collaborative systems. They bring in data from various sources across the enterprise, and provide new ways for end-users to manipulate and mine the information. He notes that SOA, for example, is bringing modular, plug-and-play approaches into the equation, giving rise to possibilities of new types of analysis applications being integrated into various enterprise applications.

The data warehouse approach fell out of vogue, as it became untenable to bring together so much information exploding all across the enterprise into a single, centralized database.

Devlin urges that this model be revisited, however – in an updated form. He observes, in line with Occam's razor-style thinking, that a data warehouse architecture is the simplest solution to an increasingly complex issue:

“The simpler the answer, the better the solution. If you want to create a consistent, integrated information resource, you must stop creating duplicates of existing information that have to be managed to consistency, and you must eliminate, or substantially reduce, existing data duplication. The original data warehouse architecture did this. It proposed a logically single data store – the business data warehouse – modeled at the enterprise level as the consistent and integrated source of all information for decision making. This simplicity was ultimately lost with the emergence of the layered architecture, due to a combination of database performance and enterprise modeling issues.”

Devlin proposes an architectural approach he calls “Business Integrated Insight” (or BI 2). This is an architecture that “gathers all the information of the enterprise, hard and soft, operational, informational and collaborative into a single logical component.” A second integration point consists of “a logically single, consistent and integrated set of processes that spans all aspects of business and IT needs is required to enable the flexibility and adaptability that modern business requires.”

Devlin also posted a white paper explaining Business Integrated Insight architecture in more detail.

Essentially, Devlin proposes a three-tier architecture:

• Layer 1 – Business Information Resource: This is the single foundational layer with all information —operational, informational and collaborative information, as well as metadata.
• Layer 2 – Business Function Assembly: “The middle layer consists of processes, workflows, tasks, applications, tools and so on: in short, all functionality and processing that runs on the business’ computers.”
• Layer 3 – Personal Action Domain: This is the interface layer to all business users within the company.

The value of such a simplified view is to provide an new architected approach to support the demands on businesses that  are highly interlinked and interdependent, with entwined processes and pressured reaction times, Devlin says. “Business and IT, in particular, need to take a fully-integrated view of what is required from its IT environment.”

While it's uncertain whether Devlin's proposal will catch on, he has pinpointed one of the greatest challenges in bringing together multiple initiatives across the enterprise, involving applications and end-user interfaces. Effective data management can't happen in a vacuum, and enterprises are begging for simpler, more straightforward architectural solutions to out-of-control complexity.

In fact, what currently limits the number of cloud deployments is the lack of a clear understanding of data integration in the context of cloud computing. This is a rather easy problem to solve, but it’s often an afterthought.

The core issue is that cloud computing providers, other than Salesforce.com, don’t consider integration. Perhaps they are thinking, “If you use our cloud, then there is no reason to sync your data back to your enterprise. After all, we’re the final destination for your enterprise data, right?” Wrong.

Cloud computing is a great way to go in many instances, but if you think you don’t have to provide core data integration from the cloud computing platforms to the on-premise systems, you have another thing coming.

Indeed, there are many reasons to provide data integration within cloud computing problem domains. The first and foremost is that you need to maintain an up-to-date copy of your enterprise data on-premise in case of trouble that could come in the form of cloud computing outages, clouds going out of business, or clouds purchased by companies that have no interest in staying in that business. There are many examples of these occurrences now, and they will only get worse in the future.

However, the primary purpose of data integration within the context of cloud computing is to assist in driving processes between systems, on-premise and cloud-delivered, and to manage data integration across those very different and geographically dispersed platforms. Those who only had to deal with systems talking intra-data center have some new challenges when they consider cloud computing.

These are the problem domains that require, dare I say it, data integration architecture and strategic technology. This means that you need to consider all of the source and target schemas, and how you’re going to securely and reliably move data between those points, accounting for the differences on the fly. Moreover, you need to consider MDM issues, as mentioned above, as well, as security, and data governance.

Here are a few words of advice:

First, consider the overall requirements of the business. Sounds obvious, but many who deploy cloud computing systems do not have a complete understanding of the overall business requirements.

Second, focus on the holistic architecture, on-premise and cloud-delivered, including how they will and should exchange data to support the core business.

Finally, select the right data integration technology for the job, and do so only after taking everything into account. You’ll find that there are both on-premise and on-demand options, and in many instances you may have to mix and match solutions.

You can’t do the cloud without data integration, and until we get data integration right, you can’t do the cloud. Pretty simple, if you think about it.

There is an apparent lag between recovery in the private sector and a state receiving tax revenue. So what can states do about this problem while they suffer in the red? Mr. Denning said, "Since states can't run general funding deficits, closing gaps mean raising taxes, cutting services and resorting to one-time measures." Mr. Denning's list of solutions is certainly accurate, but does it include all options that states have? What about employing new technology to discover fraud or recover uncollected revenue?

The many disparate systems, data silos, and various records throughout any single Department of Revenue can result in missed revenue opportunities. Using technology to integrate these various systems and records to obtain a single view of tax payer across these systems is a first step in recovering lost revenue—refilling those coffers!

In a recent webinar Government Technology presented how new technology can be used for improving data quality to reduce fraud and increase revenue.

Are there other ways states can use technology to refill their coffers?

In my last posting, I talked about the Dr. Jekyll/Mr. Hyde nature of data—data being both an asset and a liability. The CIOs talked about playing offense vs. defense. They really wanted to be engaged with business transformation and growth initiatives. But many were saddled with cumbersome, expensive IT environments that sucked up 90% of their IT spend and resources just to “keep the lights on”. In other words, they were spending almost all their time and energy playing defense.

A key reason why CIOs get stuck in defense mode is because data has become a liability. Their information and integration infrastructures are crumbling under the weight of the explosion in data, the rapid acceleration of business demands and the massive complexity and brittleness of the “integration hairball” present in the vast majority of enterprises. As chief information officers, information is their key responsibility, and they’re unable to get all the relevant data to the business when it’s needed.

To play offense, to drive the business instead of merely supporting it, CIOs need to start chipping away at the hairball, incrementally implementing a far more rational approach to data integration and information management. They need to make it a priority for the IT organization. They need to put the right people in place who have this as their sole focus. They need to make it clear, from the top down, that the status quo for tackling integration and data isn’t good enough any more. In short, if CIOs don’t want to continue to be back on their heels in a defensive posture, data integration needs to become a CIO priority.

Next posting I’ll talk about why data is transcendent over applications.

By moving a bulk of the information out of the production database, the application’s performance will improve drastically.  Additionally, all copies made for test, development and backup will be that much smaller reducing infrastructure needs and cost.

With the aged data residing in an archive database, you can then execute the data destruction process and start purging data in a central environment.  If you can consolidate all of your database records to a single archive, searching for records during an eDiscovery is easier because all of the structured content resides in one place. For operational data that has shorter retention periods, this information should be cleaned out of the source database on a regular, frequent schedule.

Before you can claim victory, there are a few cleanup steps that need to occur before you can realize the true benefit of deleting database data.  In our next and final blog of this series, we will review the final necessary tasks and then calculate our benefit.

Julie Lockner, President & Founder, CentricInfo

The Informatica Marketplace has generated a lot of buzz with our customers and partners. Last week while I was presenting at the Pacific Crest Annual Cloud Computing Conference I heard the excitement about our new offering from many of the attendees. Mostly, people appreciated our continued thought leadership and commitment to providing an open platform to host solutions for data integration, data quality and data management. And yes, it is an open platform to host solutions that not only support Informatica, but other vendors too - even competitors. Our fundamental belief is that the hand of free enterprise will ultimately win, so we are willing to provide an open platform to do so.

Here are a few questions I've heard and answered over the last few weeks:

What is the purpose of the Marketplace? Our interest is to serve the needs of the broader data integration ecosystem. Informatica itself has over 52,000 active members in our TechNet developer network today. These members asked us to provide a platform for them to build and interact. Given our 15-year heritage, use cases have now been repeated and solutions have been built. Now, sellers can share these solutions with interested buyers. For buyers, they will enjoy faster time to market with high quality solutions evaluated and certified by Informatica. For sellers, they will have an open channel to reach buyers. Our goal is to promote the Marketplace community where proceeds will be reinvested into funding it.

Why is Informatica launching a Marketplace now? As mentioned, we have over 52,000 active members in our developer network. This community has existed for more than 10 years. Within this community, developers share downloads, but have no means to monetize their solutions, nor were they reviewed by anyone. The Informatica Marketplace provides the mechanism to monetize a solution. The bottom line is that our customers have asked for this capability and existing, successful Marketplaces (like salesforce.com's AppExchange) validate their success.

Can anyone contribute to the Marketplace and what can be contributed? Anyone with solutions around data integration, data quality and data management can participate. As described previously, this is an open market. We envision solutions will be vertically or platform (horizontally) focused. They may be data models, mappings, mapplets, tools/utilities, methodologies, white papers, connectors, etc. We believe the wisdom of crowds will prevail and market demand will dictate which solutions will be offered. Our initial approach is to work with our User Groups to determine the solutions they need and want, prioritize the responses and recruit developers to fulfill these needs. The "How to Contribute" web page is going live in mid-March and we will begin posting the most popular solutions on the site so developers can select the ones they want to support.

Will assets be validated and what do you envision assets to sell for? The Informatica brand is known for its quality. In keeping with our brand image, all assets on the Informatica Marketplace will be evaluated in order to receive a "seal of approval" or certification. This certification will have different criteria depending on the type of solution contributed. We expect solutions will range in cost from $0 to the tens of thousands of dollars. Regardless, all products will be evaluated for quality and value before becoming available.

How many solution do you expect at launch? Our goal is to launch with 25 high demand, high quality solutions. We have had the opportunity to speak with leaders of other successful marketplaces and the consistent feedback is to focus on quality over volume at launch.

Does this replace TechNet? The TechNet download capabilities will move to the Informatica Marketplace. However, TechNet still provides tremendous value by enabling the developer community to collaborate and the core purpose of TechNet will remain in place to provide an inclusive, independent and interactive forum for sharing data integration knowledge, showcasing solutions, impacting technology trends and developing cutting-edge skills.

Will Informatica employees contribute? Yes, we are creating a group called Informatica Labs for our employees to contribute solutions. These solutions will be free and all contributions will be identified with a "labs" logo.

There are many more questions to be answered. My commitment is to answer them as quickly as possible. We realize that we must be fluid in our ability to change the Marketplace dynamically to meet the demands of our buyers and sellers.

Do you have any questions for me?

Bay & Bay is using the turnkey, on-demand Informatica Cloud Services to provide robust, bi-directional synchronization between its logistics management databases and applications with Salesforce CRM. Using an intuitive web based integration wizard, the company automatically maps source and target fields, configures powerful data transformations, and sets automated synchronization schedules. And being a true multi-tenant cloud-based service, there is no hardware, software, or infrastructure for them to install, manage, or maintain.

According to Rob Adams, CIO at Bay and Bay Transportation:

“If Bay & Bay didn't have Informatica, we couldn't have done any customer relationship management. By taking charge of our sales operations and putting all the sales information in one place, the Bay & Bay sales teams are devoting less time to administration and spending more time closing deals. Our sales managers can also use the real-time visibility into their team’s activities to forecast sales more effectively. Overall, if Bay & Bay achieves a one percent increase in sales revenues from this Informatica implementation, it will translate into a 900 percent return on investment.”

Earlier this year Chris Boorman predicted that, “Cloud Data Integration will eclipse both security and availability to be the defining capability that drives Cloud Computing success.” Informatica was also recognized as the top-ranked data integration application on the AppExchange in 2009. Ron Papas, Informatica Cloud general manager, explains the impact of Informatica Cloud Services this way:

“Informatica Cloud Services helps growing companies, such as Bay & Bay, quickly and cost-effectively address today’s data management challenges regardless of their business systems landscape, while retaining control and doing so in a secure manner.”

Check out our cloud customer success stories to learn more.

Data is a critical business asset. It used to be that physical assets—equipment, land, inventory—determined the value of an organization. Now, as we have moved into the “information age”, most organizations have come to realize that their data and information are also critical assets. But few have set themselves up to make sure they are managing and utilizing their data in an optimal way to really impact how they run their business. This includes the people, processes and technology needed to know where the data resides, to understand it, to clean it and keep it clean, to get it to where it is needed, when and how it is needed.

What is the value of data to an organization? More specifically, what is the value of effectively managing and utilizing that data? Armed with data that is timely and data that they can trust, organizations can rapidly uncover new markets, attract and retain valuable customers, eliminate costly operational errors and delays, deliver products faster and make smarter business decisions. In short, they can consistently outperform their competitors. These enterprises are data-driven.

Data-driven enterprises maximize the business value of their data by establishing the organization, processes and infrastructure necessary to manage their data as a strategic asset, ensuring that relevant, trusted data can be delivered quickly when, where and how needed to support the changing needs of the business. These firms will be referred to as Agile Data-Driven Enterprises.

To accomplish this vision, organizations will adopt formal management disciplines such as Lean Integration and data valuation accounting. At the present time, these practices appear to be leading edge – or even bleeding edge. In a few years they will be commonplace. To quote Erwin Drahl from Informatica, “Lean Integration is a discovery not an invention. That is why everyone will eventually be doing it.”

For the record, here is the complete list of postings on this series:

1. Should Data be on your Balance Sheet?
2. Valuing Data Using Managerial Accounting Practices
3. Calculating EVA For Data Assets
4. A Market-based Approach To Valuing Data
5. Case Study In Managing Data Assets
6. Data as a Liability