Monthly Archives: September 2008
How SOA Enhances Data Warehousing and Business Intelligence
“Despite certain rumors to the contrary, data warehousing is thriving.”
I couldn’t agree more with Judy Ko in her recent post, in which she points out that predictions that data warehousing was going to be abstracted away — by service-oriented architecture (SOA) and other new approaches — didn’t quite pan out. Instead, if anything, the need for data warehousing solutions only continues to grow. Data volumes are growing, and businesses are demanding ever-more sophisticated business intelligence and analytics to run against that data.
If anything, approaches such as SOA promise to greatly enhance – not replace – data warehousing, (more…)
Data and Processes are Intertwined!
In one of my earlier posts I discussed the need for a sophisticated data services-driven technology serving as the foundation for SOA and BPM.
When I was poking around the web recently, I ran into a powerful statement by Michael Blechar from Gartner, covered in the DAMA keynote, titled Survival of the Data Management Fittest:
“Data and processes are intertwined. It will fundamentally change the way organizations think about your roles, and your roles are going to need to evolve”.
At this year’s Data Management Association (DAMA) International Symposium,
Michael is quoted saying that:
“In this world there’s a very loosely coupled user interface from the assembled services that in turn share access to data. SOA exposes data issues to more people, places and processes, and what I tell companies is that without a focus on information management and meta data management they’re going to fail.”
It is in speaking to numerous customers, prospects and technologists that I had gathered that without accurate, consistent and timely information, SOA and BPM deployments will face serious information-centric hurdles, affecting the cost-effectiveness and success of the project. As we move towards more agile architectures, I believe that we need to grow typical process-centric approaches to include information centricity as well.
As Michael states:
“Where we are going is beyond the first generation of BPM and SOA [that is process-centric],” he said, “to the next generation of SOA that is information-centric.”
Observe that the key word here is “information-centric.” Reading such statements from Michael and many others definitely validates the strategy I have been defining for building out an effective IT infrastructure that can benefit from the flexibility of a services and process-driven approach, in the data integration layer. Simply wrapping data access with a web service does not qualify as a sophisticated data service and hence, stringing together such simple services with a BPM tool also does not guarantee agility.
As discussed in Services to Orient your Enterprise Data Layer, Joe McKendrick is of the opinion that neither SOA nor enterprise-application integration alone can effectively handle the enterprise data layer. However, data services delivered within an SOA framework can create a data-abstraction layer to address the complexities seen across enterprise data environments.
I have always said that without serving up good quality, consistent and timely information as a data service or a comprehensive data service built using a sophisticated data integration platform, SOA and BPM deployments will not be able to deliver on their promise of agility.
What are your experiences? What kind of information-centric issues have you run into in your service-oriented deployments? Is inaccurate, stale and inconsistent information passing through your IT infrastructure holding you back?
Setting up ICC across BI and EAI practice- Round 1 Breaking the Silos
To date many companies have fragmented their integration efforts across applications and groups. The classic split is between the group developing data warehouse and business intelligence applications versus the ERP (enterprise resource application) applications. Typically with this fragmentation different integration approaches are taken with ETL being what is used with data warehousing /business intelligence and either EAI or EII used with the ERP applications. In addition to these integration silos, many companies today have, or are launching SOA (service oriented architecture) initiatives generally independent of either the data warehousing/business intelligence or ERP applications.
If achieving consistent numbers across reporting is a goal maybe the investments should be in data integration. When people look at inconsistent reports or analysis that use different BI tools it is easy to understand why they assume the problem is using different tools. Using different BI tools, however, is a symptom rather than the reason for the inconsistent numbers. The symptom is silos using different BI tools but the underlying reason are data silos created using different data integration tools, processes, standards and people. The best practice is to establish an ICC (Integration Competency Center.)
ICCs also need to take into account the BI practice. There has been a good deal of attention paid to BI tools over the years. Many companies have instituted a BI-CC (business intelligence competency center), a concept promoted by Gartner research. (more…)
Data Governance and SOA Governance are Interlinked
Governance is a tricky and ill-defined area. For example, in the emerging SOA space, listen to the drumbeat of messages from consultants, analysts, and vendors, and the message is clear: Service oriented architecture won’t work without governance.
However, establishing effective governance has been a vexing challenge, with a lot of disagreement and debate amongst governance proponents. (more…)
IT-Business Collaboration: Is It Always “Lost in Translation”?
All of us in the data management and integration arena know how difficult and time-consuming it can be to understand business requirements and ensure that data warehousing and other implementations actually meet the real needs of the business. How many of us have spent days or even weeks in tedious requirements gathering sessions, asking what the business wants, and getting very fuzzy answers back? It’s hard to ask the right questions which the business can actually answer, and even then, it’s highly likely that the documented requirements are incomplete or inaccurate. Dan Lindstedt has an intriguing posting on the topic of business requirements that’s well worth a read.
One of the fundamental problems is that the business does understand their data, what’s wrong with it, and what they need to be changed. But translating that into language that IT can understand can be a very tedious and often precarious endeavor. So why not short-circuit the translation process?
(more…)
Moore’s Law for Organizations
Fortune Magazine listed Walmart as the largest company on the Fortune 500 list this year with annual revenue of just under $379B. This is more revenue than the entire Fortune 500 companies realized in 1968 – just 40 years ago. This is not just a Wal-Mart phenomenon – the average size of organizations as measured by revenue has roughly doubled every 10 years for the past 50 years. While this isn’t exactly Moore’s law, the idea of organizations doubling in size over a consistent interval follows a similar formula. While these are fascinating statistics, what do they have to do with Integration? (more…)
IT Budgets, Clouds and Virtualisation
A blog at IT-Director.com caught my eye yesterday. Entitled “IT Budgets, Clouds and Virtualization” it included the following comment:
“For Cloud computing, chief amongst these concerns is the readiness of commercial organisations to trust significant proportions of their essential, and hence incredibly valuable, corporate information to platforms and suppliers over whom they have little control and who might hold the data wherever they wish. Such a leap of faith is today beyond consideration in many business scenarios.”
This is spot-on. There is so much talk about “cloud computing this, and cloud computing that”. When it comes to corporations there are many examples of outsourcing non-core business processes to the cloud. Here at Informatica we use over 17 different services ourselves. I’d say the most mission-critical of these is our email marketing system (can’t tell you who or I’d have to shoot you!). We’re rolling it out worldwide across our marketing team and have spent the last few months integrating it with our own on-premise CRM system, contact hub and datawarehouse. Not a trivial task but incredibly important for me (well, I’m a marketeer) but probably less mission-critical to our CFO!
At the end of the day corporations WILL move data into the clouds so whilst I agree with Tony in the above-mentioned article, I also disagree with him (OK, bit of a split personality here now). I agree that it is foolish to simply “go to the clouds”, but I disagree about the state of the industry. It is possible to keep the data secure and we, amongst others, have proven that with our on-demand integration service. It is also possible to integrate such services into core business processes. My statement would be – don’t overlook the integration. You do it at your peril. We’ve had a LOT of experience of helping companies do this effectively – after all we’re the data integration company!
Oh … and we can do it from in the clouds too 
Business Intelligence Strategies in a Down Economy
There is a lot of worry on Wall Street and Main Street these days. Are we in a mild or severe recession? Is it the next Great Depression? How long will it last? No one knows the answers to these lofty questions, but Forrester Research has been busy recalibrating on the impact the economy is having on IT spending.
First, the good news, IT spending was better in the first half of this year than expected. The bad news, IT spending is being hit adversely now and probably into 2009. According to Forrester Research:
“The economy’s affect on IT spending is evident in some specific data points contained in the report: Forty-three percent of firms have already cut their overall IT budgets in 2008 in reaction to the slow down in the global economy, while 24 percent of firms have put discretionary spending on hold. Twenty-eight percent of respondents said the economy has had no impact on their IT budgets.”
“Forrester: Impact Of Economic Downturn On Tech Spending Varies By Region And Sector”, Forrester Research, September 9, 2008
Even under the best circumstances it’s important to maximize the value from your BI/DW projects. But with these conditions it becomes even more of an imperative.
No one can afford to be sloppy or wasteful in their business intelligence and data integration strategies. Cost cutting and getting by with what you have is the norm.
But mistakes are expensive. Businesses, now more than ever, need to understand who their current and potential customers are as well as how much revenue and profit each product or service line generates. This demands current, consistent, clean and comprehensive data. (more…)
Cloud Computing – Integration Is key
The airways are a buzz with cloud computing. I was reading an article just yesterday on a blog posting about building datawarehouses in the clouds. Entitled “Google, Panorama and the BI of the Future” the article made reference to ourselves and the work we are doing in this area:
… Once your data’s in the cloud, you’re going to want to load it into a hosted data warehouse of some kind, and I don’t think that’s too much to imagine given the cloud databases already mentioned. But how to load and transform it? Not so much of an issue if you’re doing ELT, but for ETL you’d need a whole bunch of new hosted ETL services to do this. I see Informatica has one in Informatica On Demand;
I think there are great opportunities here in helping to utilize the cloud to deliver not just business applications but also areas such as CDW (or cloud-based data warehousing). This is something that we are looking at closely. It starts with being able to access data in different systems (both on-premise and on-demand). You may have already seen the work we are doing in this area: take a look at our on-demand web-site at www.informaticaondemand.com. Here you can see our cloud-computing data integration services – multi-tenanted, in the clouds with no software! Pretty cool. If you want to see a demo of what it’s all about, take a look at our cloud computing integration demo.
There’s a lot happening in the world of cloud computing and I’m delighted to see Informatica leading the charge in helping corporations to integrate and drive data quality across the clouds.
Cloud computing cannot succeed without integration. The last thing anyone wants is to simply fragment their data across the clouds. Integration prevents this and ensures you can retain control of your data assets.
Do you agree? let me know!
Master Data Management Paves the Way for SOA
What role does master data management (MDM) play in laying the ground work for service oriented architecture? Apparently, a very crucial role – SOA in and of itself holds little value to an organization unless it provides the capability to open up information to the enterprise. As is the case with SOA, successful MDM is a silo-breaker, invoking collaboration across the enterprise. This demands adroit governance that not only closely links new initiatives to what the business needs, but also assures that the information populating SOA-based services is accurate, timely, and consistent.
A panel of leading experts in a recent Webinar explored the emerging role of MDM as an enabler of SOA. (more…)
Loading ...