“We do nightly updates to our data warehouse, but we have no way to validate that the data was moved and transformed correctly in the time available. We only have time to test a subset and hope that it is the right subset.”
“We have tools to tell us about performance after an issue occurs, but nothing that helps us prevent the issue in the first place. So, we find out about failures from the end-users looking at a bad report. This causes delayed or poor business decision making, and also impacts our departments’ reputation.”
These are just a couple of quotes I have heard from data integration end users. We have been collecting similar information from thousands of our data integration customers around their challenges across the data integration lifecycle. What we are hearing is that the growth of data within organizations and ever increasing demand for more timely data has introduced a number of threats. In turn, these new demands have created massive variability in the way customers approach their projects, which introduces a host of data integration challenges, especially in production.
So naturally, organizations have taken a variety of approaches to combat these threats – ranging from adding full-time employee/contractor teams to test and monitor workflows to developing customized scripts or a combination of both. Ok – so there are monitoring tools out there. But, the fact remains that generic monitoring tools don’t uncover deep data integration issues. This was discussed at length here. Additionally, typical testing efforts such as “stare and compare” and hand-coding are manual, un-repeatable, and un-auditable as discussed here.
Now recall the point about added pressure of explosive data growth and increasing demands for timely delivery, and add to it the wide variability in how organizations approach projects – it’s scary. But what’s more concerning is that this variability riddles production environments with errors, inefficiencies, and security threats. Manual and reactive approaches to remedy the problem only exacerbate issues by increasing complexity. Hence the delays in identifying, monitoring and fixing issues, typically requiring fire drills, manual solutions and reactive measures, but not addressing problems.
If any of this sounds familiar to you, you may want to take advantage of what we are doing at Informatica World 2013 to help you address these challenges. To make things convenient for you, I have prepared a customized guide on relevant sessions, hands-on-labs and booths to help you understand how automated, repeatable and auditable testing, along-with a pre-emptive approach to diffusing threats before they erupt into full-blown issues, can help you. Please feel free to sign-up here for some of the breakout sessions we are hosting on this topic, or swing by one of the labs or booths:
Tuesday, June 4
- 9:00AM – 10:00AM – Platform & Products Track – Best Practices for Doing Data Replication the Right Way
- 2:00OM – 3:00PM – Architecture Track – PowerCenter Architecture
Wednesday, June 5
- 2:30PM – 3:30PM – Platform & Products Track – New Approaches to Reducing PowerCenter Testing and Monitoring Time
Thursday, June 6
- 9:00AM – 10:00AM – Tech Talk Track – Proactive Monitoring: Greater IT Productivity with Streamlined Data Integration
- 10:15AM – 11:15AM – Platform & Products Track – What’s New from Informatica to Improve Data Warehouse Performance and Lowering Costs
BIRDS OF A FEATHER ROUNDTABLES (Check Agenda for Daily Timings):
- Testing Strategies and Tools
- Automating Administrative Maintenance Tasks
HANDS-ON LABS (Check Agenda for Daily Timings):
- Table 42 – Informatica Data Validation
- Table 43 – Informatica Proactive Monitoring
BOOTHS (Check Agenda for Daily Timings):
- PowerCenter Developer Productivity and Production Manageability
I look forward to seeing you at Informatica World 2013.