Informatica Perspectives

Just as they say success is 10% inspiration and 90% perspiration, it can also be said that the success of a data integration project is 10% technology and 90% chemistry. And when I say chemistry, I'm not talking about hydrocarbons and nitrates, but the chemistry of people.

The success of any complex data integration depends on how the people that make things happen - the teams of administrators, analysts, managers, end-users, and business partners - can collaborate in establishing the business case, setting requirements, selecting technology, and putting all the pieces together.

However, two of the key players in data integration - analysts and administrators - don't necessarily see eye to eye, and this is costing enterprises in terms of staff resources and quality. Wayne Eckerson, director of research at The Data Warehousing Institute, just published a report (PDF link) documenting the ongoing tension between IT administrators and business analysts - the two categories of professionals that need to collaborate to make data warehousing and business intelligence projects tick.

"Business analysts who use statistics and sophisticated machine learning techniques to coax hidden patterns and relationships out of large data sets to solve business problems are engaged in a never-ending battle with IT professionals charged with safeguarding corporate data warehouses and ensuring reliable operations of operational applications and systems," he says.

This conflict isn't just an internal office spat. The ramifications for businesses include a lack of timeliness, in which, as Eckerson points out, "analysts spend upwards of 50% of their time accessing, exploring, and manipulating data-tasks that the IT department could assist with." As a result, analytic modelers spend their time managing data instead of "performing the work they were hired to do, which is to create and analyze models. Instead, they spend half their time working as glorified IT administrators. Freeing analysts from having to prepare and manage data enables companies to radically improve the productivity of their analysts."

In addition, model quality suffers when analysts and IT aren't aligned, Eckerson adds. "When denied open and easy access to data, analysts perform fewer iterations of their model because IT has put up too many roadblocks. As a result, their models are less optimized, resulting in lost revenue for the business."

The challenge is that analysts and administrators tend to hail from different work cultures, have different agendas, and tend to view each other with suspicion.  Administrators worry about uptime, stability, service level agreements, and security. In contrast, analysts "work iteratively, rather than systematically, constantly experimenting with new ways to tune their models," Eckerson explained.

To the more free-form analyst community, "IT seems paralyzed by process and blinded by methodology. IT's systematic approach to development seems more like a smokescreen to deflect criticism, should a project fail, rather than a way to ensure a successful outcome." They see "IT as a big obstacle to be circumvented."

The IT department, on the other hand, sees analysts as "fickle pickles-people who keep changing their minds about what data they need. Modelers are the users who deliver the classic data warehousing line when asked what data they need: 'I don't know. Just give me all of it.'" They also worry about analysts' lack of security sense: "Most of them want to download large volumes of data, including customer records, to their desktops or laptops, which can easily walk out the door or be stolen."

Eckerson recommends a number of proactive steps to achieve greater communication and cooperation between professionals engaged in data integration, including identifying an executive sponsor and fostering a dialog between the groups.  He also advocates training, noting, "both sides may want to change for the benefit of the organization, but may lack the skills and knowledge to make it happen. A multi-tiered training and buddy program can help grease the wheels of change. He even says companies should put up "virtual sandboxes" that enable analysts and administrators to collaborate and try out new approaches.

The divide between business and technology professionals has been legend, and it's often said that members of the two groups speak different languages. There's a gradual convergence taking place, however, as business professionals gain a greater understanding and comfort with technology - and technologists gain greater business savvy. However, as with any situation where there are groups that need to work together but have conflicting agendas, the enterprise needs to step up and provide a proactive approach to teamwork and sense of shared mission.