Informatica Perspectives

Continuing the Lean versus Agile discussion, these methods work well for small-scale projects, but are they effective for large programs? This post addresses the question “Should an Integration Factory do anything different when supporting a project that is using a waterfall methodology versus one that is using an agile methodology?” The short answer is no, you shouldn’t do anything different for one simple reason; focus on the customer and deliver what valuable from his/her perspective.

If the customer (the project being supported) desires rapid delivery of incremental integrations, then the Integration Factory should deliver that. If the customer desires formal delivery and signoffs of stage-gate deliverables for a waterfall methodology, then the Integration Factory should deliver that. The beauty of the metadata-driven Integration Factory is that it can just as easily respond to either approach since it views the factory information flows to be just as important as the material flows. The Lean disciplines involved in defining, optimizing and automating both of these flows makes is easy to tap into the metadata repository and extract formal reports and deliverables that the waterfall project requires or to focus on rapid delivery of small batches of functionality that the Agile project demands.

An interesting observation is that large complex IT projects which use a waterfall methodology still require an agile integration approach. The reality is that the requirements document that is produced in the early stages of the project never fully, or accurately, specifies the interfaces, data quality profile, or integration requirements. As a result the integration phase always takes longer than anticipated and involves multiple iterations as the “real” integration needs are uncovered. Once again the factory approach handles this situation perfectly by using a metadata-driven mass-customization approach to rapidly deliver and re-deliver integration components.

Another question is “Which approach is better for a large-scale enterprise systems integration project; waterfall or agile?” I generally recommend agile methods, but there are times when it is challenging to do so. For example, there is often more than one customer or organization involved in a large-scale program and everyone needs to buy-in to the agile approach for it to work effectively. Some customers (business sponsors) may demand certainty around project scope and cost which often steers teams down a waterfall methodology path. It also may be difficult to replace a collection of tightly integrated legacy systems in an incremental fashion while also maintaining existing service levels and hence a big-bang conversion may appear more feasible which also steers teams down the waterfall path. Finally, integration teams may be separated geographically in large multi-national firms, or with vendor teams operating under different contractual terms, which makes it difficult to apply Agile principles of face-to-face communications. These and other real-world issues make it difficult for many organizations to adopt agile methods for large-scale projects.

The reality is that Agile methods work best for smaller scale software development projects. As projects or programs get larger, there is less empirical evidence that Agile is better – partly because it simply isn’t used as often for large programs. Our instinct tells us that Agile principles will improve the probability of success for large projects, but the challenges mentioned above need to be addressed early in the project life-cycle.

For additional related material, please check out www.integrationfactory.com which includes an overview of my (and David Lyle’s) upcoming book Lean Integration and a number of useful reference links.