This article is the next installment of the 10 weeks to Lean Integration series.  If you are joining the discussion now, you may want to start by reading the first posting.

The original seven wastes (muda in Japanese) that were targeted in manufacturing and production lines are:

1. Transportation (unnecessary movement of materials)
2. Inventory (excess inventory including work-in-progress)
3. Motion (extra steps by people to perform the work)
4. Waiting (periods of inactivity)
5. Overproduction (production ahead of demand)
6. Over Processing (rework and reprocessing)
7. Defects (effort involved in inspecting for and fixing defects)

Developing software has many parallels with the production of physical goods and Mary and Tom Poppendieck have a done a terrific job in their book Implementing Lean Software Development to translate these wastes into the world of software development. Many of these are fairly obvious of course such as wasted effort fixing defects rather than developing defect-free code in the first place. Overproduction in the software world is the equivalent of adding features to a software component that were not requested or are not immediately required (sometimes referred to as “gold plating”). As per Poppendieck, “Wise software development organizations place top priority on keeping the code base simple, clean, and small.”

Other parallels between the physical product world and the virtual software world are less obvious such as transportation waste being the equivalent to loss of information in hand-offs between designers, coders, testers and implementers or motion waste being the equivalent of task-switching in an interrupt-driven work environment. In any event, Poppenddieck does an excellent job describing the wastes for software development. These details are best discovered by reading the book.

That said, Implementing Lean Software Development does not address all the integration disciplines and there are several areas where we often see a tremendous amount of wasted time, effort and money.

First, building functional integration capabilities or extra features before they are needed is a waste. There is a strong desire among integration teams to anticipate organizational needs and build interfaces, data marts, canonical messages, or SOA services with the needs of the entire enterprise in mind. This is an admirable objective indeed, but the integration teams typically don’t have the resources and funding to build the common capability and so they run into trouble when the first project that could use the capability is saddled with the full cost to build the enterprise-wide solution. This practice is referred to as “the first person on the bus pays for the bus” or similar metaphors. Business leaders whose project budgets are impacted by this practice hate it. Let me repeat – they hate it. The business units are given a budget to optimize their function and they become frustrated, and even angry, when told that the implementation will take longer and cost more money because it must be built as a generic re-usable capability for other future users. This may indeed be the policy of the organization, but in the end it pits the integration team against the project sponsor rather than fostering alignment.

There is however another approach – build only the features/functions that the first project requires, and do it in such a way that it can be extended in the future when the second project comes along. And if the nature of change is such that the second project requires refactoring of the solution that was developed for the first project – then so be it. This approach is still by far more desirable because of the downsides of the “build it and they will come” approach. If we build integration capabilities before they are needed:

• There is an imperfect understanding of requirements for future projects if it is built in advance so it is possible that the wrong thing will be developed
• The additional code or data will cost money to develop and maintain without any benefits until the next project comes along to use it
• The business benefits from the first project are delayed while building the supposedly “ideal” solution that will meet future needs
• The business sponsor for the first project will be dissatisfied with the implementation team (which is a good way to chase away your customer)

Organizations would be much better off adding the cost of refactoring the first implementation to the second project when (and if) it comes along. Under this scenario, the needs of the second project are clear as are the needs of the first project (since of course it is already in operation) so there is no ambiguity about what to build. Furthermore, the benefits of the first project will be realized sooner which can in essence help to fund the cost of the 2nd project (siloed accounting practices may still make this difficult in many organizations – but nonetheless the advantage is clear). The bottom line advice is to “refactor integrations constantly so that they don’t become legacy”.

Second, applying middleware technology everywhere is a waste. This may sound like heresy to some integration specialists since it is well known that the way to facilitate loose coupling between components is to add an abstraction layer. No debate on that front. But the reality is that while each layer adds a potential decoupling benefit, it also adds a cost. So this is really a cost/benefit tradeoff decision.

As integration professionals, we often deride point-to-point integrations as “evil” since they tightly couple components and, if applied without standards, over time will result in an integration hairball. True enough. But that doesn’t mean that point-to-point interfaces for specific high-volume data exchanges with stringent performance requirements isn’t the best solution. Each middleware layer bears a cost in terms of technology, people development, organizational change, and complexity on an ongoing basis, so they should only be added when the benefits of the abstraction layer outweigh the cost of sustaining it.

Another example of a middleware abstraction layer is canonical models or common data definitions. It is hard to argue against the principle of having a common definition of data across systems with incompatible data models, but nonetheless common data models are not static objects that are created at a point in time – they evolve constantly and need to be maintained. Unless you can justify the incremental staff to maintain the canonical model, don’t add this layer since it will surely become stale and irrelevant within just a few short years which is yet another example of waste.

Third, not taking advantage of economies of scale and instead re-inventing the wheel. The reality is that there are a relatively small number of integration patterns that can satisfy the integration needs of even the largest corporations. While the total number of integrations may be large (thousands or tens of thousands) the number of patterns is quite small – probably no more than a handful for 90% or more of the data exchanges. But if the integration development work is treated as a unique effort by each project team, then the result over time will be thousands of “works of art”. We know from years of experience in manufacturing lines, that cost savings of 15-25% accrue every time volume doubles (see George Stalk reference). Our experience in software development aligns with these savings due to two factors – benefits of the learning curve (the more times you do something the better you get at it) and visibility into re-use opportunities (the more integrations one does, the more obvious the patterns become).

Fourth, unnecessary variation in tools and standards is a waste. We have also learned from the world of manufacturing that costs increase 20-35% every time variation doubles (Stalk). Variation in the integration arena arises from different middleware platforms, development tools, interchange protocols, data formats, interface specifications and metadata standards to name a few. In the absence of governance around these and other sources of variation, the variety of integrations is huge.

If we combine the lack of scale and unnecessary variation, the cost and quality implications are an order of magnitude difference. To take a simple example, consider an organization with eight groups where each develops one integration per year with their own preferred set of tools, techniques and standards. It is fairly intuitive to see that if they were to combine efforts and have one team build multiple integrations, there would be cost savings from a reduced learning curve, re-use of tools and common components. The basic idea is that a team that builds eight integrations per year rather than one will see more opportunities for re-use and will get much faster simply by doing similar work with similar tools on a repeated basis. A centralized team that is producing higher volumes sees the patterns and can take advantage of them.

Using this simple example, if the cost of building an integration for each of the eight groups was $10,000, the cost for a central team would be $1,159-$3,144 – a 70-90% reduction! This result comes from doubling volumes three times with savings of 15-25% each time and cutting variation in half three times with a 20-35% saving each time. Real world case studies have shown that this kind of dramatic improvement in integration development savings is in fact achievable.

In summary, four integration areas of waste are:

1. Building integration capabilities before they are needed
2. Applying middleware technology everywhere
3. Not leveraging economies of scale (re-inventing the wheel)
4. Unnecessary variation in tools and standards

Tune in next week for Part 3 in the series where I address the topic of building and sustaining knowledge. In other words, how to create a “learning organization” from an integration perspective.

Further Reading References:

1. Mary and Tom Poppendieck, Implementing Lean Software Development: From Concept to Cash, Addison Wesley, 2007.
2. George Stalk, Time – The Next Source of Competitive Advantage, Harvard Business Review, 1988

If you’d like to get Informatica blogs on the fly, just click the ‘Subscribe’ button on the right side of the screen.