Manufacturing Best Practices Have Changed Next-Generation Data Integration

Over the course of the last century, manufacturing has improved from individual craftsman to Ford’s assembly lines to Toyota’s Production System.  In particular, Toyota’s Production System has been generalized to be used in all other business areas, from Health Care to Retail to Financial Services to Public Sector organizations.  By incorporating the thought processes that guide the continuous improvement programs and repeatable best practices found in optimized manufacturing organizations, all kinds of business processes can eliminate the waste and deliver higher customer value more quickly, more cheaply, and with lower risk and higher quality.

This approach is also being adopted with organizations adopting Next-Generation Data Integration software, processes, and skills.  I have personally found an order-of-magnitude difference between the most effective organizations and delivering data integration and management projects compared to organizations that are least effective.  And I haven’t found that success is correlated to size of the organization.  Success is more correlated with leadership (having an effective change agent) and marketing the successes of the organization broadly.

To continue with the manufacturing metaphor, Next-Generation Data Integration realizes that accessing, discovering data problems, cleansing, conforming and transforming the data, and delivering the data in either batches or streaming that data in real-time, falls into a relatively small number of repeating patterns.  Think of these like the different assembly lines required to manufacture different classes of cars: pick-up trucks, sedans, sports cars, 18-wheelers, vans, etc.  Data Warehousing has its share of patterns.  Data Migration has its share; Data Replication has a small number, etc.  If we focus architects and developers on the building the assembly lines rather than coding individual integration logic as if they are craftspeople building a work of art, we gain many advantages:

  1. We build quality in because the assembly line guides the developer down the line of “best practices”.
  2. Where appropriate, the assembly lines can be simple enough for IT to provide a Self-Service model to the business so that they can get data for themselves, for analysis.  IT maintains oversight of what’s going on in this scenario, preventing chaos but allowing IT to shine a light on the inevitable “shadow IT” that results when IT isn’t agile or cost-effective enough.
  3. We enable greater re-use and agility because we’re defining patterns at a higher level of abstraction.  This allows us to define the business definition of “What” we want to happen, allowing the data integration platform the ability to decide at run-time “How” it gets executed (Informatica platform, Hadoop cluster, Teradata or Oracle database, etc).

We could probably list more advantages or phrase these in different ways.  The point is that organizations learning from manufacturing best practices can actually achieve a “Better, Cheaper AND Faster” result, where we have historically had to pick two of those choices.  Adopting a manufacturing mentality to our Data Integration world allows us to not have to compromise and achieve all three of those benefits.



You can hear me discuss this topic more with HealthNow New York and Corporate Technologies in the webinar, “Next Generation Data Integration. For the New Data Warehouse.” to hear how world-class organizations are applying lean principles and new technologies to re-engineer their processes and architecture.

This entry was posted in Data Integration. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>