Integration At The Speed Of Data

Computer systems and their component parts change at different rates. Things that change quickly tend to become grouped together and managed separately from things that change more slowly. The different rates of change are like fault lines that help foster the emergence of enduring abstractions and are described as Shearing Layers in the seminal paper Big Ball of Mud[1] by Brian Foote and Joseph Yoder.

The Ball of Mud is similar to the Integration Hairball.  These are architectural styles (if you want to give them that much credit) where systems evolve in piecemeal fashion from a haphazard process that is dictated more by expediency than design. The Big Ball of Mud was written about 10 years ago and continues to serve as an insightful perspective on how enduring architectures emerge from real-world dynamics. One of the more astute observations is that systems are not monolithic static objects, but rather that constituent elements change at different rates which creates Shearing Layers.

The paper makes the comparison to buildings which also are not static and that some aspects of the building change quickly while others change slowly. In fact six layers are defined as “Site, Structure, Skin, Services, Space Plan, and Stuff. Site is geographical setting. Structure is the load bearing elements, such as the foundation and skeleton. Skin is the exterior surface, such as siding and windows. Services are the circulatory and nervous systems of a building, such as its heating plant, wiring, and plumbing. The Space Plan includes walls, flooring, and ceilings. Stuff includes lamps, chairs, appliances, bulletin boards, and paintings.” These layers change at different rates which in turn sets the stage for different roles (professions).

Foote and Yoder provide some thoughts about layers in software systems, but stop short of developing a specific list similar to the shearing layers of a building – so this is where I pick up the ball (no pun intended). My classification of software system layers is Standards, Infrastructure, Database Schemas, Frameworks, Code, and Data. 

  1. Standards are the most enduring and may last for 50 years or more; examples include Cobol (50 years), SWIFT (35 years) and Ethernet (30 years).  These standards have of course evolved as well, but standards organizations, by design, ensure that they don’t change too rapidly. The computer industry is too young to have a track record for how long standards should last. Cobol and SWIFT may well still be in use 100 years after their inception.
  2. The Infrastructure layer includes operating systems (MVS, UNIX, Windows) and computer communication networks. While vendors may release new versions every few years, once implemented in an organization, they usually remain fixed for 10 years or more.  I would also put large-scale ERP systems in this category since changes to systems that affect many organizations change rather slowly.
  3. Database Schemas also change rather slowly – especially enterprise data models. While schema extensions may happen quite frequently, major restructuring or refactoring of fundamental data structures are in the seven-plus year time frame.
  4. Software Frameworks, or libraries of reusable components, turnover about every three-five years.  This category also includes enterprise-class SOA services.  While specific objects or services, once established, may remain fixed for much longer, on average these elements are more dynamic and tend to be refactored on a periodic basis.
  5. Application code, internally developed that is, tends to change about once every year or two.  Some organizations that have adopted agile methods and continuous integration make changes even more rapidly while large internally developed systems may take two or more years between significant releases.
  6. Data changes the most rapidly – in fact every day. Sales transactions are recorded, new accounts established, customer information updated, and service calls logged to name just a few.

The business operates at the speed of data – changing every day.  IT, including integration activities, operates at the speed of code and infrastructure – changing every three-10 years. This is the first clue as to the persistent source of tension between business users and IT staff.  So how can we close this gap?  The answer is to turn as much of the code, frameworks, database schemas and infrastructure into data – Metadata to be more accurate. To quote the Ball of Mud paper, “Metadata allow systems to adapt more quickly to changing requirements by pushing power into the data, and out onto users.”

Of course you need more than just metadata, you must also provide access to it through self-service portals.  When you do this in a sustained and systematic manner, then IT begins to move at the speed of data – at least to a large degree from the business users perspective. Under the covers the infrastructure and frameworks still move slowly, but the users can serve themselves for much of their information and processing needs.

David Lyle and I devoted several chapters in our recent book, Lean Integration[2], to this idea. Chapter three describes the Integration Factory and its goal of being a self-service capability. Chapter 9 discusses one of the seven Lean principles, Automation, and how IT should use the computer, not just as a business service, but as a tool to more efficiently and rapidly do the work that IT does. In short, everyone is happier.

[1] Brian Foote and Joseph Yoder, Big Ball of Mud, Addison-Wesley Software Patterns Series, 2000

[2] John G. Schmidt and David Lyle, Lean Integration: An Integration Factory Approach to Business Agility, Addison-Wesley, May 2010

This entry was posted in Business/IT Collaboration, Data Integration Platform, Integration Competency Centers. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>