With the ready availability of data integration technology, it’s amazing to me that the use of manual coding for data integration flows is even a consideration. However, based upon this article in SearchDataManagement, the concept is still out there.
Of course the gist of the article is that hand coding is no longer considered the most productive way to go, which is correct. However, just the fact that this is still an issue and a consideration for anyone moving to data integration solutions perplexes me. Perhaps it’s the new generation of architects and data management professionals who need a quick lesson on the pitfalls of doing data integration by hand.
When I wrote the EAI book back in the day, the motivation came from seeing any number of projects where the interfaces to the source and target database where hand coded. After several years of this hand coded practice, the links between data sources look something like a very dysfunctional highway system. It did not scale. It was too complex. Worst of all, it limited the ability to change integration flows, thus limiting agility, thus limiting the business.
Hand coding was never a good architectural option, in my opinion. Indeed, it was actually a step backwards. Hand coding simply digs you into a deeper and deeper hole. Indeed, today I still work with companies that try to fix the limitations with data integration approaches in this manner. It’s a bell that’s difficult to un-ring.
So, let’s start with the basics. Data integration tools are required to provide a few core capabilities:
- Connect into the source and target data sources without disturbing operations, or forcing changes to those data sources or targets.
- Extract the right data and the right time, typically on an event.
- Transform the structure and content of the source data as it flows from place to place so it appears native to the target data source.
- Split and combine flows as required.
- Provide management of the data integration system, as well as data governance and security services.
The most valuable and perhaps most misunderstood capability has to do with the data integration technologies’ ability to add, update, or remove data integration flows as required. Thus, the more your organization changes, the more value that data integration technology brings to the table.
The ability to change, or place volatility into a single domain that allows these changes to occur with minimal disruption, is the deeper reason that you never want to hand code data integration flows. While it may seem like a good idea, considering that you may be attempting to minimize cost, and the initial requirements seem simple enough, you’ll pay dearly down the road. Trust me on that one.