Stan Dorcey
Hitting the Batch Wall, Part 2: Hardware Scaling
This is the second installment of my multi-part blog series on “hitting the batch wall.” Well, it’s not so much about hitting the batch wall, but what you can do to avoid hitting the wall. Today’s topic is “throwing hardware” at the problem (a.k.a. hardware scaling). I’ll discuss the common approaches and the tradeoffs of hardware scaling with Informatica software.
Before I can begin to discuss hardware scaling, I start with this warning: faster hardware only improves the load window situation when it resolves a bottleneck. Data integration jobs are a lot like rush hour traffic, they can only run as fast as the slowest component. It doesn’t make any sense to buy a Ferrari if you will always be driving behind a garbage truck. In other words, if your ETL jobs are constrained by the source/target systems or I/O or even just memory, then faster/more CPUs will rarely improve the situation. Understand your bottlenecks before you start throwing hardware at them! (more…)
Hitting the Batch Wall
Is the constant drum beat to process more data in less time keeping you up at night? Are your business users beating you up to deliver their data faster? Monthly loads turn into daily which turn into hourly which turn into… I don’t know… and you don’t know where to turn? Cheer up, you are not alone, and you’ve come to the right place. (more…)
Loading ...