It’s official, “big data” is here to stay and the solutions, concepts, hardware and services to support these massive implementations are going to continue to grow at a rapid pace. However, every organization has their own definition of big data and how it plays in their organization. One area that we are seeing a lot of activity in is “big transaction data” for OLTP and relational databases because relational databases often lack the true management capabilities to scale transactional applications into the higher TBs and PBs. In this post we will explore some ways that your existing OLTP system can scale without crushing IT and your budget in the process.
OLTP and Big Data
OLTP systems by their very definition are transactional in nature and are often expected to provide real-time insights on recent activity and trends. However, smart organizations have realized for years that the more data they keep accessible within these systems the better insights they can gain from trending and spotting opportunities/risks within a larger set of data. However, the problems begin to build because as the ability to store and process this data becomes easier the business demands for this data rise even faster. Organizations are forced to satisfy the business needs while balancing the massive data growth that is accumulating in multiple OLTP systems. We are seeing OLTP applications that are 30-100TBs in production and growing at two-threeTBs a month. Because of today’s storage and processing capability these volumes are not only possible but will begin to the norm over the next three-five years.
Balancing the Business/IT needs
Understanding the balance between OLTP transaction systems and their use as a historical trending and reporting environment is important. I highlight balance because IT organizations must implement a system that will support the business while still maintaining an effective cost structure. If the balance of performance/cost is not maintained all of the value will be lost. Four areas of balance:
- Managing the everyday performance of the system for users that are working on the most recent data (i.e. order entry, customer service, billing, etc)
- Managing the performance and accessibility for analyst and business users for historical trending and reporting (i.e. financial analysts, end of period closing, trend reporting)
- Enabling IT to support the operational aspects of “big data” within the OLTP environment (shrinking the storage footprint, managing backups, creating business copies for testing)
- Securing sensitive data dynamically in production and non-production boundary systems
These critical areas of balancing business needs with IT capability can be a daunting task for many IT organizations that lack the solutions to manage big transaction data proactively. Informatica’s ILM portfolio has the most unique capability in the market to manage big transaction data from day one through the end of its lifecycle easing the burden on IT and delivering real value to the business.