In December, 2005 Sun Microsystems conducted an interview with Bill Inmon, the father of the data warehouse concept. He said, “ILM keeps a data warehouse from costing huge amounts of money and maintains good performance consistently throughout the data warehouse environment.” Four years later, the average size of a data warehouse has increased by 200%, surpassing the multi-terabyte size benchmark.
With these mammoth databases comes an increase in cost to manage them and a potential deterioration in performance. It is common practice to leverage techniques like indexing and database partitioning to address query performance issues with very large databases but those techniques do not address challenges associated with the raw volumes of data.
More and more IT shops are looking at archiving solutions to help reduce the size of the data warehouse database. An archiving solution can help in the following ways:
- An archiving solution, such as Informatica’s Data Archive allows for simple to complex archiving policies. ILM policies may involve more dimensions than just time – such as business unit, geography, and business transaction status. Partitions are typically aligned with just one of these dimensions making it difficult to use existing partition schemes for selecting data eligible for archiving.
- Keeping archived data online and available in a reduced compressed footprint means that you get the benefit of an archive while keeping business users happy. They will be able to access the data as if it were still in the original database. Also, if the data model changes, an online archive can be updated simplifying maintenance. If an offline archive has been used, it can be difficult to restore aged data when the data models are different.
- Archiving solutions can be leveraged across multiple applications. Most companies have several OLTP databases as well as a data warehouse. Informatica’s Data Archive solution can be installed once, leveraged across the different applications, and now with the new partnership with CommVault, can also be integrated with a single archive platform such as Simpana to address archiving and data retention policies across all data in the data center.
When assessing an ILM solution for your Data Warehouse, consider these points when evaluating technologies.
Julie Lockner, President & Founder, CentricInfoTM