Tag Archives: data warehouse
New Methods to Optimize Data Warehouse Performance and Lower Costs
Data warehouses tend to grow very quickly because they integrate data from multiple sources and maintain years of historical data for analytics. A number of our customers have data warehouses in the hundreds of terabytes to petabytes range. Managing such a large amount of data becomes a challenge. How do you curb runaway costs in such an environment? Completing maintenance tasks within the prescribed window and ensuring acceptable performance are also big challenges.
We have provided best practices to archive aged data from data warehouses. Archiving data will keep the production data size at almost a constant level, reducing infrastructure and maintenance costs, while keeping performance up. At the same time, you can still access the archived data directly if you really need to from any reporting tool. Yet many are loath to move data out of their production system. This year, at Informatica World, we’re going to discuss another method of managing data growth without moving data out of the production data warehouse. I’m not going to tell you what this new method is, yet. You’ll have to come and learn more about it at my breakout session at Informatica World: What’s New from Informatica to Improve Data Warehouse Performance and Lower Costs.
I look forward to seeing all of you at Aria, Las Vegas next month. Also, I am especially excited to see our ILM customers at our second Product Advisory Council again this year.
Column-oriented Part 2: Flexible Data Modeling for a Simplified End User Experience
In my previous blog, I explained how Column-oriented Database Management Systems (CDBMS), also known as columnar databases or CBAT, offer a distinct advantage over the traditional row-oriented RDBMS in terms of I/O workload, deriving primarily from basing the granularity of I/O operations on the column rather than the entire row. This technological advantage has a direct impact on the complexity of data modeling tasks and on the end-user’s experience of the data warehouse, and this is what I will discuss in today’s post. (more…)
What the Physical Infrastructure Can Teach Us About Testing IT Applications
I was at an IT conference a few years ago. The speaker was talking about application testing. At the beginning of his talk, he asked the audience:
“Please raise your hand if you flew here from out of town.”
Most of the audience raised their hands. The speaker then said:
“OK, now if you knew that the airplane you flew on had been tested the same way your company tests its applications, would you have still flown on that plane?
After some uneasy chuckling, every hand went down. Not a great affirmation of the state of application testing in most IT shops. (more…)
The Broken Telephone of the Data Warehouse
Jim Harris, OCDQ
In many ways, a data warehouse resembles the children’s game broken telephone, where a message is distributed across a group by being whispered into the ear of one player after another, until the last player announces the message to the entire group. Since errors typically accumulate in the retellings, the final version of the message often differs significantly from its source. Some players appear to deliberately alter what’s being said, guaranteeing a garbled message by the end of the game.
As data journeys from operational sources through the staging area, the data warehouse, the data marts, and finally into dashboards and reports, a lot could be lost in translation. As it is processed, data is often deliberately altered to make it accommodate the structure of its next target. Every time data moves from point to point, there is a possibility for semantic inconsistencies to be introduced. (more…)
When Delivery is Job One, Quality is Job None
Jim Harris, OCDQ
The reality in data warehousing is that the primary focus is on delivery. The data warehouse team is tasked with extracting, transforming, integrating, and loading data into the warehouse within increasingly tight timeframes. Twenty years ago, monthly data warehouse loads were common. Ten years ago, weekly loads became the norm. Five years ago, daily loads were called for. Nowadays, near-real-time analytics demands the data warehouse be loaded more frequently than once a day. (more…)
Big Data in India
Last week I was in both Bangalore and Mumbai providing the keynote address at the Informatica World Tour events. I also had the opportunity to visit with customers and speak with media. Much like the rest of the world, big data is all the buzz. I was particularly taken by a set of questions asked by local media. I thought I’d share them with you along with some of my perspectives.
Who needs big data? What kind of business problems should an enterprise be facing to consider big data? (more…)
Lean Manufacturing in the Data Warehouse Factory – Building Trusted Data
Lean manufacturing, as defined by Wikipedia, is “a production practice that considers the expenditure of resources for any goal other than the creation of value for the end customer to be wasteful…Essentially, lean is centered on preserving value with less work” and is a management philosophy derived mostly from the Toyota Production System (TPS). I’ve been having discussions with Jim Harris from OCDQ Blog and Reuben Vandeventer, Director of Data Governance for CNO Financial and others about best practices for data quality management and the applicability of lean management practices as it relates to data warehousing. Click here to hear Jim, Reuben and I discuss three critical areas of data quality to focus on for building data warehouses that people actually use and trust.
How to Avoid the Big Data Trough of Disillusionment
Has big data entered the “trough of disillusionment?” That’s what I’ve heard recently. Like many hyped up technology trends the trough can be deep and long as project failures accumulate, or for ‘hot’ trends that evolve and mature quickly the trough can be shallow and short, leading to broader and rapid adoption. Is the big data hype failing to deliver on its promise of increased revenue and competitive advantage for companies that leverage big data to introduce new products and services and improve business operations? Why is it that some big data projects fail to deliver on their promise? Svetlana Sicular, Research Director, Gartner points out in her blog Big Data is Falling into the Trough of Disillusionment that, “These [advanced client] organizations have fascinating ideas, but they are disappointed with a difficulty of figuring out reliable solutions.” There are several reasons why big data projects may fail to deliver on their promise: (more…)
Building the Business Case for Big Data: Learn to Walk Before You Run
In a recent webinar, Mark Smith, CEO at Ventana Research and David Lyle, vice president, Product Strategy at Informatica discussed: “Building the Business Case and Establishing the Fundamentals for Big Data Projects.” Mark pointed out that the second biggest barrier that impedes improving big data initiatives is that the “business case is not strong enough.” The first and third barriers respectively, were “lack of resources” and “no budget” which are also related to having a strong business case. In this context, Dave provided a simple formula from which to build the business case:
Return on Big Data = Value of Big Data / Cost of Big Data (more…)
Big Data: Changing How Business and IT Interact
Most of the big data discussions have been on the technology or the numerously re-played business discoveries used as examples of big data’s power. Many companies are still in the experimental stages of big data, asking for guidance regarding what their benefits would be, how they can re-align themselves to take advantage, and what new processes might be helpful to make them successful with these powerful new capabilities. (more…)