Category Archives: Data Integration
Leo Eweani makes the case that the data tsunami is coming. “Businesses are scrambling to respond and spending accordingly. Demand for data analysts is up by 92%; 25% of IT budgets are spent on the data integration projects required to access the value locked up in this data “ore” – it certainly seems that enterprise is doing The Right Thing – but is it?”
Data is exploding within most enterprises. However, most enterprises have no clue how to manage this data effectively. While you would think that an investment in data integration would be an area of focus, many enterprises don’t have a great track record in making data integration work. “Scratch the surface, and it emerges that 83% of IT staff expect there to be no ROI at all on data integration projects and that they are notorious for being late, over-budget and incredibly risky.”
The core message from me is that enterprises need to ‘up their game’ when it comes to data integration. This recommendation is based upon the amount of data growth we’ve already experienced, and will experience in the near future. Indeed, a “data tsunami” is on the horizon, and most enterprises are ill prepared for it.
So, how do you get prepared? While many would say it’s all about buying anything and everything, when it comes to big data technology, the best approach is to splurge on planning. This means defining exactly what data assets are in place now, and will be in place in the future, and how they should or will be leveraged.
To face the forthcoming wave of data, certain planning aspects and questions about data integration rise to the top:
Performance, including data latency. Or, how quickly does the data need to flow from point or points A to point or points B? As the volume of data quickly rises, the data integration engines have got to keep up.
Data security and governance. Or, how will the data be protected both at-rest and in-flight, and how will the data be managed in terms of controls on use and change?
Abstraction, and removing data complexity. Or, how will the enterprise remap and re-purpose key enterprise data that may not currently exist in a well-defined and functional structure?
Integration with cloud-based data. Or, how will the enterprise link existing enterprise data assets with those that exist on remote cloud platforms?
While this may seem like a complex and risky process, think through the problems, leverage the right technology, and you can remove the risk and complexity. The enterprises that seem to fail at data integration do not follow that advice.
I suspect the explosion of data to be the biggest challenge enterprise IT will face in many years. While a few will take advantage of their data, most will struggle, at least initially. Which route will you take?
The transition to value-based care is well underway. From healthcare delivery organizations to clinicians, payers, and patients, everyone feels the impact. Each has a role to play. Moving to a value-driven model demands agility from people, processes, and technology. Organizations that succeed in this transformation will be those in which:
- Collaboration is commonplace
- Clinicians and business leaders wear new hats
- Data is recognized as an enterprise asset
The ability to leverage data will differentiate the leaders from the followers. Successful healthcare organizations will:
1) Establish analytics as a core competency
2) Rely on data to deliver best practice care
3) Engage patients and collaborate across the ecosystem to foster strong, actionable relationships
Trustworthy data is required to power the analytics that reveal the right answers, to define best practice guidelines and to identify and understand relationships across the ecosystem. In order to advance, data integration must also be agile. The right answers do not live in a single application. Instead, the right answers are revealed by integrating data from across the entire ecosystem. For example, in order to deliver personalized medicine, you must analyze an integrated view of data from numerous sources. These sources could include multiple EMRs, genomic data, data marts, reference data and billing data.
A recent PWC survey showed that 62% of executives believe data integration will become a competitive advantage. However, a July 2013 Information Week survey reported that 40% of healthcare executives gave their organization only a grade D or F on preparedness to manage the data deluge.
What grade would you give your organization?
You can improve your organization’s grade, but it will require collaboration between business and IT. If you are in IT, you’ll need to collaborate with business users who understand the data. You must empower them with self-service tools for improving data quality and connecting data. If you are a business leader, you need to understand and take an active role with the data.
To take the next step, download our new eBook, “Potential Unlocked: Transforming healthcare by putting information to work.” In it, you’ll learn:
- How to put your information to work
- New ways to govern your data
- What other healthcare organizations are doing
- How to overcome common barriers
So go ahead, download it now and let me know what you think. I look forward to hearing your questions and comments….oh, and your grade!
Over the last 40 years, data has become increasingly distributed. It used to all sit on storage connected to a mainframe. It used to be that the application of computing power to solve business problems was limited by the availability of CPU, memory, network and disk. Those limitations are no longer big inhibitors. Data fragmentation is now the new inhibitor to business agility. Data is now generated from distributed data sources not just within a corporation, but from business partners, from device sensors and from consumers Facebook-ing and tweeting away on the internet.
So to solve any interesting business problem in today’s fragmented data world, you now have to pull data together from a wide variety of data sources. That means business agility 100% depends on data integration agility. But how do you do deliver that agility in a way that is not just fast, but reliable, and delivers high quality data?
First, to achieve data integration agility, you need to move from a traditional waterfall development process to an agile development process.
Second, if you need reliability, you have to think about how you start treating your data integration process as a critical business process. That means thinking about how you will make your integration processes highly available. It also means you need to monitor and validate your operational data integration processes on an ongoing basis. The good news is that the capabilities you need for data validation as well as operational monitoring and alerting for your data integration process are now built into Informatica’s newest PowerCenter Edition, PowerCenter Premium Edition.
Lastly, the days where you can just move data from A to B without including a data quality process are over. Great data doesn’t happen by accident, it happens by design. And that means you also have to build in data quality directly into your data integration process.
Great businesses depend on great data. And great data means data that is delivered on time, with confidence and with high quality. So think about how your understanding of data integration and great data can make your career. Great businesses depend on great data and people like you who have the skills to make a difference. As a data professional, the time has never been better for you to make a contribution to the greatness of your organization. You have the opportunity to make a difference and have an impact because your skills and your understanding of data integration has never been more critical.
When I was seven years old, Danny Weiss had a birthday party where we played the telephone game. The idea is this: there are 8 people sitting around a table, the first person tells the next person a little story. They tell the next person, the story, and so on, all the way around the room. At the end of the game, you compare the original story that the first person tells and compare it to the story the 8th person tells. Of course, the stories are very different and everyone giggles hysterically… we were seven years old after all.
The reason I was thinking about this story is that data integration development is similarly inefficient as a seven year old birthday party. The typical process is that a business analyst, using the knowledge in their head about the business applications they are responsible for, creates a spreadsheet in Microsoft Excel that has a list of database tables and columns along with a set of business rules for how the data is to be transformed as it moved to a target system (a data warehouse or another application). The spreadsheet, which is never checked against real data, is then passed to a developer who then creates code in separate system in order to move the data, which is then checked by a QA person which is then checked again by the business analyst at the end of the process. This is the first time the business analyst verifies their specification against real data.
99 times out of 100, the data in the target system doesn’t match what the business analyst was expecting. Why? Either the original specification was wrong because the business analyst had a typo or the data is inaccurate. Or the data in the original system wasn’t organized the way the analyst thought it was organized. Or the developer misinterpreted the spreadsheet. Or the business analyst simply doesn’t need this data anymore – he needs some other data. The result is lots of errors, just like the telephone game. And the only way to fix it is with rework and then more rework.
But there is a better way. What if the data analyst could validate their specification against real data and self correct on the fly before passing the specification to the developer. What if the specification were not just a specification, but a prototype that could be passed directly to the developer who wouldn’t recode it, but would just modify it to add scalability and reliability? The result is much less rework and much faster time to development. In fact, up to 5 times faster.
That is what Agile Data integration is all about. Rapid prototyping and self-validation against real data up front by the business analyst. Sharing of results in a common toolset back and forth to the developer to improve the accuracy of communication.
Because we believe the agile process is so important to your success, Informatica is giving all of our PowerCenter Standard Edition (and higher editions) customers agile data integration for FREE!!! That’s right, if you are a current customer of Informatica PowerCenter, we are giving you the tools you need to go from the old fashion error-prone, waterfall, telephone game style of development to a modern 21st century Agile process.
• FREE rapid prototyping and data profiling for the data analyst.
• Go from prototype to production with no recoding.
• Better communication and better collaboration between analyst and developer
PowerCenter 9.6. Agile Data Integration built in. No more telephone game. It doesn’t get any better than that.
If you build an IT Architecture, it will be a constant up-hill battle to get business users and executives engaged and take ownership of data governance and data quality. In short you will struggle to maximize the information potential in your enterprise. But if you develop and Enterprise Architecture that starts with a business and operational view, the dynamics change dramatically. To make this point, let’s take a look at a case study from Cisco. (more…)
Maybe the word “death” is a bit strong, so let’s say “demise” instead. Recently I read an article in the Harvard Business Review around how Big Data and Data Scientists will rule the world of the 21st century corporation and how they have to operate for maximum value. The thing I found rather disturbing was that it takes a PhD – probably a few of them – in a variety of math areas to give executives the necessary insight to make better decisions ranging from what product to develop next to who to sell it to and where.
Don’t get me wrong – this is mixed news for any enterprise software firm helping businesses locate, acquire, contextually link, understand and distribute high-quality data. The existence of such a high-value role validates product development but it also limits adoption. It is also great news that data has finally gathered the attention it deserves. But I am starting to ask myself why it always takes individuals with a “one-in-a-million” skill set to add value. What happened to the democratization of software? Why is the design starting point for enterprise software not always similar to B2C applications, like an iPhone app, i.e. simpler is better? Why is it always such a gradual “Cold War” evolution instead of a near-instant French Revolution?
Why do development environments for Big Data not accommodate limited or existing skills but always accommodate the most complex scenarios? Well, the answer could be that the first customers will be very large, very complex organizations with super complex problems, which they were unable to solve so far. If analytical apps have become a self-service proposition for business users, data integration should be as well. So why does access to a lot of fast moving and diverse data require scarce PIG or Cassandra developers to get the data into an analyzable shape and a PhD to query and interpret patterns?
I realize new technologies start with a foundation and as they spread supply will attempt to catch up to create an equilibrium. However, this is about a problem, which has existed for decades in many industries, such as the oil & gas, telecommunication, public and retail sector. Whenever I talk to architects and business leaders in these industries, they chuckle at “Big Data” and tell me “yes, we got that – and by the way, we have been dealing with this reality for a long time”. By now I would have expected that the skill (cost) side of turning data into a meaningful insight would have been driven down more significantly.
Informatica has made a tremendous push in this regard with its “Map Once, Deploy Anywhere” paradigm. I cannot wait to see what’s next – and I just saw something recently that got me very excited. Why you ask? Because at some point I would like to have at least a business-super user pummel terabytes of transaction and interaction data into an environment (Hadoop cluster, in memory DB…) and massage it so that his self-created dashboard gets him/her where (s)he needs to go. This should include concepts like; “where is the data I need for this insight?’, “what is missing and how do I get to that piece in the best way?”, “how do I want it to look to share it?” All that is required should be a semi-experienced knowledge of Excel and PowerPoint to get your hands on advanced Big Data analytics. Don’t you think? Do you believe that this role will disappear as quickly as it has surfaced?
I love exploring new places. I’ve had exceptional experiences at the W in Hong Kong, El Dorado Royale in the Riviera Maya and Ventana Inn in Big Sur. I belong to almost every loyalty program under the sun, but not all hospitality companies are capitalizing on the potential of my customer information. Imagine if employees had access to it so they could personalize their interactions with me and send me marketing offers that appeal to my interests.
Do I have high expectations? Yes. But so do many travelers. This puts pressure on marketing and sales executives who want to compete to win. According to Deloitte’s report, “Hospitality 2015: Game changers or spectators?,” hospitality companies need to adapt to meet consumers’ increasing expectations to know their preferences and tastes and to customize packages that suit individual needs.
In this interview, Jeff Klagenberg, senior principal at Myers-Holum, explains how one of the largest, most customer-focused companies in the hospitality industry is investing in better customer, product, and asset information. Why? To personalize customer interactions, bundle appealing promotion packages and personalize marketing offers across channels.
Q: What are the company’s goals?
A: The executive team at one of the world’s leading providers of family travel and leisure experiences is focused on achieving excellence in quality and guest services. They generate revenues from the sales of room nights at hotels, food and beverages, merchandise, admissions and vacation club properties. The executive team believes their future success depends on stronger execution based on better measurement and a better understanding of customers.
Q: What role does customer, product and asset information play in achieving these goals?
A: Without the highest quality business-critical data, how can employees continually improve customer interactions? How can they bundle appealing promotional packages or personalize marketing offers? How can they accurately measure the impact of sales and marketing efforts? The team recognized the powerful role of high quality information in their pursuit of excellence.
Q: What are they doing to improve the quality of this business-critical information?
A: To get the most value out of their data and deliver the highest quality information to business and analytical applications, they knew they needed to invest in an integrated information management infrastructure to support their data governance process. Now they use the Informatica Total Customer Relationship Solution, which combines data integration, data quality, and master data management (MDM). It pulls together fragmented customer information, product information, and asset information scattered across hundreds of applications in their global operations into one central, trusted location where it can be managed and shared with analytical and operational applications on an ongoing basis.
Q: How will this impact marketing and sales?
A: With clean, consistent and connected customer information, product information, and asset information in the company’s applications, they are optimizing marketing, sales and customer service processes. They get limitless insights into who their customers are and their valuable relationships, including households, corporate hierarchies and influencer networks. They see which products and services customers have purchased in the past, their preferences and tastes. High quality information enables the marketing and sales team to personalize customer interactions across touch points, bundle appealing promotional packages, and personalize marketing offers across channels. They have a better understanding of which marketing, advertising and promotional programs work and which don’t.
Q: What is the role did the marketing and sales leaders play in this initiative?
A: The marketing leaders and sales leaders played a key role in getting this initiative off the ground. With an integrated information management infrastructure in place, they’ll benefit from better integration between business-critical master data about customers, products and assets and transaction data.
Q. How will this help them gain customer insights from “Big Data”?
A. We helped the business leaders understand that getting customer insights from “Big Data” such as weblogs, call logs, social and mobile data requires a strong backbone of integrated business-critical data. By investing in a data-centric approach, they future-proofed their business. They are ready to incorporate any type of data they will want to analyze, such as interaction data. A key realization was there is no such thing as “Small Data.” The future is about getting very bit of understanding out of every data source.
Q: What advice do you have for hospitality industry executives?
A: Ask yourself, “Which of our strategic initiatives can be achieved with inaccurate, inconsistent and disconnected information?” Most executives know that the business-critical data in their applications, used by employees across the globe, is not the highest quality. But they are shocked to learn how much this is costing the company. My advice is talk to IT about the current state of your customer, product and asset information. Find out if it is holding you back from achieving your strategic initiatives.
Also, many business executives are excited about the prospect of analyzing “Big Data” to gain revenue-generating insights about customers. But the business-critical data about customers, products and assets is often in terrible shape. To use an analogy: look at a wheat field and imagine the bread it will yield. But don’t forget if you don’t separate the grain from the chaff you’ll be disappointed with the outcome. If you are working on a Big Data initiative, don’t forget to invest in the integrated information management infrastructure required to give you the clean, consistent and connected information you need to achieve great things.
As covered by Loraine Lawson, “When it comes to data, the U.S. federal government is a bit of a glutton. Federal agencies manage on average 209 million records, or approximately 8.4 billion records for the entire federal government, according to Steve O’Keeffe, founder of the government IT network site, MeriTalk.”
Check out these stats, in a December 2013 MeriTalk survey of 100 federal records and information management professionals. Among the findings:
- Only 18 percent said their agency had made significant progress toward managing records and email in electronic format, and are ready to report.
- One in five federal records management professionals say they are “completely prepared” to handle the growing volume of government records.
- 92 percent say their agency “has a lot of work to do to meet the direction.”
- 46 percent say they do not believe or are unsure about whether the deadlines are realistic and obtainable.
- Three out of four say the Presidential Directive on Managing Government Records will enable “modern, high-quality records and information management.”
I’ve been working with the US government for years, and I can tell that these facts are pretty accurate. Indeed, the paper glut is killing productivity. Even the way they manage digital data needs a great deal of improvement.
The problem is that the issues are so massive that’s it’s difficult to get your arms around it. Just the DOD alone has hundreds of thousands of databases on-line, and most of them need to exchange data with other systems. Typically this is done using old fashion approaches, including “sneaker-net,” Federal Express, FTP, and creaky batching extracts and updates.
The “digital data diet,” as Loraine calls it, really needs to start with a core understanding of most of the data under management. That task alone will take years, but, at the same time, form an effective data integration strategy that considers the dozens of data integration strategies you likely formed in the past that did not work.
The path to better data management in the government is one where you have to map out a clear path from here to there. Moreover, you need to make sure you define some successes along the way. For example, the simple reduction of manual and paper processes by 5 or 10 percent would be a great start. It’s something that would save the tax payers billions in a short period of time.
Too many times the government gets too ambitious around data integration, and attempts to do too much in too short an amount of time. Repeat this pattern and you’ll find yourself running in quicksand, and really set yourself up for failure.
Data integration is game-changing technology. Indeed, the larger you are, the more game-changing it is. You can’t get much larger than the US government. Time to get to work.
Rob Karel has been doing a nice job explaining Big Data, Metadata and other topics for Mom, so now I’d like to tackle another key group of stakeholders – your children. My kids have been asking me for years what I do at work. It hasn’t been easy to come up with an explanation that they can understand, so I usually just end up with something like “I go to meetings and stuff.” That works for a while, but it’s not very informative or inspiring. So if their friends ask “what does your dad do for work”, I can’t imagine what stories they make up. So here goes my attempt to explain to a sixth-grader what the job of a systems integration professional is. (more…)