Tag Archives: enterprise
Building an Enterprise Data Hub with proper Data IntegrationData flows into the enterprise from many sources, in many formats, sizes, and levels of complexity. And as enterprise architectures have evolved over the years, traditional data warehouses have become less of a final staging center for data, but rather, one component of the enterprise that interfaces with significant data flows. But since data warehouses should focus on being powerful engines for high value analytics, they should not be the central hub for data movement and data preparation (e.g. ETL/ELT), especially for the newer data types–such as social media, clickstream data, sensor data, internet-of-things-data, etc.–that are in use today.
When you start seeing data warehouse capacity consumed too quickly and performance degradation where end users are complaining about slower response times, and you risk not meeting your service-level agreements, then it might be time to consider an enterprise data hub (EDH). With an EDH, especially one built on Apache™ Hadoop®, you can plan a strategy around data warehouse optimization to get better use out of your entire enterprise architecture.
Of course, whenever you add another new technology to your data center, you care about interoperability. And since many systems in today’s architectures interoperate via data flows, it’s clear that sophisticated data integration technologies will be an important part of your EDH strategy. Today’s big data presents new challenges as relates to a wide variety of data types and formats, and the right technologies are needed to glue all the pieces together, whether those pieces are data warehouses, relational databases, Hadoop, or NoSQL databases.
Choosing a Data Integration Solution
Data integration software, at a high level, has one broad responsibility: to help you process and prepare your data with the right technology. This means it has to get your data to the right place in the right format in a timely manner. So it actually includes many tasks, but the end result is that timely, trusted data can be used for decision-making and risk management throughout the enterprise. You end up with a complete, ready-for-analysis picture of your business, as opposed to segmented snapshots based on a limited data set.
When evaluating a data integration solution for the enterprise, look for:
- Ease of use to boost developer productivity
- A proven track record in the industry
- Widely available technology expertise
- Experience with production deployments with newer technologies like Hadoop
- Ability to reuse data pipelines across different technologies (e.g. data warehouse, RDBMS, Hadoop, and other NoSQL databases)
Data integration is only part of the story. When you’re depending on data to drive business decisions and risk management, you clearly want to ensure the data is reliable. Data governance, data lineage, data quality, and data auditing remain as important topics in an EDH. Oftentimes, data privacy regulatory demands must be met, and the enterprise’s own intellectual property must be protected from accidental exposure.
To help ensure that data is sound and secure, look for a solution that provides:
- Centralized management and control
- Data certification prior to publication, transparent data and integration processes, and the ability to track data lineage
- Granular security, access controls, and data masking to protect data both in transit and at the source to prevent unauthorized access to specific data sets
Informatica is the data integration solution selected by many enterprises. Informatica’s family of enterprise data integration, data quality, and other data management products can manage data — of any format, complexity level, or size –from any business system, and then deliver that data across the enterprise at the desired speed.
Watch the latest Gartner video to see Todd Goldman, Vice President and General Manager for Enterprise Data Integration at Informatica, as well as executives from Cisco and MapR, give their perspective on how businesses today can gain even more value from big data.
Eighteen months ago, I was sitting in a conference room, nothing remarkable except for the great view down 6th Avenue toward the Empire State Building. The pre-sales consultant sitting across from me had just given a visually appealing demonstration to the CIO of a multinational insurance corporation. There were fancy graphics and colorful charts sharply displayed on an iPad and refreshing every few seconds. The CIO asked how long it had taken to put the presentation together. The consultant excitedly shared that it only took him four to five hours, to which the CIO responded, “Well, if that took you less than five hours, we should be able to get a production version in about two to three weeks, right?”
The facts of the matter were completely different however. The demo, while running with the firm’s own data, had been running from a spreadsheet, housed on the laptop of the consultant and procured after several weeks of scrubbing, formatting, and aggregating data from the CIO’s team; this does not even mention the preceding data procurement process. And so, as the expert in the room, the voice of reason, the CIO turned to me wanting to know how long it would take to implement the solution. At least six months, was my assessment. I had seen their data, and it was a mess. I had seen the flow, not a model architecture and the sheer volume of data was daunting. If it was not architected correctly, the pretty colors and graphs would take much longer to refresh; this was not the answer he wanted to hear.
The advancement of social media, new web experiences and cutting edge mobile technology have driven users to expect more of their applications. As enterprises push to drive value and unlock more potential in their data, insurers of all sizes have attempted to implement analytical and business intelligence systems. But here’s the truth: by and large most insurance enterprises are not in a place with their data to make effective use of the new technologies in BI, mobile or social. The reality is that data cleanliness, fit for purpose, movement and aggregation is being done in a BI when it should be done lower down so that all applications can take advantage of it.
Let’s face it – quality data is important. Movement and shaping of data in the enterprise is important. Identification of master data and metadata in the enterprise is important and data governance is important. It brings to mind episode 165, “The Apology”, of the mega-hit show Seinfeld. Therein George Costanza accuses erstwhile friend Jason Hanky of being a “step skipper”. What I have seen in enterprise data is “step skipping” as users clamor for new and better experiences, but the underlying infrastructure and data is less than ready for consumption. So the enterprise bootstraps, duct tapes and otherwise creates customizations where it doesn’t architecturally belong.
Clearly this calls for a better solution; A more robust and architecturally sustainable data ecosystem, which shepherds the data from acquisition through to consumption and all points in between. It also must be attainable by even modestly sized insurance firms.
First, you need to bring the data under your control. That may mean external data integration, or just moving it from transactional, web, or client-server systems into warehouses, marts or other large data storage schemes and back again. But remember, the data is in various stages of readiness. This means that through out of the box or custom cleansing steps the data needs to be processed, enhanced and stored in a way that is more in line with corporate goals for governing the quality of that data. And this says nothing of the need to change a data normalization factor between source and target. When implemented as a “factory” approach, the ability to bring new data streams online, integrate them quickly and maintain high standards become small incremental changes and not a ground up monumental task. Move your data shaping, cleansing, standardization and aggregation further down in the stack and many applications will benefit from the architecture.
Critical to this process is that insurance enterprises need to ensure the data remains secure, private and is managed in accordance with rules and regulations. They must also govern the archival, retention and other portions of the data lifecycle.
At any point in the life of your information, you are likely sending or receiving data from an agent, broker, MGA or service provider, which needs to be processed using the robust ecosystem, described above. Once an effective data exchange infrastructure is implemented, the steps to process the data can nicely complement your setup as information flows to and from your trading partners.
Finally, as your enterprise determines “how” to implement these solutions, you may look to a cloud based system for speed to market and cost effectiveness compared to on-premises solutions.
And don’t forget to register for Informatica World 2014 in Las Vegas, where you can take part in sessions and networking tailored specifically for insurers.
“We have 20% duplicates in our data source”. This is how the conversation began. It was not that no one cared about the level of duplicates, it’s just that the topic of duplicate records did not get the business excited – they have many other priorities (and they were not building a single view of customer).
The customer continued the discussion thread on how to make data quality relevant to each functional leader reporting to C-level executives. The starting point was affirmation that the business really only care about data quality when it impacts the processes that they own e.g. order process, invoice process, shipping process, credit process, lead generation process, compliance reporting process, etc. This means that data quality results need to be linked to the tangible goals of each business process owner to win them over as data advocates. (more…)
Today Informatica completed the acquisition of Agent Logic.
Whenever a company acquires another, the first question is why. I wanted to give my personal perspective on why this deal is a good one for the industry and our customers. In addition to my comments below, you can read the press release: Informatica Acquires Agent Logic for more information. (more…)