Tag Archives: data analytics
Well, so much for conventional wisdom! The scalability of Hadoop, and the emergence of highly elastic Cloud analytic services like AWS EMR and Redshift, Microsoft Azure, and Salesforce Wave Analytics have invalidated the last argument against moving all IT infrastructure to the Cloud. So here’s my own proclamation:
Cloud Analytics is the killer application that will accelerate the coming of a Cloud-only world within the next decade.
At Informatica, we are in the business of data – of ensuring that businesses are able to integrate, relate, manage, and secure their data. In my 10 years at Informatica, I have seen the evolution of different analytics platforms – specialized OLAP databases, columnar databases, analytics appliances, Hadoop, and agile BI applications. However, I am convinced that Cloud Analytics is the end state of Analytics for the coming decades – the omega of Analytics so to speak. The infinite scalability, the unbeatable price performance, and the simplicity of offering domain-specific, and user-specific capabilities through the Cloud makes it an unbeatable choice. Cloud Analytics has not just arrived, but it has made an amazing splash and proven that it is here to dominate.
At Informatica, we have built a Cloud Integration Platform-as-a-Service (iPaaS) that is designed from the ground up to support the scalability, reliability and security required for Cloud Analytics. Our customers process over 6 Billion transactions every day using the Informatica Cloud; to integrate data into AWS Redshift, Azure, and Salesforce Wave. Use cases range from sales analytics to analysis of IoT data. It’s exciting to see how Informatica Cloud is fast becoming THE Cloud to connect all Clouds to each other (and even to on-premise applications). It will be interesting to see how iPaaS leaders like Informatica partner with Cloud Analytics vendors to move our industry from a “Cloud-first” landscape to a “Cloud-only” future.
Last fall, at a large industry conference, I had the opportunity to conduct a series of discussions with industry leaders in a portable video studio set up in the middle of the conference floor. As part of our exercise, we had a visual artist do freeform storyboarding of the discussion on large swaths of five-foot by five-foot paper, which we then reviewed at the end of the session. For example, in a discussion of cloud computing, the artist drew a rendering of clouds, raining data on a landscape below, illustrated by sketches of office buildings. At a glance, one could get a good read of where the discussion went, and the points that were being made.
Data visualization is one of those up-and-coming areas that has just begin to breach the technology zone. There are some powerful front-end tools that help users to see, at a glance, trends and outliers through graphical representations – be they scattergrams, histograms or even 3D diagrams or something else eye-catching. The “Infographic” that has become so popular in recent years is an amalgamation of data visualization and storytelling. The bottom line is technology is making it possible to generate these representations almost instantly, enabling relatively quick understanding of what the data may be saying.
The power that data visualization is bringing organizations was recently explored by Benedict Carey in The New York Times, who discussed how data visualization is emerging as the natural solution to “big data overload.”
This is much more than a front-end technology fix, however. Rather, Carey cites a growing body of knowledge emphasizing the development of “perceptual learning,” in which people working with large data sets learn to “see” patterns and interesting variations in the information they are exploring. It’s almost a return of the “gut” feel for answers, but developed for the big data era.
As Carey explains it:
“Scientists working in a little-known branch of psychology called perceptual learning have shown that it is possible to fast-forward a person’s gut instincts both in physical fields, like flying an airplane, and more academic ones, like deciphering advanced chemical notation. The idea is to train specific visual skills, usually with computer-game-like modules that require split-second decisions. Over time, a person develops a ‘good eye’ for the material, and with it an ability to extract meaningful patterns instantaneously.”
Video games may be leading the way in this – Carey cites the work of Dr. Philip Kellman, who developed a video-game-like approach to training pilots to instantly “read” instrument panels as a whole, versus pondering every gauge and dial. He reportedly was able to enable pilots to absorb within one hour what normally took 1,000 hours of training. Such perceptual-learning based training is now employed in medical schools to help prospective doctors become familiar with complicated procedures.
There are interesting applications for business, bringing together a range of talent to help decision-makers better understand the information they are looking at. In Carey’s article, an artist was brought into a medical research center to help scientists look at data in many different ways – to get out of their comfort zones. For businesses, it means getting away from staring at bars and graphs on their screens and perhaps turning data upside down or inside-out to get a different picture.
When it comes to cloud-based data analytics, a recent study by Ventana Research (as found in Loraine Lawson’s recent blog post) provides a few interesting data points. The study reveals that 40 percent of respondents cited lowered costs as a top benefit, improved efficiency was a close second at 39 percent, and better communication and knowledge sharing also ranked highly at 34 percent.
Ventana Research also found that organizations cite a unique and more complex reason to avoid cloud analytics and BI. Legacy integration work can be a major hindrance, particularly when BI tools are already integrated with other applications. In other words, it’s the same old story:
The ability to deal with existing legacy systems when moving to concepts such as big data or cloud-based analytics is critical to the success of any enterprise data analytics strategy. However, most enterprises don’t focus on data integration as much as they should, and hope that they can solve the problems using ad-hoc approaches.
You can’t make sense of data that you can’t see.
These approaches rarely work as well a they should, if at all. Thus, any investment made in data analytics technology is often diminished because the BI tools or applications that leverage analytics can’t see all of the relevant data. As a result, only part of the story is told by the available data, and those who leverage data analytics don’t rely on the information, and that means failure.
What’s frustrating to me about this issue is that the problem is easily solved. Those in the enterprise charged with standing up data analytics should put a plan in place to integrate new and legacy systems. As part of that plan, there should be a common understanding around business concepts/entities of a customer, sale, inventory, etc., and all of the data related to these concepts/entities must be visible to the data analytics engines and tools. This requires a data integration strategy, and technology.
As enterprises embark on a new day of more advanced and valuable data analytics technology, largely built upon the cloud and big data, the data integration strategy should be systemic. This means mapping a path for the data from the source legacy systems, to the views that the data analytics systems should include. What’s more, this data should be in real operational time because data analytics loses value as the data becomes older and out-of-date. We operate a in a real-time world now.
So, the work ahead requires planning to occur at both the conceptual and physical levels to define how data analytics will work for your enterprise. This includes what you need to see, when you need to see it, and then mapping a path for the data back to the business-critical and, typically, legacy systems. Data integration should be first and foremost when planning the strategy, technology, and deployments.
A lot of media reports have been surfacing lately about “secretive” data mining activities taking place within the presidential campaign. Many articles paint the efforts with a sinister caste, implying that underhanded invasions of privacy are taking place.
But to any seasoned data professional, data mining is a discovery tool that pulls nuggets of insight out of mountains of data. For any business that wants to get ahead in today’s hyper-competitive global economy, advanced data mining and analysis is not a luxury, it is a necessity. As USA Today’s Jack Gillum describes the Romney campaign’s data analytics: (more…)
In this video, Richard Cramer, chief healthcare strategist, and Scott Fingerhut, senior director, product marketing, CEP, Informatica, discuss healthcare and CEP (Complex Event Processing).
Richard and Scott cover the following topics:
– What is CEP;
– How CEP pertains to healthcare;
– How CEP differs from data warehouse analytics;
– What some of the applications of CEP are in the healthcare environment; and,
– Where the opportunities are for companies who have already invested heavily in meaningful use and EHRs.