Hosting Big Data applications in the cloud has compelling advantages. Scale doesn’t become as overwhelming an issue as it is within on-premise systems. IT will no longer feel compelled to throw more disks at burgeoning storage requirements, and performance becomes the contractual obligation of someone else outside the organization.
Cloud may help clear up some of the costlier and thornier problems of attempting to manage Big Data environments, but it also creates some new issues. As Ron Exler of Saugatuck Technology recently pointed out in a new report, cloud-based solutions “can be quickly configured to address some big data business needs, enabling outsourcing and potentially faster implementations.” However, he adds, employing the cloud also brings some risks as well.
Data security is one major risk area, and I could write many posts on this. But management issues also present other challenges. Too many organizations see cloud as an cure-all for their application and data management ills, but broken processes are never fixed when new technology is applied to them. There are also plenty of risks with the misappropriation of big data, and the cloud won’t make these risks go away. Exler lists some of the risks that stem from over-reliance on cloud technology, from the late delivery of business reports to the delivery of incorrect business information, resulting in decisions based on incorrect source data. Sound familiar? The gremlins that have haunted data analytic and management for years simply won’t disappear behind a cloud.
Exler makes three recommendations for moving big data into cloud environments – note that the solutions he proposes have nothing to do with technology, and everything to do with management:
1) Analyze the growth trajectory of your data and your business. Typically, organizations will have a lot of different moving parts and interfaces. And, as the business grows and changes, it will be constantly adding new data sources. As Exler notes, “processing integration or hand off points in such piecemeal approaches represent high risk to data in the chain of possession – from collection points to raw data to data edits to data combination to data warehouse to analytics engine to viewing applications on multiple platforms.” Business growth and future requirements should be analyzed and modeled to make sure cloud engagements will be able “to provide adequate system performance, availability, and scalability to account for the projected business expansion,” he states.
2) Address data quality issues as close to the source as possible. Because both cloud and big data environments have so many moving parts, “finding the source of a data problem can be a significant challenge,” Exler warns. “Finding problems upstream in the data flow prevent time-consuming and expensive reprocessing that could be needed should errors be discovered downstream.” Such quality issues have a substantial business cost as well. When data errors are found, it becomes “an expensive company-wide fire drill to correct the data,” he says.
3) Build your project management, teamwork and communication skills. Because big data and cloud projects involve so many people and components from across the enterprise, requiring coordination and interaction between various specialists, subject matter experts, vendors, and outsourcing partners. “This coordination is not simple,” Exler warns. “Each group involved likely has different sets of terminology, work habits, communications methods, and documentation standards. Each group also has different priorities; oftentimes such new projects are delegated to lower priority for supporting groups.” Project managers must be leaders and understand the value of open and regular communications.
There are organizations truly reaping the rewards of Big Data, and then there are those who are just trying to catch up. What are the Big Data “leaders” doing that the “laggards” are missing? (more…)
Evolving from Chaos to Competitiveness: The Emerging Architecture of Next-Generation Data Integration
To compete on Big Data and analytics, today’s always-on enterprise needs a well-designed evolving high-level architecture that continuously provides trusted data originating from a vast and fast-changing range of sources, often with different formats, and within different contexts.
To meet this challenge, the art and science of data integration is evolving, from duplicative, project-based silos that have consumed organizations’ time and resources to an architectural approach, in which data integration is based on sustainable and repeatable data integration practices – delivering data integration automatically anytime the business requires it. (more…)
Last fall, The New York Times resident numbers geek Nate Silver provided a lesson in predictive analytics for the whole world to see – crunching big data to predict, with almost pinpoint accuracy – the winner of the U.S. presidential election.
The success of this high-profile project thrust big data analytics into the limelight, but there are many, somewhat more mundane applications, but with even more unforeseen revelations. (more…)
Data scientist may be the hot job of 2013, but many data professionals report they are already doing much of the work that would be defined as the data scientist role. They just aren’t calling themselves data scientists – at least not yet.
In a new survey of 199 data managers I conducted as part of my work with Unisphere Research and Information Today, Inc., we found that the traits of data scientists – individuals whose backgrounds include IT and programming; math and statistics; and a willingness to look at things differently—are already seen within today’s organizations, in the day to day work performed by database administrators, analysts, managers and consultants. The survey was conducted among members of the Independent Oracle Users Group. (more…)
Finally, there is now evidence of a clear link between financial performance and the broad use of data by employees. Specifically, organizations that take the lead in data analytics are more than three times more likely to be leaders within their industry groups than companies with standard analytics environments.
That’s the finding of a new survey of 530 senior executives, conducted by the Economist Intelligence Unit. There is little disagreement that the ability to make data available across the entire enterprise means greater productivity and performance. More than 80 percent of respondents believe that employees across their organizations “can and should be using data to do their jobs.” (more…)
Many organizations are rushing into big data efforts before it’s clear what business benefits will come from this new paradigm. And this is creating problems for big data analytics proponents. “At one very large financial services firm, we’ve heard that the next executive that uses the word ‘big data’ without a very precise explanation of how it will be used for the organization will be fired,” says Randy Bean, co founder of NewVantage Partners, quoted in MIT Sloan Management Review.
“The point is that there’s been so much overuse and misuse of the term that organizations need and want to understand precisely how big data capabilities and big data initiatives will help them,” he explains. (more…)
From the “it’s-About-Time” Department: More enterprises are embracing – or will soon be embracing – access to data analytics via mobile apps.
Having analytics available in a simple app fashion could be a major boost for efforts to “democratize” analytics in organizations. I once heard Competing on Analytics guru and best-selling author Tom Davenport wonder out loud at a conference why there weren’t more analytics being made available as a “cute little app.” By offering analytics through simple, single-purpose mobile apps, decision-making can be brought into a whole new realm. “I’ve heard of 50 analytical apps for the iPhone so far,” he points out. Examples include a nursing-productivity app, a truck-loading analysis app, and a social sentiment analysis app. (more…)