In Today’s Cloud Polyglot, Data Integration Takes Center Stage
These days, just about every enterprises is looking to deploy cloud or Software as a Service in one form or another – from messaging and data storage services to running mission-critical applications. The result is a polyglot computing environments, with data being moved about and residing both on-premises and in the cloud. It isn’t just one or two clouds where data may end up – it may end up being transferred between on-premises systems and public clouds, between private and public clouds, between different hybrid clouds, or between different public clouds.
There are two sides to the cloud data integration picture. One is the need to bring data together from no matter where it resides, be it in the cloud or from traditional on-premises systems. The other is serving as an integration platform itself. Extract, transform and load (ETL) and data replication approaches — long the standards for traditional, on-premises relational environments – continue to dominate within enterprises, used by a majority of organizations, a new survey of 342 data managers finds. In many cases, enterprise data shops still accomplish integrate via manual scripting. Significantly, cloud has entered the picture as a solution to data integration challenges, with close to one-third of respondents employing cloud-based services for application or data integration.
The survey, conducted by Unisphere Research/Information Today, Inc., among members of members of the Independent Oracle Users Group (IOUG), confirmed that most enterprises are now implementing or considering cloud-based applications and infrastructure to manage their mission-critical systems. Much of the movement has been to private and hybrid cloud architectures, but public cloud adoption is not too far behind. I helped design and analyze the survey as part of my work with Unisphere/ITI, which also found that close to 30% have public Software as a Service, and 16% use public Platform as a Service or Infrastructure as a Service. Among cloud adopters, close to one-third deploy Database as a Service or Application Platform as a Service, and one in five deploy Data Integration as a Service.
Traditional methods of data integration, which may have worked well for organizations over the decades, are no longer be enough in today’s highly flexible and fast-moving cloud environments. Organizations are increasingly demanding real-time views and insights into data coming from both within and outside their organizations, which needs to be rapidly synced, stored, and managed—while maintaining peak, always-on performance. Close to one-third of data managers report they even still employ manual scripting as a data integration method, which is too overwhelming as cloud and big data adoption accelerates.
There is a rising level of comfort and confidence with cloud services, particularly private and hybrid clouds, which are now part of the corporate IT scene. Clouds are not only storing and processing data, but part of complex behind-the-scenes integration work. At least 31% of the companies in the survey now employ cloud-based integration solutions or platforms. Those leveraging data integration within cloud environments report they are seeing faster data movement to target applications (cited by 32%), increased business agility (32%), greater application interoperability (32%), and reduced costs (31%).
At the same time, while cloud is becoming an integration platform in its own right, data integration requirements may be holding back cloud deployments. Data integration is a necessity in the cloud plans of a majority of enterprises in the survey. For cloud implementers, at least nine out of 10 state that data integration is important to their efforts. About one in four data managers say application and data interoperability is one of their top challenges in moving to cloud. This rises to 42% of the public cloud users in the survey.
Half of enterprises also now require real-time data synchronization data between cloud and on-premises systems —reflecting the challenges and opportunities that lie ahead. Cloud adopters have a stronger need for real-time or near real-time data synchronization than their peers that have not adopted cloud yet. Those respondents currently using private/hybrid or public clouds mention data integration as a key challenge more frequently than who have not moved to cloud. Forty-two percent of public cloud users mention data integration as a challenge for cloud projects, compared to 19% of non-cloud users in the survey.