Tag Archives: Architecture
Today, I am going to share on what few others have so far been willing to share in public regarding big data. Before doing so, I need to first bring you up to speed on what I have already written on the topic. I shared previously that the term big data probably is not very useful or descriptive. Thomas Davenport, the author of Competing on Analytics, has found as well that “over 80 percent of the executives surveyed thought the term was overstated, confusing, or misleading”. At the same time, the CIOs that I have talking to have suggested that I tell our reps never to open a meeting with the big data topic but instead to talk about the opportunity to relate the volumes of structured data to even larger volumes of unstructured data.
I believe that the objective of these discussions increasingly should be to discover how it is possible to solve even bigger and meatier business problems. It is important, as I said recently; to take a systems view of big data and this includes recognizing that business stakeholders will not use any data unless it is trustworthy, regardless of cost. Having made these points previously, I would like to bring to the forefront another set of issues that businesses should consider before beginning a big data implementation. I have come to my point of view here by listening to the big data vanguard. Many of these early adopters have told me that they jumped onto the big data bandwagon because they heard big data would be cheaper than traditional business intelligence implementations.
However, these enterprises soon discovered that they couldn’t leer away from their Silicon Valley jobs those practiced in the fine arts of HADOOP and MapReduce. They found as well that hand coding approaches and the primitive data collection tools provided by the HADOOP vendors were not ready for prime time and did not by themselves save cost. These early pioneers found that they needed a way to automate the movement and modification of data for analysis. What they determine was needed is an automated, non-script based way to pull and transform the data that populates their HADOOP or other big data type systems. This included real time solutions like Hana and Vertica.
A new architecture for business intelligence
But as I have looked further into the needs of these early adopters, it became clear that they needed an architecture that could truly manage their end to end business intelligence requirements. They needed an architecture that would handle their entire data use lifecycle from the collection, inspection, connection, perfection, and protection of data.
Architecture requires a Data Lake
Obviously, I have already been discussing what could be called the collection phase. But to be clear, big data should be just one element of a larger collection scheme. No one is suggesting for example that existing business intelligence systems be replaced in wholesale fashion with the so called newer approaches. Given this, business architecture needs to start by establishing a data lake approach that over arches the new data storage approaches and effectively sits side by side with existing business intelligence assets.
Data discovery starts by testing data relationships
Once new forms of data are collected using HADOOP or other forms of big data storage within an overarching data lake, users and analysts need to inspect the data collected as whole and surface interrelationships with new and existing forms of data. What is needed in addition to data movement is a lake approach to deploy data and evaluate data relationships. Today, this involves enabling business intelligence users to self-service. One CIO that heard about this lit up and said “this is like orchestration. Users can assembly data and put it together and do it from different sources at different times. It doesn’t just have to be a preconceived process.”
Data Enrichment enables business decision making
Historically users needed to know what data they wanted for analyze prior to building a business intelligence system. An advantage of HADOOP plus an overarching data lake is that you can put data in a place prior to knowing if the data has an interesting business use case or not. Once data is captured, it needs tooling to evaluate and put together data and test the strength of potential data relationships. This includes enabling business users to evaluate the types of analytics that could potentially have value to them. I shared recently on just how important it was to visualize data in a way that culturally fits and derives the most potential business value.
Once data has been evaluated and relevant data relationships have been determined, then it is important to have a way to siphon off data that has been determined to have potential business interest and do what you always did to this data. This includes adding meaningful additional structure and relationship to the data and fixing the quality of data that needs to be related and created within an analytic. This can include things like data mastering. This can mean that one of two things takes place. First is data relationships are extended and data quality and consistency are improved. In this data perfection stage, it can for finance mean integrating and then consolidating data for a total view of the financial picture. For marketing people, it can involving creating an integrated customer record fusing together existing customer master data with external customer datasets to improve cross sell and customer service. With this accomplished it becomes an analysis decision and a cost decision whether data continues to be housed in HADOOP or managed in an existing traditional data warehouse structure.
Valuable data needs to be protected
Once data is created that can be used for business decision making, then we need to take the final step of protecting the data that often cost millions of dollars to create, refine, and analyze. Recently, I was with a CIO and asked about various hacks that have captured so much media attention. This CIO said that the CIOs at the companies that had been hacked were not stupid. It is hard to justify the business value of protecting the value of the data that has been created. It seems clear to me at least that we need to protect data as an asset and as well the external access to it given the brand and business impacts of being hacked.
It seems clear that we need an architecture that is built to last and deliver sustaining value to the business. So here is the cycle again–collection, inspection, connection, perfection, and protection of data. Each step matters to big data but as well to the data architecture that big data is adding onto.
Author Twitter: @MylesSuer
The start of the year is a great time to refresh and take a new look at your capabilities, goals, and plans for your future-state architecture. That being said, you have to take into consideration that the most scarce resource in your architecture is probably your own personal time.
Looking forward, here are three things that I would recommend that every architect do. I realize that all three of these relate to data, but as I have said in the eBook, Think “Data First” to Drive Business Value, we believe that data is the key bottleneck in your enterprise architecture in terms of slowing the delivery of business initiatives in support of your organization’s business strategy.
So, here are the recommendations. None of these will cost you anything if you are a current Informatica PowerCenter customer. And #2 and #3 are free regardless. It is only a matter of your time:
1. Take a look at the current Informatica Cloud offering and in particular the templating capabilities.
Informatica Cloud is probably much more capable than you think. The standard templating functionality supports very complex use cases and does it all from a very easy to use, no-coding, user interface. It comes with a strong library of integration stubs that can be dragged & dropped into Microsoft Viseo to create complex integrations. Once the flow is designed in Viseo, it can be easily imported into Informatica Cloud and from there users have a Wizard-driven UI to do the final customization for sources, targets, mappings, transformations, filters, etc. It is all very powerful and easy to use.
- YouTube: Building Custom templates https://www.youtube.com/watch?v=yHmFkxov6bs
- 30 day free Informatica Cloud trial. http://more.informatica.com/en/cloud_trial/org?offer=30day-ICwebPage
Why This Matters to Architects
- You will see how easy it is for new groups to get going with fairly complex integrations.
- This is a great tool for departmental or new user use, and it will be completely compatible with the rest of your Informatica architecture – not another technology silo for you to manage.
- Any mapping created for Informatica on-premise can also run on the cloud version.
2. Download Informatica Rev and understand what it can do for your analysts and “data wranglers.”
Your data analysts are spending 80% of their time managing their data and only 20% on the actual analysis they are trying to provide. Informatica Rev is a great way to prepare your data before use in analytics tools such as Qlik, Tableau, and others.
With Informatica Rev, people who are not data experts can access, mashup, prototype and cleanse their data all in a User Interface that looks like a spreadsheet and requires no previous experience in data tools.
- For a free Informatica Rev download https://rev.informatica.com/
- Informatica Rev (Project Springbok) demo https://www.youtube.com/watch?v=0F_58bHKDDs
Why This Matters for Architects
- Your data analysts are going to use analytics tools with or without the help of IT. This enables you to help them while ensuring that they are managing their data well and optimizing their productivity.
- This tool will also enable them to share their “data recipes” and for IT to be involved in how they access and use the organization’s data.
3. Look at the new features in PowerCenter 9.6. First, upgrade to 9.6 if you haven’t already, and particularly take a good look at these new capabilities that are bundled in every version. Many people we talk to have 9.6 but don’t realize the power of what they already own.
- Profiling: Discover and analyze your data quickly. Find relationships and data issues.
- Data Services: This presents any JDBC or ODBC repository as a logical data object. From there you can rapidly prototype new applications using these logical objects without worrying about the complexities of the underlying repositories. It can also do data cleansing on the fly.
- Webinar: Great Data by Design. https://www.brighttalk.com/webcast/10477/104939
- PowerCenter 9.6 deep dive demo https://www.brighttalk.com/webcast/10477/110535
Why This Matters for Architects
- The key challenge for IT and for Architects is to be able to deliver at the “speed of business.” These tools can dramatically improve the productivity of your team and speed the delivery of projects for your business “customers.”
Taking the time to understand what these tools can do in terms of increasing the productivity of your IT team and enabling your end users to self-service will make you a better business partner overall and increase your influence across the organization. Have a great year!
In my previous blog, I talked about how a business-led approach can displace technology-led projects. Historically IT-led projects have invested significant capital while returning minimal business value. It further talks about how transformation roadmap execution is sustainable because the business is driving the effort where initiative investments are directly traceable to priority business goals.
For example, an insurance company wants to improve the overall customer experience. Mature business architecture will perform an assessment to highlight all customer touch points. It requires a detailed capability map, fully formed, customer-triggered value streams, value stream/ capability cross-mappings and stakeholder/ value stream cross-mappings. These business blueprints allow architects and analysts to pinpoint customer trigger points, customer interaction points and participating stakeholders engaged in value delivery.
One must understand that value streams and capabilities are not tied to business unit or other structural boundaries. This means that while the analysis performed in our customer experience example may have been initiated by a given business unit, the analysis may be universally applied to all business units, product lines and customer segments. Using the business architecture to provide a representative cross-business perspective requires incorporating organization mapping into the mix.
Incorporating the application architecture into the analysis and proposed solution is simply an extension of business architecture mapping that incorporates the IT architecture. Robust business architecture is readily mapped to the application architecture, highlighting enterprise software solutions that automate various capabilities, which in turn enable value delivery. Bear in mind, however, that many of the issues highlighted through a business architecture assessment may not have corresponding software deployments since significant interactions across the business tend to be manual or desktop-enabled. This opens the door to new automation opportunities and new ways to think about business design solutions.
Building and prioritizing the transformation strategy and roadmap is dramatically simplified once all business perspectives needed to enhance customer experience are fully exposed. For example, if customer service is a top priority, then that value stream becomes the number one target, with each stage prioritized based on business value and return on investment. Stakeholder mapping further refines design approaches for optimizing stakeholder engagement, particularly where work is sub-optimized and lacks automation.
Capability mapping to underlying application systems and services provides the basis for establishing a corresponding IT deployment program, where the creation and reuse of standardized services becomes a focal point. In certain cases, a comprehensive application and data architecture transformation becomes a consideration, but in all cases, any action taken will be business and not technology driven.
Once this occurs, everyone will focus on achieving the same goals, tied to the same business perspectives, regardless of the technology involved.
- Home Hubs from Google, Samsung, and Apple (who did not attend the show but still had a significant impact).
- Home Hub Ecosystems providing interoperability with cars, door locks, and household appliances.
- Autonomous cars, and intelligent cars
- Wearable devices such as smart watches and jewelry.
- Drones that take pictures and intelligently avoid obstacles. …Including people trying to block them. There is a bit of a creepy factor here!
- The next generation of 3D printers.
- And the intelligent baby pacifier. The idea is that it takes the baby’s temperature, but I think the sleeper hit feature on this product is the ability to locate it using GPS and a smart phone. How much money would you pay to get your kid to go to sleep when it is time to do so?
Digital Strategies Are Gaining Momentum
There is no escaping the fact that the vast majority of companies out there have active digital strategies, and not just in the consumer space. The question is: Are you going to be the disruptor or the disruptee? Gartner offered an interesting prediction here:
“By 2017, 60% of global enterprise organizations will execute on at least one revolutionary and currently unimaginable business transformation effort.”
It is clear from looking at CES, that a lot of these products are “experiments” that will ultimately fail. But focusing too much on that fact is to risk overlooking the profound changes taking place that will shake out industries and allow competitors to jump previously impassible barriers to entry.
IDC predicted that the Internet of Things market would be over $7 Trillion by the year 2020. We can all argue about the exact number, but something major is clearly happening here. …And it’s big.
Is Your Organization Ready?
A study by Gartner found that 52% of CEOs and executives say they have a digital strategy. The problem is that 80% of them say that they will “need adaptation and learning to be effective in the new world.” Supporting a new “Internet of Things” or connected device product may require new business models, new business processes, new business partners, new software applications, and require the collection and management of entirely new types of data. Simply standing up a new ERP system or moving to a cloud application will not help your organization to deal with the new business models and data complexity.
Architect’s Call to Action
Now is the time (good New Year’s resolution!) to get proactive on your digital strategy. Your CIO is most likely deeply engaged with her business counterparts to define a digital strategy for the organization. Now is the time to be proactive in terms of recommending the IT architecture that will enable them to deliver on that strategy – and a roadmap to get to the future state architecture.
Key Requirements for a Digital-ready Architecture
Digital strategy and products are all about data, so I am going to be very data-focused here. Here are some of the key requirements:
- First, it must be designed for speed. How fast? Your architecture has to enable IT to move at the speed of business, whatever that requires. Consider the speed at which companies like Google, Amazon and Facebook are making IT changes.
- It has to explicitly directly link the business strategy to the underlying business models, processes, systems and technology.
- Data from any new source, inside or outside your organization, has to be on-boarded quickly and in a way that it is immediately discoverable and available to all IT and business users.
- Ongoing data quality management and Data Governance must be built into the architecture. Point product solutions cannot solve these problems. It has to be pervasive.
- Data security also has to be pervasive for the same reasons.
- It must include business self-service. That is the only way that IT is going to be able to meet the needs of business users and scale to the demands of the changes required by digital strategy.
For a webinar on connecting business strategy to the architecture of business transformation see; Next-Gen Architecture: A “Business First” Approach for Agile Architecture. With John Schmidt of Informatica and Art Caston, founder of Proact.
For next-generation thinking on enterprise data architectures see; Think “Data First” to Drive Business Value
For more on business self-service for data preparation and a free software download.
Transformation roadmaps in many businesses tend to have a heavy technology focus, to the point where organizations invest millions of dollars in initiatives with no clear business value. In addition, numerous tactical projects funded each year have little understanding of how or even if, they align from a business perspective. Management often fall victim to the latest technology buzzwords, while stakeholder value, business issues, and strategic considerations take a backseat. When this happens, executives who should be focused on business scenarios to improve stakeholder value fall victim to technology’s promise of the next big thing.
I recently participated in the writing and reviewing a series of whitepapers on Business-led transformation at Informatica’s Strategic Services Group. These whitepapers discusses how executives can leverage business architecture to reclaim their ability to drive a comprehensive transformation strategy and roadmap. I will try to summarize them into this blog.
Consider the nature of most initiatives found within a corporate program office. They generally focus on enhancing one system or another, or in more extreme cases a complete rebuild. The scope of work is bounded by a given system, not by the business focal point, whether that is a particular business capability, stakeholder, or value delivery perspective. These initiatives generally originate within the IT organization, not the business, and launched in response to a specific business need quickly translated into a software enhancement, rewrite, or database project. Too often, however, these projects have myopia and lack an understanding of cross-impacts to other projects, business units, stakeholders, or products. Their scope is constrained, not by a given customer or business focus, but by technology.
Business led transformation delivers a value centric perspective and provides the underlying framework for envisioning and crafting a more comprehensive solution. In some cases, this may begin with a quick fix if that is essential, but this must be accompanied by a roadmap for a more transformative solution. It provides a more comprehensive issue analysis and planning perspective because it offers business specific, business first viewpoints that enable issue analysis and resolution through business transparency.
On November 13, 2014, Informatica acquired the assets of Proact, whose Enterprise Architecture tools and delivery capability link architecture to business strategy. The BOST framework is now the Informatica Business Transformation Toolkit which received high marks in a recent research paper:
“(BOST) is a framework that provides four architectural views of the enterprise (Business, Operational, Systems, and Technology). This EA methodology plans and organizes capabilities and requirements at each view, based on evolving business and opportunities. It is one of the most finalized of the methodologies, in use by several large enterprises.”  (more…)
This got me thinking: What is the biggest bottleneck in the delivery of business value today? I know I look at things from a data perspective, but data is the biggest bottleneck. Consider this prediction from Gartner:
“Gartner predicts organizations will spend one-third more on app integration in 2016 than they did in 2013. What’s more, by 2018, more than half the cost of implementing new large systems will be spent on integration. “
When we talk about application integration, we’re talking about moving data, synchronizing data, cleansing, data, transforming data, testing data. The question for architects and senior management is this: Do you have the Data Foundation for Execution you need to drive the business results you require to compete? The answer, unfortunately, for most companies is; No.
All too often data management is an add-on to larger application-based projects. The result is unconnected and non-interoperable islands of data across the organization. That simply is not going to work in the coming competitive environment. Here are a couple of quick examples:
- Many companies are looking to compete on their use of analytics. That requires collecting, managing, and analyzing data from multiple internal and external sources.
- Many companies are focusing on a better customer experience to drive their business. This again requires data from many internal sources, plus social, mobile and location-based data to be effective.
When I talk to architects about the business risks of not having a shared data architecture, and common tools and practices for enterprise data management, they “get” the problem. So why aren’t they addressing it? The issue is that they find that they are only funded to do the project they are working on and are dealing with very demanding timeframe requirements. They have no funding or mandate to solve the larger enterprise data management problem, which is getting more complex and brittle with each new un-connected project or initiative that is added to the pile.
Studies such as “The Data Directive” by The Economist show that organizations that actively manage their data are more successful. But, if that is the desired future state, how do you get there?
Changing an organization to look at data as the fuel that drives strategy takes hard work and leadership. It also takes a strong enterprise data architecture vision and strategy. For fresh thinking on the subject of building a data foundation for execution, see “Think Data-First to Drive Business Value” from Informatica.
* By the way, Informatica is proud to announce that we are now a sponsor of the MIT Center for Information Systems Research.
If you have been following publications in the Potential at Work Community or any number of Linkedin discussions such this one on the DrJJ group (a think-tank for information management best practices), you will have noticed the Agile methodology topic come up time and time again. For instance, check out the article Architect Your Way From Sluggish to Speed or the video Focus on Agility Adaptability. It hasn’t always been this way. For many years the architectural focus was on RASP.
In this video, Rob Karel, vice president of product strategy, Informatica, outlines the Informatica Data Governance Framework, highlighting the 10 facets that organizations need to focus on for an effective data governance initiative:
- Vision and Business Case to deliver business value
- Tools and Architecture to support architectural scope of data governance
- Policies that make up data governance function (security, archiving, etc.)
- Measurement: measuring the level of influence of a data governance initiative and measuring its effectiveness (business value metrics, ROI metrics, such as increasing revenue, improving operational efficiency, reducing risk, reducing cost or improving customer satisfaction)
- Change Management: incentives to workforce, partners and customers to get better quality data in and potential repercussions if data is not of good quality
- Organizational Alignment: how the organization will work together across silos
- Dependent Processes: identifying data lifecycles (capturing, reporting, purchasing and updating data into your environment), all processes consuming the data and processes to store and manage the data
- Program Management: effective program management skills to build out communication strategy, measurement strategy and a focal point to escalate issues to senior management when necessary
- Define Processes that make up the data governance function (discovery, definition, application and measuring and monitoring).
For more information from Rob Karel on the Informatica Data Governance Framework, visit his Perspectives blogs.
Those moving to cloud computing have their work cut out of for them. They need to pick a parcel of data, applications, or both to migrate to a cloud-based service. Or, perhaps build a system from the ground up on a cloud platform.
In any event, you need a few things to insure success, including a good architecture, a deployment plan, and a sound data integration strategy. (more…)