Category Archives: Architects
What is digitization?
It can take many forms. Here are a few types of digitization of business and examples:
|Products that add digital components||Sports equipment with sensors for immediate feedback|
|Products sold through digital channels||Conde Nast magazines|
|“Solutions” that are assembled and delivered in digital channels||USAA Insurance|
|Products that are entirely digital||Apple iTunes, eSurance, PayPal, Google|
|Companies monetizing their data||Healthcare clinical data|
The really interesting thing about digitization that you can see from some of the examples above is that it enables new competition to enter your space and competitors to leap industry boundaries. The concept of “barriers to entry” itself is eroding.
The Impact of Digitization on IT
Some interesting facts from MIT CISR’s research with Boards of Directors on digitization jumped out at me:
- Board members estimate that 32% of company’s revenues are under threat from digital disruption. This is a really stunning number when you think about it.
- Half of Board members believe that their board’s ability to oversee the strategic use of IT is “less than effective.”
- 26% of Boards hired consultants to evaluate major projects or the IT unit.
- 60% of Boards want to spend more time on digital issues next year.
The Impact of Digitization for Architects?
It boils down to two things:
- Architects need to deliver a digital platform to enable business agility in a time of increasing competition and disruption. This includes standardization around business processes, data, and the platform.
- Architects need to get more proactive in the strategy process for their organizations both in terms of the platforms and architecture and in terms of a general understanding of the challenges and opportunities that arise from digital disruption.
For more on enterprise data architecture, best practices and reference architectures see the eBook: Think “Data First” to Drive Business Value
I ended my previous blog wondering if awareness of Data Gravity should change our behavior. While Data Gravity adds Value to Big Data, I find that the application of the Value is under explained.
Exponential growth of data has naturally led us to want to categorize it into facts, relationships, entities, etc. This sounds very elementary. While this happens so quickly in our subconscious minds as humans, it takes significant effort to teach this to a machine.
A friend tweeted this to me last week: I paddled out today, now I look like a lobster. Since this tweet, Twitter has inundated my friend and me with promotions from Red Lobster. It is because the machine deconstructed the tweet: paddled <PROPULSION>, today <TIME>, like <PREFERENCE> and lobster <CRUSTACEANS>. While putting these together, the machine decided that the keyword was lobster. You and I both know that my friend was not talking about lobsters.
You may think that this maybe just a funny edge case. You can confuse any computer system if you try hard enough, right? Unfortunately, this isn’t an edge case. 140 characters has not just changed people’s tweets, it has changed how people talk on the web. More and more information is communicated in smaller and smaller amounts of language, and this trend is only going to continue.
When will the machine understand that “I look like a lobster” means I am sunburned?
I believe the reason that there are not hundreds of companies exploiting machine-learning techniques to generate a truly semantic web, is the lack of weighted edges in publicly available ontologies. Keep reading, it will all make sense in about 5 sentences. Lobster and Sunscreen are 7 hops away from each other in dbPedia – way too many to draw any correlation between the two. For that matter, any article in Wikipedia is connected to any other article within about 14 hops, and that’s the extreme. Completed unrelated concepts are often just a few hops from each other.
But by analyzing massive amounts of both written and spoken English text from articles, books, social media, and television, it is possible for a machine to automatically draw a correlation and create a weighted edge between the Lobsters and Sunscreen nodes that effectively short circuits the 7 hops necessary. Many organizations are dumping massive amounts of facts without weights into our repositories of total human knowledge because they are naïvely attempting to categorize everything without realizing that the repositories of human knowledge need to mimic how humans use knowledge.
For example – if you hear the name Babe Ruth, what is the first thing that pops to mind? Roman Catholics from Maryland born in the 1800s or Famous Baseball Player?
If you look in Wikipedia today, he is categorized under 28 categories in Wikipedia, each of them with the same level of attachment. 1895 births | 1948 deaths | American League All-Stars | American League batting champions | American League ERA champions | American League home run champions | American League RBI champions | American people of German descent | American Roman Catholics | Babe Ruth | Baltimore Orioles (IL) players | Baseball players from Maryland | Boston Braves players | Boston Red Sox players | Brooklyn Dodgers coaches | Burials at Gate of Heaven Cemetery | Cancer deaths in New York | Deaths from esophageal cancer | Major League Baseball first base coaches | Major League Baseball left fielders | Major League Baseball pitchers | Major League Baseball players with retired numbers | Major League Baseball right fielders | National Baseball Hall of Fame inductees | New York Yankees players | Providence Grays (minor league) players | Sportspeople from Baltimore | Maryland | Vaudeville performers.
Now imagine how confused a machine would get when the distance of unweighted edges between nodes is used as a scoring mechanism for relevancy.
If I were to design an algorithm that uses weighted edges (on a scale of 1-5, with 5 being the highest), the same search would yield a much more obvious result.
1895 births | 1948 deaths | American League All-Stars | American League batting champions | American League ERA champions | American League home run champions | American League RBI champions | American people of German descent | American Roman Catholics | Babe Ruth | Baltimore Orioles (IL) players | Baseball players from Maryland | Boston Braves players | Boston Red Sox players | Brooklyn Dodgers coaches | Burials at Gate of Heaven Cemetery | Cancer deaths in New York | Deaths from esophageal cancer | Major League Baseball first base coaches | Major League Baseball left fielders | Major League Baseball pitchers | Major League Baseball players with retired numbers | Major League Baseball right fielders | National Baseball Hall of Fame inductees | New York Yankees players | Providence Grays (minor league) players | Sportspeople from Baltimore | Maryland | Vaudeville performers .
Now the machine starts to think more like a human. The above example forces us to ask ourselves the relevancy a.k.a. Value of the response. This is where I think Data Gravity’s becomes relevant.
You can contact me on twitter @bigdatabeat with your comments.
We are way past the point where the architecture needs to be aligned with business goals and value delivery. That is necessary but no longer sufficient. We are now at the point where architecture needs to be central to the creation of an organization’s strategy process. Not to get hyperbolic, but anything less is risky for your career.
The Challenge: Digitization
I just came back from the MIT Center for Information Systems Research (CISR) research forum. One of the leading topics was digitization and how every business is becoming digitized. To those in the High Tech industry, this may be an “of course” topic, but to most other industries it is a wrenching change. Even those who are comfortable with the idea of digitization risk taking this too lightly.
The fact is that most products and services will have a digital component to them in the near future and an increasing number of products and services will be entirely digital. The fact is that digitization and the technologies that enable it are going to bring about a period of increased disruption. This will mean:
- New competitors. Examples: autonomous cars, sports equipment with embedded sensors that provide feedback, personal assistant fully capable of making decisions and taking action. Gartner is predicting that almost everything over $100 will have a sensor by the turn of the decade.
- New competitors jumping across industry boundaries. Examples: Apple iTunes and Google cars to name a few.
Why Architects Are Important
Architects are in a unique position to not only understand the technology trends driving this disruption, but they also to know how to leverage these trends to drive business value within their organizations. The very best architects are going to be those who are deeply involved in defining the organization strategy, not just figuring out how to implement it.
Evidence of Change
Many architects and CIOs currently report very little interest from upper management in IT. That is about to change, and quickly. At the MIT CISR forum I attended last week, they presented research around this area that is very telling:
- Half of Board of Directors members believe that their board’s ability to oversee the strategic use of IT is “less than effective.”
- 26% of Boards hired consultants to evaluate major projects or the IT unit.
- 60% of Boards want to spend more time on digital issues next year.
- Board members estimate that 32% of their company’s revenues are under threat from digital disruption.
That last bullet is the really interesting piece of research. 32% is a huge impact.
The Role of Data in Digitization
Digitization by its very nature is all about data. The winners in this space will be those that can manage and deliver relevant data the quickest. The question for architects is this: Do you have the architecture and agility to take advantage of the coming disruptions and opportunities? Are you actively advising your organization on how to leverage them? As we have documented in many previous blogs, many organizations are poorly positioned to manage their data as a discoverable and easily sharable asset. This will essential for:
- Delivering business initiatives and showing value faster (agility).
- Enabling business self-service so that IT is not the bottleneck in new analyses and decisions.
All of this requires new thinking around enterprise data architecture. For fresh thinking on this subject see Thinking “Data First” to Drive Business Value.
ERP systems were a true competitive advantage 20+ years ago, but not so today. ERP systems are a tool that gave people the best view into their business, but that is when there really were only ERP systems and Databases, but today that critical data resides in so many other areas. There are several reasons why ERP systems act as a data trap: technical factors, out of date management theory, and big data trends. First, let’s talk about management theory.
There are two fundamental concepts that have been driving much of the strategic planning in modern organizations in recent decades. The idea of economies of scale is deeply embedded in our thinking. The concept was first introduced by Adam Smith in the 18th century and reinforced throughout the 20th century by contemporaries such as Bruce Henderson. In 1968 Henderson wrote “”Costs characteristically decline by 20-30% in real terms each time accumulated experience doubles.“ The basic idea is that bigger is better. (more…)
Just last week, I visited a client for whom I had been consulting on-and-off for several years. On the meeting room wall, I saw their Enterprise Architecture portfolio, beautiful graphically designed and printed on a giant sheet of paper. My host proudly informed me how much she enjoyed putting that diagram together in 2009.
I jokingly reminded her of the famous notion of “art for art’s sake”; which is an appropriate phrase to describe what many architects are doing when populating frameworks. Indeed, when we refer to Enterprise Architecture, we must remember that the term ‘architecture’ is, itself, a metaphor.
In a tough economy, when competition is increasingly global and marketplaces are shifting, this ability to make tough decisions is going to be essential. Opportunities to save costs are going to be really valued, and architecture invariably helps companies save money. The ability to reuse, and thus rapidly seize the next related business opportunity, is also going to be highly valued.
The thing you have to be careful of is that if you see your markets disappearing, if your product is outdated, or your whole industry is redefining itself, as we have seen in things like media, you have to be ready to innovate. Architecture can restrict your innovative gene, by saying, “Wait, wait, wait. We want to slow down. We want to do things on our platform.” That can be very dangerous, if you are really facing disruptive technology or market changes.
Albert Camus wrote a famous essay exploring the Sisyphus myth called “The Myth of Sisyphus,” where he reinterpreted the central theme of the myth. Similarly, we need to challenge the myths of Enterprise Architecture and enterprise system/solution architecture in general – not meekly accept them.
IEEE says, “A key premise of this metaphor is that important decisions may be made early in system development in a manner similar to the early decision-making found in the development of civil architecture projects.”
Keep asking yourself, “When is what we built that’s stable actually constraining us too much? When is it preventing important innovation?” For many architects, that’s going to be tough, because you start to love the architecture, the standards, and the discipline. You love what you’ve created, but if it isn’t right for the market you’re facing, you have to be ready to let it go and go seize the next opportunity.
The central message is as follows: ‘documenting’ architecture in various layers of abstraction for the purposes of ‘completeness’ is plainly ridiculous. This is especially true when the effort to produce the artifacts takes such an amount of time as to make the whole collection obsolete on completion.
This got me thinking: What is the biggest bottleneck in the delivery of business value today? I know I look at things from a data perspective, but data is the biggest bottleneck. Consider this prediction from Gartner:
“Gartner predicts organizations will spend one-third more on app integration in 2016 than they did in 2013. What’s more, by 2018, more than half the cost of implementing new large systems will be spent on integration. “
When we talk about application integration, we’re talking about moving data, synchronizing data, cleansing, data, transforming data, testing data. The question for architects and senior management is this: Do you have the Data Foundation for Execution you need to drive the business results you require to compete? The answer, unfortunately, for most companies is; No.
All too often data management is an add-on to larger application-based projects. The result is unconnected and non-interoperable islands of data across the organization. That simply is not going to work in the coming competitive environment. Here are a couple of quick examples:
- Many companies are looking to compete on their use of analytics. That requires collecting, managing, and analyzing data from multiple internal and external sources.
- Many companies are focusing on a better customer experience to drive their business. This again requires data from many internal sources, plus social, mobile and location-based data to be effective.
When I talk to architects about the business risks of not having a shared data architecture, and common tools and practices for enterprise data management, they “get” the problem. So why aren’t they addressing it? The issue is that they find that they are only funded to do the project they are working on and are dealing with very demanding timeframe requirements. They have no funding or mandate to solve the larger enterprise data management problem, which is getting more complex and brittle with each new un-connected project or initiative that is added to the pile.
Studies such as “The Data Directive” by The Economist show that organizations that actively manage their data are more successful. But, if that is the desired future state, how do you get there?
Changing an organization to look at data as the fuel that drives strategy takes hard work and leadership. It also takes a strong enterprise data architecture vision and strategy. For fresh thinking on the subject of building a data foundation for execution, see “Think Data-First to Drive Business Value” from Informatica.
* By the way, Informatica is proud to announce that we are now a sponsor of the MIT Center for Information Systems Research.
Adrian gathered experts and built workgroups to dig into the issue and do root cause analysis. The workgroups came back with some pretty surprising results.
- Most people expected that “incorrect data” (missing, out of date, incomplete, or wrong data) would be the main problem. What they found was that this was only #5 on the list of issues.
- The #1 issue was “Too much data.” People working with the data could not find the data they needed because there was too much data available, and it was hard to figure out which was the data they needed.
- The #2 issue was that people did not know the meaning of data. And because people had different interpretations of the data, the often produced analyses with conflicting results. For example, “claims paid date” might mean the date the claim was approved, the date the check was cut or the date the check cleared. These different interpretations resulted in significantly different numbers.
- In third place was the difficulty in accessing the data. Their environment was a forest of interfaces, access methods and security policies. Some were documented and some not.
In one of the workgroups, a senior manager put the problem in a larger business context;
“Not being able to leverage the data correctly allows competitors to break ground in new areas before we do. Our data in my opinion is the ‘MOST’ important element for our organization.”
What started as a relatively straightforward data quality project became a more comprehensive enterprise data management initiative that could literally change the entire organization. By the project’s end, Adrian found himself leading the data strategy of the organization.
This kind of story is happening with increasing frequency across all industries as all businesses become more digital, the quantity and complexity of data grows, and the opportunities to offer differentiated services based on data grow. We are entering an era of data-fueled organizations where the competitive advantage will go to those who use their data ecosystem better than their competitors.
Gartner is predicting that we are entering an era of increased technology disruption. Organizations that focus on data as their competitive edge will have the advantage. It has become clear that a strong enterprise data architecture is central to the strategy of any industry-leading organization.
For more future-thinking on the subject of enterprise data management and data architecure see Think ‘Data First” to Drive Business Value
This creative thinking to solve a problem came from a request to build a soldier knife from the Swiss Army. In the end, the solution was all about getting the right tool for the right job in the right place. In many cases soldiers didn’t need industrial strength tools, all they really needed was a compact and lightweight tool to get the job at hand done quickly.
Putting this into perspective with today’s world of Data Integration, using enterprise-class data integration tools for the smaller data integration project is over kill and typically out of reach for the smaller organization. However, these smaller data integration projects are just as important as those larger enterprise projects, and they are often the innovation behind a new way of business thinking. The traditional hand-coding approach to addressing the smaller data integration project is not-scalable, not-repeatable and prone to human error, what’s needed is a compact, flexible and powerful off-the-shelf tool.
Thankfully, over a century after the world embraced the Swiss Army Knife, someone at Informatica was paying attention to revolutionary ideas. If you’ve not yet heard the news about the Informatica platform, a version called PowerCenter Express has been released and it is free of charge so you can use it to handle an assortment of what I’d characterize as high complexity / low volume data integration challenges and experience a subset of the Informatica platform for yourself. I’d emphasize that PowerCenter Express doesn’t replace the need for Informatica’s enterprise grade products, but it is ideal for rapid prototyping, profiling data, and developing quick proof of concepts.
PowerCenter Express provides a glimpse of the evolving Informatica platform by integrating four Informatica products into a single, compact tool. There are no database dependencies and the product installs in just under 10 minutes. Much to my own surprise, I use PowerCenter express quite often going about the various aspects of my job with Informatica. I have it installed on my laptop so it travels with me wherever I go. It starts up quickly so it’s ideal for getting a little work done on an airplane.
For example, recently I wanted to explore building some rules for an upcoming proof of concept on a plane ride home so I could claw back some personal time for my weekend. I used PowerCenter Express to profile some data and create a mapping. And this mapping wasn’t something I needed to throw away and recreate in an enterprise version after my flight landed. Vibe, Informatica’s build once / run anywhere metadata driven architecture allows me to export a mapping I create in PowerCenter Express to one of the enterprise versions of Informatica’s products such as PowerCenter, DataQuality or Informatica Cloud.
As I alluded to earlier in this article, being a free offering I honestly didn’t expect too much from PowerCenter Express when I first started exploring it. However, due to my own positive experiences, I now like to think of PowerCenter Express as the Swiss Army Knife of Data Integration.
To start claiming back some of your personal time, get started with the free version of PowerCenter Express, found on the Informatica Marketplace at: https://community.informatica.com/solutions/pcexpress
After I graduated from business school, I started reading Fortune Magazine. I guess that I became a regular reader because each issue largely consists of a set of mini-business cases. And over the years, I have even started to read the witty remarks from the managing editor, Andy Serwer. However, this issue’s comments were even more evocative than usual.
Connectivity is perhaps the biggest opportunity of our time
Andy wrote, “Connectivity is perhaps the biggest opportunity of our time. As technology makes the world smaller, it is clear that the countries and companies that connect the best—either in terms of, say traditional infrastructure or through digital networks are in the drivers’ seat”. Andy sees differentiated connectivity as involving two elements–access and content. This is important to note because Andy believes the biggest winners going forward are going to be the best connectors to each.
Enterprises need to evaluate how the collect, refine, and make useful data
But how do enterprises establish world class connectivity to content? I would argue–whether you are talking about large or small data—it comes from improving an enterprise’s abiity collect, refine, and create useful data. In recent CFO research, the importance of enterprise data gathering capabilities was stressed. CFOs said that their enterprises need to “get data right” at the same time as they confirmed that their enterprises in fact have a data issue. The CFOs said that they are worried about the integrity of data from the source forward. And once they manually create clean data, they worry about making this data useful to their enterprises. Why does this data matter so much to the CFO? Because as CFOs get more strategic, they are trying to make sure their firms drive synergies across their businesses.
Business need to make sense of data and get it to business users faster
One CFO said it this way, “data is potentially the only competitive advantage left”. Yet another said, “our businesses needs to make better decisions from data. We need to make sense of data faster.” At the same time leading edge thinkers like Geoffrey Moore has been suggesting that businesses need to move from “systems of record” applications to “system of engagement” applications. This notion suggests the importance of providing more digestible apps, but also the importance of recognizing that the most important apps for business users will provide relevant information for decision making. Put another way, data is clearly becoming fuel to the enterprise decision making.
“Data Fueled Apps” will provide a connectivity advantage
For this reason, “data fueled” apps will be increasingly important to the business. Decision makers these days want to practice “management by walking around” to quote Tom Peter’s Book, “In Search of Excellence”. And this means having critical, fresh data at their fingertips for each and every meeting. And clearly, organizations that provide this type of data connectivity will establish the connectivity advantage that Serwer suggested in his editor comments. This of course applies to consumer facing apps as well. Server, also, comments on the impacts of Apple and Facebook. Most consumers today are far better informed before they make a purchase. The customer facing apps, for example Amazon, that have led the way have provided the relevant information for the consumer to inform them on their purchase journey.
Delivering “Data Fueled Apps” to the Enterprise
But how do you create the enterprise wide connectivity to power the “Data Fueled Apps?” It is clear from the CFOs comments work is needed here. That work involves creating data which is systematically clean, safe, and connected. Why does this data need to be clean? The CFOs we talked to said that when the data is not clean then they have to manually massage the data and then move from system to system. This is not providing the kind of system of engagement envisioned by Geoffrey Moore. What this CFO wants to move to a world where he can access the numbers easily, timely, and accurately”.
Data, also, needs to be safe. This means that only people with access should be able to see data whether we are talking about transactional or analytical data. This may sound obvious, but very few isolate and secure data as it moves from system to system. And lastly, data needs to be connected. Yet another CFO said, “the integration of the right systems to provide the right information needs to be done so we have the right information to manage and make decisions at the right time”. He continued by saying “we really care about technology integration and getting it less manual. It means that we can inspect the books half way through the cycle. And getting less manual means we can close the books even faster. However, if systems don’t talk (connect) to one another, it is a big issue”.
Finally, whether we are discussing big data or small data, we need to make sure the data collected is more relevant and easier to consume. What is needed here is a data intelligence layer provides easy ways to locate useful data and recommend or guide ways to improve the data. This way analysts and leaders can spend less time on searching or preparing data and more time on analyzing the data to connect the business dots. This can involve mapping data relationship across all applications and being able to draw inferences from data to drive real time responses.
So in this new connected world, we need to first set up a data infrastructure to continuously make data clean, safe, and connected regardless of use case. It might not be needed to collect data, but the data infrastructure may be needed to define the connectivity (in the shape of access and content). We also need to make sure that the infrastructure for doing this is reusable so that the time from concept to new data fueled app is minimized. And then to drive informational meaning, we need to layer on top the intelligence. With this, we can deliver “data fueled apps” that enable business users the access and content to drive better business differentiation and decisioning!