Tag Archives: Healthcare
The U.S. Department of Health and Human Services declares that approximately 75 percent of the country’s eligible professionals and more than 91 percent of hospitals are on electronic health records certified for Stage 1 meaningful use. These applications along with many others are creating valuable electronic data that – when integrated, shared and analyzed – can propel transformational initiatives like accountable care, population health and risk based contracting.
Success should not be limited by technology given other industries have demonstrated that near real time sharing and analysis of sensitive data is quite possible. What, then, could hold healthcare back from success?
Informatica recently released the findings of a survey targeted at understanding the answer to this very question. Respondents revealed that while 85% of are effective at putting financial data to use to inform decision making, many less are confident about putting data to use to inform patient engagement initiatives requiring access to external data and big data which they note to be more challenging. Complex, cross organizational transformational business processes require information sharing between applications yet over 65% of respondents say data integration and data quality are significantly challenging. So healthcare organizations are collecting data but many have yet to integrate these silos of application data to realize its full potential.
Similarly, International Institute of Analytics, a little over a year ago, offered a view of the healthcare analytics maturity landscape based upon deep quantitative assessments of more than 20 healthcare provider organizations. This research uncovered two important facts:
- Hospitals and other healthcare provider organizations have gone to great effort to implement core components of an EMR, giving them access to large amounts of data on patients, processes and costs.
- Those data assets have not yet been put to their highest and best use.
This is not uncommon, across industries and firms of all shapes and sizes, many have confronted the question of how well they are leveraging data and analytics to transform their organizations. Those organizations that have made the most progress in revealing meaningful and differentiated insights are those that have intentionally built and funded enterprise information management or data management programs in support of analytics. These programs accelerate stakeholder access to trusted information when and where they need it.
Informatica worked with International Institute of Analytics to publish a new whitepaper that explores this issue, with detailed definitions for how IIA measures analytics maturity within healthcare as an independent third-party. This report looks at how provider organizations are approaching their investments in analytics, with a focus on essential attributes like data quality and management, leadership support, the culture of data, analytics talent, and an enterprise-wide approach to analytics. If you haven’t read it yet, I urge you to read the report.
If you’d like to talk about these ideas, visit Informatica at HIMSS15 in Booth #3056.
With the European Medicines Agency (EMA) date for compliance to IDMP (Identification of Medicinal Products) looming, Q1 2015 has seen a significant increase in IDMP activity. Both Informatica & HighPoint Solution’s IDMP Round Table in January, and a February Marcus Evans conference in Berlin provided excellent forums for sharing progress, thoughts and strategies. Additional confidential conversations with pharmaceutical companies show an increase in the number of approved and active projects, although some are still seeking full funding. The following paragraphs sum up the activity and trends that I have witnessed in the first three months of the year.
I’ll start with my favourite quote, which is from Dr. Jörg Stüben of Boehringer Ingelheim, who asked:
“Isn’t part of compliance being in control of your data?”
I like it because to me it is just the right balance of stating the obvious, and questioning the way the majority of pharmaceutical companies approach compliance: A report that has to be created and submitted. If a company is in control of their data, regulatory compliance would be easier and come at a lower cost. More importantly, the company itself would benefit from easy access to high quality data.
Dr. Stüben’s question was raised during his excellent presentation at the Marcus Evans conference. Not only did he question the status quo, but proposed an alternate way for IDMP compliance: Let Boehringer benefit from their investment in IDMP compliance. His approach can be summarised as follows:
- Embrace a holistic approach to being in control of data, i.e. adopt data governance practices.
- This is not about just compliance. Include optional attributes that will deliver value to the organisation if correctly managed.
- Get started by creating simple, clear work packages.
Although Dr Stüben did not outline his technical solution, it would include data quality tools and a product data hub.
At the same conference, Stefan Fischer Rivera & Stefan Brügger of Bayer and Guido Claes from Janssen Pharmaceuticals both came out strongly in favour of using a Master Data Management (MDM) approach to achieving compliance. Both companies have MDM technology and processes within their organisations, and realise the value a MDM approach can bring to achieving compliance in terms of data management and governance. Having Mr Claes express how well Informatica’s MDM and Data Quality solutions support his existing substance data management program, made his presentation even more enjoyable to me.
Whilst the exact approaches of Bayer and Janssen differed, there were some common themes:
- Consider both the short term (compliance) and the long term (data governance) in the strategy
- Centralised MDM is ideal, but a federated approach is practical for July 2016
- High quality data should be available to a wide audience outside of IDMP compliance
The first and third bullet points map very closely to Dr. Stüben’s key points, and in fact show a clear trend in 2015:
IDMP Compliance is an opportunity to invest in your data management solutions and processes for the benefit of the entire organisation.
Although the EMA was not represented at the conference, Andrew Marr presented their approach to IDMP, and master data in general. The EMA is undergoing a system re-organisation to focus on managing Substance, Product, Organisation and Reference data centrally, rather than within each regulation or program as it is today. MDM will play a key role in managing this data, setting a high standard of data control and management for regulatory purposes. It appears that the EMA is also using IDMP to introduce better data management practice.
Depending on the size of the company, and the skills & tools available, other non-MDM approaches have been presented or discussed during the first part of 2015. These include using XML and SharePoint to manage product data. However I share a primary concern with others in the industry with this approach: How well can you manage and control change using these tools? Some pharmaceutical companies have openly stated that data contributors often spend more time looking for data than doing their own jobs. A XML/SharePoint approach will do little to ease this burden, but an MDM approach will.
Despite the others approaches and solutions being discovered, there is another clear trend in Q1 2015
MDM is becoming a favoured approach for IDMP compliance due to its strong governance, centralised attribute-level data management and ability to track changes.
Interestingly, the opportunity to invest in data management, and the rise of MDM as a favoured approach has been backed up with research by Gens Associates. Messers Gens and Brolund found a rapid increase in investment during 2014 of what they term Information Architecture, in which MDM plays a key role. IDMP is seen as a major driver for this investment. They go on to state that investment in master data management programs will allow a much easier and cost effective approach to data exchange (internally and externally), resulting in substantial benefits. Unfortunately they do not elaborate on these benefits, but I have placed a summary on benefits of using MDM for IDMP compliance here.
In terms of active projects, the common compliance activities I have seen in the first quarter of 2015 are as follows:
- Most companies are in the discovery phase: identifying the effort for compliance
- Some are starting to make technology choices, and have submitted RFPs/RFQs
- Those furthest along in technology already have MDM programs or initiatives underway
- Despite getting a start, some are still lacking enough funding for achieving compliance
- Output from the discovery phase will in some cases be used to request full funding
- A significant number of projects have a goal to implement better data management practice throughout the company. IDMP will be the as the first release.
A final trend I have noticed in 2015 is regarding the magnitude of the compliance task ahead:
Those who have made the most progress are those who are most concerned about achieving compliance on time.
The implication is that the companies who are starting late do not yet realise the magnitude of the task ahead. It is not yet too late to comply and achieve long term benefits through better data management, despite only 15 months before the initial EMA deadline. Informatica has customers who have implemented MDM within 6 months. 15 months is achievable provided the project (or program) gets the focus and resources required.
IDMP compliance is a common challenge to all those in the pharmaceutical industry. Learning from others will help avoid common mistakes and provide tips on important topics. For example, how to secure funding and support from senior management is a common concern among those tasked with compliance. In order to encourage learning and networking, Informatica and HighPoint Solutions will be hosting our third IDMP roundtable in London on May 13th. Please do join us to share your experiences, and learn from the experiences of others.
As I have shared within the posts of this series, businesses are using analytics to improve their internal and external facing business processes and to strengthen their “right to win” within the markets that they operate. Like healthcare institutions across the country, UPMC is striving to improve its quality of care and business profitability. One educational healthcare CEO put it to me this way–“if we can improve our quality of service, we can reduce costs while we increase our pricing power”. In UPMC’s case, they believe that the vast majority of their costs are in a fraction of their patients, but they want to prove this with real data and then use this information drive their go forward business strategies.
Getting more predictive to improved outcomes and reduce cost
Armed with this knowledge, UPMC’s leadership wanted to use advanced analytic and predictive modeling to improve clinical and financial decision making. And taking this action was seen as producing better patient outcomes and reducing costs. A focus area for analysis involved creating “longitudinal records” for the complete cost of providing particular types of care. For those that aren’t versed in time series analysis, longitudinal analysis uses a series of observations obtained from many respondents over time to derive a relevant business insight. When I was also involved in healthcare, I used this type of analysis to interrelate employee and patient engagement results versus healthcare outcomes. In UPMC’s case, they wanted to use this type of analysis to understand for example the total end to end cost of a spinal surgery. UPMC wanted to look beyond the cost of surgery and account for the pre-surgery care and recovery-related costs. However, to do this for the entire hospital meant that it needed to bring together data from hundreds of sources across UPMC and outside entities, including labs and pharmacies. However, by having this information, UPMC’s leadership saw the potential to create an accurate and comprehensive view which could be used to benchmark future procedures. Additionally, UPMC saw the potential to automate the creation of patient problem lists or examine clinical practice variations. But like the other case studies that we have reviewed, these steps required trustworthy and authoritative data to be accessed with agility and ease.
UPMC’s starts with a large, multiyear investment
In October 2012, UPMC made a $100 million investment to establish an enterprise analytics initiative to bring together for the first time, clinical, financial, administrative, genomic and other information together in one place. Tom Davenport, the author of Competing on Analytics, suggests in his writing that establishing an enterprise analytics capability represents a major step forward because it allows enterprises to answer the big questions, to better tie strategy and analytics, and to finally rationalize applications interconnect and business intelligence spending. As UPMC put its plan together, it realized that it needed to impact more than 1200 applications. As well it realized that it needed one system manage with data integration, master data management, and eventually complex event processing capabilities. At the same time, it created the people side of things by creating a governance team to manage data integrity improvements, ensuring that trusted data populates enterprise analytics and provides transparency into data integrity challenges. One of UPMC’s goals was to provide self-service capabilities. According to Terri Mikol, a project leader, “We can’t have people coming to IT for every information request. We’re never going to cure cancer that way.” Here is an example of the promise that occurred within the first eight months of this project. Researchers were able to integrate—for the first time ever– clinical and genomic information on 140 patients previously treated for breast cancer. Traditionally, these data have resided in separate information systems, making it difficult—if not impossible—to integrate and analyze dozens of variables. The researchers found intriguing molecular differences in the makeup of pre-menopausal vs. post-menopausal breast cancer, findings which will be further explored. For UPMC, this initial cancer insight is just the starting point of their efforts to mine massive amounts of data in the pursuit of smarter medicines.
Building the UPMC Enterprise Analytics Capability
To create their enterprise analytics platform, UPMC determined it was critical to establish “a single, unified platform for data integration, data governance, and master data management,” according to Terri Mikol. The solution required a number of key building blocks. The first was data integration to collect and cleanses data from hundreds of sources and organizes them into repositories that would enable fast, easy analysis and reporting by and for end users.
Specifically, the UPMC enterprise analytics capability pulls clinical and operational data from a broad range of sources, including systems for managing hospital admissions, emergency room operations, patient claims, health plans, electronic health records, as well as external databases that hold registries of genomic and epidemiological data needed for crafting personalized and translational medicine therapies. UPMC has integrated quality checked source data in accordance with industry-standard healthcare information models. This effort included putting together capabilities around data integration, data quality and master data management to manage transformations and enforce consistent definitions of patients, providers, facilities and medical terminology.
As said, the cleansed and harmonized data is organized into specialized genomics databases, multidimensional warehouses, and data marts. The approach makes use of traditional data warehousing approaches as well as big data capabilities to handle unstructured data and natural language processing. UPMC has also deployed analytical tools that allow end users to exploit the data enabled from the Enterprise Analytics platform. The tools drive everything from predictive analytics, cohort tracking, and business and compliance reporting. And UPMC did not stop here. If their data had value then it needed to be secured. UPMC created data audits and data governance practices. As well, they implemented a dynamic data masking solution ensures data security and privacy.
As I have discussed, many firms are pushing point silo solutions into their environments, but as UPMC shows this limits their ability to ask the bigger business questions or in UPMC’s case to discover things that can change people’s live. Analytics are more and more a business enabler if they are organized as an enterprise analytics capability. As well, I have come to believe that analytics have become foundational capability to all firms’ right to win. It informs a coherent set of capabilities and establishes a firm’s go forward right to win. For this, UPMC is a shining example of getting things right.
Author Twitter: @MylesSuer
Healthcare and data have the makings of an epic love affair, but like most relationships, it’s not all roses. Data is playing a powerful role in finding a cure for cancer, informing cost reduction, targeting preventative treatments and engaging healthcare consumers in their own care. The downside? Data is needy. It requires investments in connectedness, cleanliness and safety to maximize its potential.
- Data is ubiquitous…connect it.
4400 times the amount of information held at the Library of Congress – that’s how much data Kaiser Permanente alone has generated from its electronic medical record. Kaiser successfully makes every piece of information about each patient available to clinicians, including patient health history, diagnosis by other providers, lab results and prescriptions. As a result, Kaiser has seen marked improvements in outcomes: 26% reduction in office visits per member and a 57% reduction in medication errors.
Ongoing value, however, requires continuous investment in data. Investments in data integration and data quality ensure that information from the EMR is integrated with other sources (think claims, social, billing, supply chain) so that clinicians and decision makers have access in the format they need. Without this, self-service intelligence can be inhibited by duplicate data, poor quality data or application silos.
- Data is popular…ensure it is clean.
Healthcare leaders can finally rely on electronic data to make strategic decisions. A CHRISTUS Health anecdote you might relate to – In a weekly meeting each executive reviews their strategic dashboard; these dashboards drive strategic decision making about CPOE adoption (computerized physician order entry), emergency room wait times and price per procedure. Powered by enterprise information management, these dashboards paint a reliable and consistent view across the system’s 60 hospitals. Previous to the implementation of an enterprise data platform, each executive was reliant on their own set of data.
In the pre-data investment era, seemingly common data elements from different sources did not mean the same thing. For example, “Admit Date” in one report reflected the emergency department admission date whereas “Admit Date” in another report referred to the inpatient admission date.
- Sharing data is necessary…make it safe.
To cure cancer, reduce costs and engage patients, care providers need access to data and not just the data they generate; it has to be shared for coordination of care through transitions of care and across settings, i.e. home care, long term care and behavioral health. Fortunately, Consumers and Clinicians agree on this, PWC reports that 56% of consumers and 30% of physicians are comfortable with data sharing for care coordination. Further progress is demonstrated by healthcare organizations willingly adopting cloud based applications –as of 2013, 40% of healthcare organizations were already storing protected health information (PHI) in the cloud.
Increased data access carries risk, leaving health data exposed, however. The threat of data breach or hacking is multiplied by the presence (in many cases necessary) of PHI on employee laptops and the fact that providers are provided increased access to PHI. Ponemon Institute, a security firm estimates that data breaches cost the industry $5.6 billion each year. Investments in data-centric security are necessary to assuage fear, protect personal health data and make secure data sharing a reality.
Early improvements in patient outcomes indicate that the relationship between data and healthcare is a valuable investment. The International Institute of Analytics supports this, reporting that although analytics and data maturity across healthcare lags other industries, the opportunity to positively impact clinical and operational outcomes is significant.
The signs that healthcare is becoming a more consumer (think patients, members, providers) driven industry are evident all around us. I see provider and payer organizations clamoring for more data, specifically data that is actionable, relatable and has integrity. Armed with this data, healthcare organizations are able to differentiate around a member/patient-centric view.
These consumer-centric views convey the total value of the relationships healthcare organizations have with consumers. Understanding the total value creates a more comprehensive understanding of consumers because they deliver a complete picture of an individual’s critical relationships including: patient to primary care provider, member to household, provider to network and even members to legacy plans. This is the type of knowledge that informs new treatments, targets preventative care programs and improves outcomes.
Payer organizations are collecting and analyzing data to identify opportunities for more informed care management and segmentation to reach new, high value customers in individual markets. By segmenting and targeting messaging to specific populations, health plans generate increased member satisfaction and cost effectively expands and manages provider networks.
How will they accomplish this? Enabling members to interact in health and wellness forums, analyzing member behavior and trends and informing care management programs with a 360 view of members… to name a few . Payers will also drive new member programs, member retention and member engagement marketing and sales programs by investigating complete views of member households and market segments.
In the provider space, this relationship building can be a little more challenging because often consumers as patients do not interact with their doctor unless they are sick, creating gaps in data. When provider organizations have a better understanding of their patients and providers, they can increase patient satisfaction and proactively offer preventative care to the sickest (and most likely to engage) of patients before an episode occurs. These activities result in increased market share and improved outcomes.
Where can providers start? By creating a 360 view of the patient, organizations can now improve care coordination, open new patient service centers and develop patient engagement programs.
Analyzing populations of patients, and fostering patient engagement based on Meaningful Use requirements or Accountable Care requirements, building out referral networks and developing physician relationships are essential ingredients in consumer engagement. Knowing your patients and providing a better patient experience than your competition will differentiate provider organizations.
You may say “This all sounds great, but how does it work?” An essential ingredient is clean, safe and connected data. Clean, safe and connected data requires an investment in data as an asset… just like you invest in real estate and human capital, you must invest in the accessibility and quality of your data. To be successful, arm your team with tools to govern data –ensuring ongoing integrity and quality of data, removes duplicate records and dynamically incorporates data validation/quality rules. These tools include master data management, data quality, metadata management and are focused on information quality. Tools focused on information quality support a total customer relationship view of members, patients and providers.
This week, another reputable organization, Anthem Inc, reported it was ‘the target of a very sophisticated external cyber attack’. But rather than be upset at Anthem, I respect their responsible data breach reporting.
In this post from Joseph R. Swedish, President and CEO, Anthem, Inc., does something that I believe all CEO’s should do in this situation. He is straight up about what happened, what information was breached, actions they took to plug the security hole, and services available to those impacted.
When it comes to a data breach, the worst thing you can do is ignore it or hope it will go away. This was not the case with Anthem. Mr Swedish did the right thing and I appreciate it.
You only have one corporate reputation – and it is typically aligned with the CEO’s reputation. When the CEO talks about the details of a data breach and empathizes with those impacted, he establishes a dialogue based on transparency and accountability.
Research that tells us 44% of healthcare and pharmaceutical organizations experienced a breach in 2014. And we know that when personal information when combined with health information is worth more on the black market because the data can be used for insurance fraud. I expect more healthcare providers will be on the defensive this year and only hope that they follow Mr Swedish’s example when facing the music.
Patient experience is key to growth and success for all health delivery organizations. Gartner has stated that the patient experience needs to be one of the highest priorities for organizations. The quality of your data is critical to achieving that goal. My recent experience with my physician’s office demonstrates how easy it is for the quality of data to influence the patient experience and undermine a patient’s trust in their physician and the organization with which they are interacting.
I have a great relationship with my doctor and have always been impressed by the efficiency of the office. I never wait beyond my appointment time, the care is excellent and the staff is friendly and professional. There is an online tool that allows me to see my records, send messages to my doctor, request an appointment and get test results. The organization enjoys the highest reputation for clinical quality. Pretty much perfect from my perspective – until now.
I needed to change a scheduled appointment due to a business conflict. Since I expected some negotiation I decided to make the phone call rather than request it on line…there are still transactions for which human to human is optimal! I had all my information at hand and made the call. The phone was pleasantly answered and the request given. The receptionist requested my name and date of birth, but then stated that I did not have a future appointment. I am looking at the online tool, which clearly states that I am scheduled for February 17 at 8:30 AM. The pleasant young woman confirms my name, date of birth and address and then tells me that I do not have an appointment scheduled. I am reasonably savvy about these things and figured out the core problem, which is that my last name is hyphenated. Armed with that information, my other record is found and a new appointment scheduled. The transaction is completed.
But now I am worried. My name has been like this for many years and none of my other key data has changed. Are there parts of my clinical history missing in the record that my doctor is using? Will that have a negative impact on the quality of my care? If I were to be unable to clearly respond, might that older record be accessed and my current medications and history not be available? The receptionist did not address the duplicate issue clearly by telling me that she would attend to merging the records, so I have no reason to believe that she will. My confidence is now shaken and I am less trustful of the system and how well it will serve me going forward. I have resolved my issue, but not everyone would be able to push back to insure that their records are now accurate.
Many millions of dollars are being spent on electronic health records. Many more millions are being spent to redesign work flow to accommodate the new EHR’s. Physicians and other clinicians are learning new ways to access data and treat their patients. The foundation for all of this is accurate data. Nicely displayed but inaccurate data will not result in improved care or enhanced member experience. As healthcare organizations move forward with the razzle dazzle of new systems they need to remember the basics of good quality data and insure that it is available to these new applications.
CMS points out the overall improvement in quality which they position as the result of focusing, and incenting quality. I agree that putting funding behind a quality program was a valuable strategy to motivate the industry. This has not always been the case, in fact a former colleague who related a common dialog previous to this program:
- He would present a quality initiative to executive management
- They would nod politely and say, “Yes, of course we are interested in quality!”
- The conversation would continue until the cost of the program was disclosed.
The faces would change, and the response was, “Well, yes, quality is important, but funding is tight right now. We need to focus on programs with a clear ROI”.
Thankfully the Star program has given quality initiatives a clear ROI – for which we are all grateful!
The other dynamic which is positive is that Medicare Advantage has provided a testing ground for new programs, largely the result of ACA. Programs very similar to the Star program are part of the ACO program and the marketplace membership. Risk Adjustment is being fitted to meet these programs also. Private insurance will likely borrow similar structures to insure quality and fair compensation in their various risk sharing arrangements. MA is a significant subset of the population and is providing an excellent sandbox for these initiatives while improving the quality of care that our senior population receives.
My concerns are around the cultures and mission of those plans who are struggling to get to the magic four star level where they will receive the bonus dollars.
Having worked in a health plan for almost nine years, and continuing to interact with my current customers, has shown me the dedication of the staffs that work in these plans. One of my most rewarding experiences was leading the call center for the Medicare population. I was humbled each day by the caring and patience the reps on the phones showed to the senior population. I have also seen the dedication of clinical staffs to insuring the care for members is carefully coordinated and that their dignity and wishes were always respected. I sincerely hope that plans with a clear mission find the right tools and supports to improve their ratings to the point where they receive the additional funding to maintain their viability and continue to serve their members and the medical community. I am sure that there are poor quality plans out there, and I agree that they should be eliminated. But I am also rooting for the plans with a mission who are striving to be a bit better.
Saeed, what does Decision Point do?
We are a healthcare engagement analytics company…essentially we help clients that are “at risk” organizations to improve performance, including STAR ratings. We do this by providing data driven insights to more effectively engage members and providers.
What type of data do you use to make these recommendations?
Well, taking better care of members is about emotionally involving them in their care. Information to help do this resides in data that plans already have available, i.e. utilization patterns, distance to doctors, if they are compliant with evidence based guidelines, do they call into the call center. We also seek to include information about their behavior as a consumer. such as their lifestyles, their access to technology, and so forth.
Claims data makes sense, everyone has that but the other data you mentioned, that can be harder to capture. Why does non-claims oriented data matter?
We develop predictive models that are unique for each client – specifically based on the demographics and variables of their population. Variables like exercise and technology access matter because — for example, exercise habits influence mood and access to technology demonstrates a way to contact them or invite them to participate in online communities with other members like themselves.
The predictive models then determine which members are at most risk?
Yes, yes they do but they can also determine a member’s barriers to desired behavior, and their likelihood of responding to and acting on health plan communications. For example, if we identified a diabetic member as high risk of non-compliance, found their primary barrier to compliance as health literacy, and determined that the member will likely respond positively to a combination of health coaching and mobile health initiatives, we would recommend outreach that directly addresses these findings..
Noreen, when you were working on the payer side of the house, how were you going about determining which members were in your at risk population?
We had teams of people doing mining of claims data and we were asking members to complete surveys. This made for more data but the sheer volume of data made it complex to accurately review and assess which members were at highest risk. It was very challenging to take into consideration all of the variables that impact each member. Taking data from so many disparate sources and bringing it together is a big challenge.
What made it (and continues to make it) it so challenging, specifically to STARS?
So much of the data is collected as surveys or in other non-standard formats. Members inherently are unique which creates a lot of variability and it is often difficult to interpret the relationships that exist between members and primary care physicians, specialists, facilities and the rest of their care team. Relationships are important because they can provide insights into utilization patterns, potential overlaps or gaps in care and how we can most effectively engage those members in their care.
What are Informatica and Decision Point doing together?
To optimize the predictive models, as Saeed described, it’s imperative to feed them as much data and as accurate of data as possible. Without data, insights will be missed… and insights are the path to discovery and to improving CMS STARS ratings. Informatica is the data integration company — we ensure that data is reliable), connected (from any source to any target) and safe (avoiding data breaches or HIPAA violations). Informatica is delivering data to Decision Point efficiently and effectively so that clients have access to the best data possible to derive insights and improve outcomes. Our technology also provided the Star team with a member profile which brings together that disparate data and organizes it into the 360 degree view of that member. In addition to fueling Decision Point’s powerful algorithms, this is a tool that can be used for ongoing insights into the members.
Excellent, how can readers learn more?
In my last blog, I talked about the dreadful experience of cleaning raw data by hand as a former analyst a few years back. Well, the truth is, I was not alone. At a recent data mining Meetup event in San Francisco bay area, I asked a few analysts: “How much time do you spend on cleaning your data at work?” “More than 80% of my time” and “most my days” said the analysts, and “they are not fun”.
But check this out: There are over a dozen Meetup groups focused on data science and data mining here in the bay area I live. Those groups put on events multiple times a month, with topics often around hot, emerging technologies such as machine learning, graph analysis, real-time analytics, new algorithm on analyzing social media data, and of course, anything Big Data. Cools BI tools, new programming models and algorithms for better analysis are a big draw to data practitioners these days.
That got me thinking… if what analysts said to me is true, i.e., they spent 80% of their time on data prepping and 1/4 of that time analyzing the data and visualizing the results, which BTW, “is actually fun”, quoting a data analyst, then why are they drawn to the events focused on discussing the tools that can only help them 20% of the time? Why wouldn’t they want to explore technologies that can help address the dreadful 80% of the data scrubbing task they complain about?
Having been there myself, I thought perhaps a little self-reflection would help answer the question.
As a student of math, I love data and am fascinated about good stories I can discover from them. My two-year math program in graduate school was primarily focused on learning how to build fabulous math models to simulate the real events, and use those formula to predict the future, or look for meaningful patterns.
I used BI and statistical analysis tools while at school, and continued to use them at work after I graduated. Those software were great in that they helped me get to the results and see what’s in my data, and I can develop conclusions and make recommendations based on those insights for my clients. Without BI and visualization tools, I would not have delivered any results.
That was fun and glamorous part of my job as an analyst, but when I was not creating nice charts and presentations to tell the stories in my data, I was spending time, great amount of time, sometimes up to the wee hours cleaning and verifying my data, I was convinced that was part of my job and I just had to suck it up.
It was only a few months ago that I stumbled upon data quality software – it happened when I joined Informatica. At first I thought they were talking to the wrong person when they started pitching me data quality solutions.
Turns out, the concept of data quality automation is a highly relevant and extremely intuitive subject to me, and for anyone who is dealing with data on the regular basis. Data quality software offers an automated process for data cleansing and is much faster and delivers more accurate results than manual process. To put that in math context, if a data quality tool can reduce the data cleansing effort from 80% to 40% (btw, this is hardly a random number, some of our customers have reported much better results), that means analysts can now free up 40% of their time from scrubbing data, and use that times to do the things they like – playing with data in BI tools, building new models or running more scenarios, producing different views of the data and discovering things they may not be able to before, and do all of that with clean, trusted data. No more bored to death experience, what they are left with are improved productivity, more accurate and consistent results, compelling stories about data, and most important, they can focus on doing the things they like! Not too shabby right?
I am excited about trying out the data quality tools we have here at Informtica, my fellow analysts, you should start looking into them also. And I will check back in soon with more stories to share..