Machine Learning Talent in Short Supply: Opportunity for Some, Crises for Others

Machine LearningMachine learning – a piece of the artificial intelligence constellation – holds a lot of promise for enterprises, enabling programs and algorithms to become ever more intelligent.  However, there’s one problem: even the best-educated humans need more learning before they can understand machine learning.

Bob Hayes, a professional data scientist and keen observer of all things data, picked up on a survey by Kaggle that finds that even data scientists still have a grasp on machine learning. The survey “revealed that a limited number of data professionals possess competency in advanced machine learning skills,” says Hayes. “About half of data professionals said they were competent in supervised machine learning (49%) and logistic regression (53%). Deep learning techniques were among the ML skills with the lowest competency rates.”

The survey covered 16,000 data scientists, so it can be assumed to be statistically valid. The top 10 machine learning areas in which data professionals are competent include the following:

Supervised Machine Learning (49%)

Unsupervised Learning (26%)

Time Series (25%)

Natural Language Processing (19%)

Outlier detection (16%)

Computer Vision (15%)

Recommendation Engines (14%)

Survival Analysis (8%)

Reinforcement Learning (6%)

Adversarial Learning (4%)

So, is this a problem? AI and machine learning are seen as the most powerful tools to date to boost productivity and innovation. But if skills in these areas continue to lag demand, such efforts may be slowed, or, worse yet, misdirected. Hayes cites statistics that show a majority of enterprises (80%) have some form of artificial intelligence (machine learning, deep learning) in production today. “About a third of enterprises are planning on expanding their AI efforts over the next 36 months. But who will lead these data science projects? Who will do the work?” he asks.

At the same time, it may be an opportunity for those seeking to rise in the data science and analytics field to develop skills in these areas.

Basic analytic skills. These include supervised machine learning, logistic regression and decision trees, Hayes says. “These types of skills apply to a wide variety of data science problems that focus on identifying predictors of important organizational outcomes.”

Software development: “The days of one team writing throwaway models and another team implementing them in production are slowly coming to an end,” according to Quora’s Vladimir Novakovski. “With programming languages like Python and R and their packages making it easy to work with data and models, it is reasonable to expect a data scientist or machine learning engineer to attain a high level of programming proficiency and understand the basics of system design.”

Ability to work with large data sets. “The cost of data storage is on a dramatic downward trend,” says Novakovski. “This means that there are more and more data sets from different domains to work with and apply models to.”

Understand related technologies. Both Hayes and Novakovski advocate developing knowledge about at least one of the popular areas of the field that have gotten traction lately — deep learning for computer vision and perception, recommendation engines, and natural language processing.

There are many other machine learning areas “in which there is a paucity of talent,” says Hayes. “A majority of data professionals lack competency in many advanced machine learning areas and techniques like neural networks, evolutionary techniques, reinforcement learning and adversarial learning.”