Good luck finding a really good data scientist these days.
McKinsey reports that the United States faces a shortage of 140,000 to 190,000 data scientists, plus a need for 1.5 million managers and analysts who can work with these data scientists. Droves of professionals are studying for data science certification and the lucrative salary it can bring them. But there is a lag for this talent and even that certification does not ensure the kind of data scientist that businesses need.
The best are even harder to come by. A data scientist isn’t just a collection of skills posted at the top of a resume – today, a really good data scientist has business acumen, technical know-how, an ability to lead and teach others, and sometimes the political chops to navigate corporate bureaucracies and sell across divisions.
The good news is that many businesses are finding that they might not need a data scientist to own their big data cluster as technical business analysts are filling in the gaps with quiet aplomb. Technology is catching up to the everyday data analyst and simplifying tasks that only a few years ago required a delicate combination of data modeling, scripting, statistical analysis, and communication skills.
Your new data scientist may very well be that technically savvy business analyst who is currently juggling three projects and attending scrums even though she doesn’t have to, and who knows more about your business software applications than the programmers who are writing them. She may not only know the business, she may know the data and how it is collected, understand the business problems that are being asked, and hopefully can translate difficult business questions into clear, technical requirements. She probably knows SAS or R, and can answer questions with the data and SQL to back it up. This is an all-star data scientist, and she provides substantial value.
Businesses that can’t find data science talent – or don’t want to pay top dollar for it – need to learn to trust and enable their technical business analysts. Give them the data first, trust them to understand it, facilitate their ability to do things with the data using their own creativity and imagination, and make sure they have the tools they need.
What hasn’t changed is that business analysts are the intermediaries between the business and IT. We have seen many smart business analysts who understand the business and how the data comes in, including all the nuances, obscure business rules, and “oh-by-the-ways” that seem to come with traditional data analysis. The best ones aren’t afraid to teach themselves new skills and they know how to manage the rigor and processes that come with internal IT controls. These are skills to consider when picking a business analyst to work on your next product launch or other initiative and help you wring value from your data assets.
Due in large part to Hadoop, organizations are re-defining the way that data is shared. Even in the context of Sarbanes-Oxley, HIPAA, and other regulations, companies are finding ways to disentangle personally identifiable information and other sensitive pieces from their data sets and land the data quickly into sandboxes so that business analysts can play with it and find patterns.
Advances by business intelligence vendors are empowering technical BAs to do their own data discovery and blend it with the traditionally curated business intelligence sources. These analysts might even know how to call an API and parse a JSON file using Python or grab an R package from GitHub to profile data.
For complex data sources, new tools allow business analysts to do their own self-service transformations and then introduce it to the data analysis stack. This inexorable push to make data blending easy, once the heart and soul of traditional ETL, is ending up at the fingertips of savvy BAs and falling out of the purview of IT. Before these advances, it was up to the data scientist to clean and prepare the data. Together, these technology advancements sound the death knell for Excel’s VLOOKUP function, long the only feasible path for the DIY datasets blenders.
Technology is converging to enable the technical business analyst. From advancing simple data availability, to the cleansing / prepping / blending stage, then to the discovery and mining of data assets, business analysts now have the tools they need to become as functionally effective as data scientists. No stodgy mainframe is needed to do all that “stuff” to the data.
Technical business analysts often know the business inside and out, so ask them what tools they want to use and trust them to learn it quickly, with little hand-holding. They may not all have doctorates in mathematics or statistics, but smart analysts can offer organizations access to the fundamentals of the data science discipline through the breadth of available open source and commercial software systems. Successful organizations are doing this today, and technically adventurous business analysts are leading the charge.
Chael Christopher is Senior Principal and Practice Lead, Business Intelligence for NewVantage Partners, a world-class provider of data management and analytics-driven strategic consulting services to Fortune 1000 firms, and the industry leader in Big Data strategy consulting, thought-leadership, execution, and business value realization.
Subscribe to Data Informed for the latest information and news on big data and analytics for the enterprise.