Chances are, if you’re staffing up now to meet the challenge of big data, you’re going to end up stealing talent from your competitors. But that will get you only so far.
Demand for all manner of data experts is already high: tech recruiting website Dice.com lists 32,000 openings for data professionals, including analysts, modelers, architects and data scientists—an emerging role that combines engineering skills, subject matter expertise and a background in mathematics or statistics. An oft-cited report by the McKinsey Global Institute estimates that by 2018, demand for employees with deep analytical expertise will outpace supply by up to 190,000 openings.
If you want your team to deliver value from massive data sets, they need to look beyond known data, well-defined questions and traditional business intelligence techniques, observes Douglas Laney, vice president for research, business analytics and performance management with Gartner: “That’s where the real win is. People who can do that are in very, very short supply.” Which means that, for now, you have to build that team yourself. Training staff to use big data technologies is the easy part.
Lots of Options for IT Training
By now, most IT pros know that they’ll have to keep learning new skills to advance their careers. In this respect, the transition to big data isn’t that different than any other major technology shift. “Closing the gap on the technology skills is more of a vocational exercise, and that will happen fairly quickly,” says Laney.
IT leaders have numerous options for training staff to use big data tools. Academic institutions and professional organizations offer classes and certifications, as do vendors.
For example, IBM launched its Big Data University in October. Since then, more than 18,000 students have enrolled in its online courses. Many are statisticians, mathematicians, computer scientists and engineers in addition to IT professionals.
“We’re giving the data scientists a lot of toolkits,” says Anjul Bhambhri the company’s vice president of big data, and they need to understand how the tools work in order to decide whether to “pick a hammer or a spanner” for the problem they want to solve. In addition to online courses, the company offers in-person classes, on-site training for customers and partnerships with more than 200 universities.
Technologists might also opt to train themselves. The online advertising and marketing company Epic Media Group deployed Vertica data warehousing technology as its platform for collecting and analyzing data from 300 million-plus ad impressions per day. As CIO, Rick Okin (who now heads IT for an Epic Media spinoff, Kinetic-Social) sent his systems administrators, database administrators and developers for vendor training. But when it came time to learn Hadoop, they just dove in. “That’s just part of their DNA,” Okin says. “I work in open source, thus I have to learn by getting my hands dirty and scouring the web.”
No Clear Path for Data Scientists
Developing data scientists is less straightforward, not least because it’s an emerging role without a well-defined education or career path. A recent study led by Barbara Wixom, associate professor at the University of Virginia’s McIntire School of Commerce concluded that neither business schools or computer science and engineering programs currently teach the whole package of quantitative skills, functional business knowledge, data management expertise and communications skills that companies want—even for positions with less lofty titles.
John Rauser, principal quantitative engineer with Amazon, began his career as a software engineer, then taught himself statistics and data visualization. If you’re looking for data scientists, he told an audience at the Strata New York 2011 conference last fall, “it might be easier to find a promising software engineer or statistician and give them problems that will allow them to grow into the role,” he said.
But again, it’s easier to learn how to use big data technology than acquire many of the “unique skills” asked of data scientists, says Laney. Gartner analyzed hundreds of job listings to determine what hiring managers wanted from the position. “Some things really jumped out at us: communications, teamwork, the ability to prepare data, machine learning, a passion for the truth in data.”
IBM’s Bhambhri says the industry as a whole—vendors, academia and end user organizations—needs to focus more on creating well-rounded professionals. At universities, having a variety of courses that teach students how to understand and work with big data could lead to a new field of study.
The Data Scientist Skill Set
A recent Gartner report found that few individuals have all the necessary expertise in three key areas required to be a data scientist:
Data management experience: knowing how to integrate and manipulate data, as well as being able to validate it and prep it for analysis.
Analytics modeling knowledge: being familiar with the data and able to determine the best techniques for analyzing and interpreting it.
Business analysis: being able to frame problems and develop hypotheses that are relevant to business decisions, and to communicate insights from analysis.
Source: Gartner: Emerging Role of the Data Scientist and the Art of Data Science, March 20, 2012.
Savvy Managers Wanted
Meanwhile, functional leaders have to know enough about big data to make sure they’re working on the right business problems. As a business school professor, Wixom says she’s less concerned with turning out data scientists right now than educating business-savvy analysts and data-savvy managers. McKinsey Global Institute projects a shortfall of 1.5 million such workers.
At McIntire, Wixom is co-director, with a marketing professor, of an initiative to incorporate analytics training into every business discipline. “It will be a more enterprise view to analytics than the previously siloed view,” which focused, separately, on IT and marketing.
“The well-rounded individual, the data manager who is evolving, is the future,” agrees Kristen Lamoreaux, president and CEO of Lamoreaux Search, a Philadelphia-area recruiting firm. “I only have two clients that are looking for hard core statistics and analysis quant people,” she says, and that’s because their organizations are already set up to support them.
Epic Media’s Okin recalls that a couple of years ago, the company parted ways with a Ph.D. “who was always busy, but what he produced wasn’t that usable.” He blames, in part, “our failure in understanding enough about that world at the time to manage him appropriately.” Okin and other senior IT leaders know more now about the nature of working with big data and the results they want to achieve.
“Maybe we would have butted heads,” Okin says. “But maybe he would have listened [when we told him] great, that’s a cool problem to solve, but this particular business issue, here’s exactly what your goal needs to be.”
Elana Varon is a freelance writer based in the Boston area. Follow her on Twitter at @elanavaron.