Data Firm Uses Development Tools To Bypass Shortage of Hadoop Skills

by   |   September 10, 2013 11:26 am   |   0 Comments

Big data development platforms can simplify the difficult task of running and building complex Apache Hadoop applications. But there’s an unexpected upside to using these tools that extends beyond development cycles and infrastructure stacks. For some, a powerful development platform can also help cut workforce costs and recruit hard-to-find talent. That’s because tools like Continuuity’s Reactor, Hadapt’s Adaptive Analytical Platform and Hortonworks Data Platform, provide workers with a less stress-inducing framework for building big data analytics and applications.

Related Stories

How Pig, Hive and Zookeeper build apps on Hadoop and MapReduce.
Read the story »

Enterprises value data pros with multiple skills, Dice survey shows.
Read the story »

Hadoop sandboxes provide low-risk entry for new programmers.
Read the story »

Focus on analytics skills.
Read the story »

Just ask Lotame, a Columbia, Md.-based company that provides publishers and marketers with a data management platform called Crowd Control for managing their online marketing data. Lotame has long relied on Hadoop to crunch “hundreds of terabytes” of audience data, says Jeremy Pinkham, Lotame’s CTO. However, migrating this data from one flow to the next in order to build applications for their customers has never been easy.

“Our infrastructure is fairly large with hundreds of nodes on Hadoop,” says Pinkham. “Just managing that infrastructure is a challenge.” Popular programming tools like Pig have helped Lotame ease the process, but Pinkham says they can still “feel awkward, like you’re still trying to work around the tool and fit whatever it is you’re doing into Pig or MapReduce.”

Tired of juggling all the “plumbing and framework” involved in building big data apps for its customers, Lotame turned to Continuuity for its development platform. The tool works by adding layers of abstraction to Lotame’s complex programming infrastructure so that its team of developers can build big data applications while avoiding the complicated mechanics of Hadoop.

“Continuuity provides APIs that are building blocks for developing apps so that you’re not even thinking about using Hadoop,” says Pinkham. But while these building blocks were intended to help Lotame deliver real-time data analytics to its customers, they are also producing some surprising workforce dividends.

Working with Hadoop in a development environment requires specialized skills in large-scale data analysis, programming tools like Pig and Hive, as well as an in-depth understanding of complex infrastructures. However, by simplifying the process of building big data apps, Continuuity has freed up more of Pinkham’s team of 12 developers to work on apps. Lotame can now use as many as two times the number of developers on building apps rather than the miniscule team of two it used to rely on for its Hadoop know-how, Pinkham says.

Eliminating the need for specialized Hadoop skills also promises to have a significant impact on Lotame’s payroll expenses. Pinkham estimates that the average salary of a developer with Hadoop expertise is 25 to 30 percent higher than that of your average Java developer. A recent survey from put the average salaries of IT professionals with Hadoop skills at between $84,000 and $118,000 depending on the specifics of the role and what complementary skills were required.

To keep up with Lotame’s growth, Pinkham estimates that the company would have had to hire another six developers with Hadoop expertise over the next year, or pull existing developers off the job for extensive in-house training. With Continuity, Pinkham predicts the company will be able to avoid such a costly hiring spree and investment in training, representing “a very big deal for us in savings and productivity.”

Another workforce advantage of simplifying Hadoop involves app development cycles. It’s not uncommon for Lotame’s developers to work on as many as five new apps at a time. Integrating Continuuity’s platform has “eliminated half the work” involved in building these apps, enabling team members to focus on innovating rather than juggling huge batch processes, says Pinkham. During a recent flight from New York to Palo Alto, Pinkham tried rebuilding an old Lotame app that had taken a full month to develop on MapReduce. “By the time I landed, from East Coast to West Coast, I pretty much had the app re-implemented,” he says.

The ability to churn out new apps gives Lotame a competitive edge in a highly niche market. But it also serves as a compelling selling feature for new recruits.

“As a technology company, one of the things you need to attract engineers is the chance for them to do interesting things,” says Pinkham. “[The Continuuity platform]  is a cool technology for them to play with.” And because the platform eliminates the need for specialized Hadoop skills, Lotame’s pool of potential candidates in today’s tight labor market automatically widens.

“Finding engineers who can build apps on Hadoop effectively is a challenge,” says Pinkham. “So one of the big benefits of Continuuity is you don’t need to know all that knowledge. The platform takes care of all that plumbing for you so that you can really concentrate on just building apps.”

Cindy Waxer, a contributing editor who covers workforce analytics and other topics for Data Informed, is a Toronto-based freelance journalist and a contributor to publications including The Economist and MIT Technology Review. She can be reached at or via Twitter: @Cwaxer.

Tags: , ,

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>