Far from the big industry convention halls, grassroots user group meetings are now a place where developers can openly swap trade secrets with competitors, land a job without a resume and listen to top companies confess their open source failures. Hadoop-related gatherings are certainly no exception.
Just ask Leon Katsnelson, a program director for big data and cloud computing at IBM in Toronto. “I’ve been attending these gatherings long before they were called Meetups,” says Katsnelson, a 28-year veteran of Big Blue. “They’ve just become more mainstream.”
Today, Katsnelson is a member of multiple user groups including HackerNest Toronto, HTML5 Toronto Web Developers, New York Hadoop User Group, San Francisco Hadoop Users and Toronto Ruby Brigade. He is also an organizer of THUG (Toronto Hadoop User Group), arranging gatherings for the group on everything from Apache Pig to using Hadoop in the cloud.
Industry trade shows may showcase the latest and greatest Hadoop technologies, and vendors such as Hortonworks, Cloudera and MapR offer Hadoop training courses. But only user group gatherings provide a vendor-agnostic space where developers – even those from competing companies – can speak freely about their trials and tribulations crunching big data, according to Katsnelson.
“Where Meetups become really useful is when your colleagues present their experiences,” says Katsnelson. “That’s not something you’re going to hear from a vendor at a conference.” Katsnelson cites a recent gathering where the Ontario Institute for Cancer Research discussed the various open source technologies they use to store and annotate biological data such as cancer mutations.
By listening to “how many machines organizations like the OICR use, the size of their machines, and how much workload those machines can handle,” Katsnelson says he can better determine the size of data clusters he needs to complete similar types of projects. “Attending Meetups has helped me validate the criteria we use for selecting data cluster sizes,” he says. “It gives me greater ammunition for making my decisions.”
For every user group gathering organized by a user like Katsnelson, there’s a big data vendor sponsoring a pub night or seminar. Take, for example, DataStax. DataStax has arranged 40 Meetups in 25 cities this year alone. Seminars have included “We Messed Up So You Don’t Have To,” where vendors share their Cassandra-related horror stories, to sessions on Cassandra performance and scalability.
For example, a recent DataStax user group gathering featured Netflix’s cloud architect Adrian Cockcroft performing a live demo of Cassandra performance and scalability on Amazon Web Services platform. Cockcroft illustrated how the streaming media giant managed to increase the number of nodes in a cluster without compromising performance. “The community loves use cases,” says Christian Hasker, DataStax’s senior director of product marketing, who leads the big data platform provider’s Apache Cassandra-related community events.
Of course, not all companies are willing to share their open source experiences. “It’s a delicate dance,” says Hasker. “The engineers have an open source perspective – they’re technology evangelists that want to tell the world about what they’re doing and how they’re doing it.” At the same time, Hasker says it’s not uncommon for protective public relations teams to put the kibosh on candor, preventing presenters from revealing their “secret sauce.”
Regardless of just how “open” the discussions are around “open source,” one thing is for certain: user group gatherings are a prime opportunity for job-seeking developers and recruiters to connect.
“Companies like to put on seminars and Meetups because there is a skills shortage in the big data space right now,” says Hasker. “Netflix is really loud and proud about what it’s doing because it wants to attract very savvy engineers who want to work at Netflix.”
Katsnelson agrees. Currently in the process of hiring a developer he met at a Toronto Ruby Brigade gathering, he says, “If you’re looking for a particular skill, Meetups are the best place. Frankly, people’s resumes are all the same. But when you see people going to Meetups after work, it tells me that they’re passionate about what they do and that they’re the best people to hire.”
The job opportunities alone might be worth the price of admission.
Cindy Waxer, a contributing editor who covers workforce analytics and other topics for Data Informed, is a Toronto-based freelance journalist and a contributor to publications including The Economist and MIT Technology Review. She can be reached at email@example.com or via Twitter @Cwaxer.