Businesses are looking to incorporate social media analytics into their business decisions, and are searching for a new generation of tools using both machine learning and natural language processing algorithms to ensure the insights gained from listening online are accurate enough to be valuable.
Keyword-based sentiment analysis alone, while potentially useful when put into the right context, can be inaccurate. That’s why Jeff Doak, the social platform director at the advertising agency Team Detroit, said he discourages his team from using an automated sentiment analysis as a tool for business decisions; he said keyword-based sentiment analysis is accurate between 50 and 60 percent of the time.
“I advise against that in general, because it’s misleading and inaccurate, and because there is no context around it you don’t know what it means,” Doak said. “It’s not actionable in any way.”
Team Detroit’s biggest client is Ford Motor Company, for which is has created campaigns including one featuring an orange puppet called Doug to support a new model of the Focus. In his job, Doak said he experiments with a lot of data relating to marketing campaigns, auto sales and feedback on social media and blogs. One key challenge: trying to establish benchmarks to understand just what correlation there is between what people are saying online and how people buy cars.
“How do we roll all of this up to something that’s really measuring how many cars are we selling because of the way that we’re engaged on social media?” Doak said. “That’s the Holy Grail, and that’s really tricky. We’re trying to get where we’re predictive a little bit, where we can say if we change this social media, this will be the effect.” Doak added that the experiments in making social media analysis into a decision-support tool are ongoing.
Such experiments are at part of marketers’ search for better ways to measure online sentiment. It’s a quest Doak is familiar with because before Team Detroit, he worked at the social media listening company Converseon.
A Road to Machine Learning Starts with Human Labor
Converseon ’s newest release, the ConveyAPI, which provides annotated information with customized classifiers about brands, offering measurements for sentiment, intensity, emotion and relevance. The system is based on several years of collecting and human-coding social media data during Converseon’s days as a marketing and brand research services company. The company was founded in 2001 and started developing software, its focus now, in 2007.
Converseon CEO Rob Key said the labor was manual and covered a wide range of content. “We ended up with millions of meticulously human-coded records across 50,000 brands and various industries.” Three coders would look at a particular record and all had to agree on its classifications. “It was a very manual process. But what we ended up with was this incredible training corpus which we then layered into the machine learning technologies,” he said.
Converseon hired computational linguists Jason Baldridge of the University of Texas and Philip Resnik of the University of Maryland, who dug into that machine learning corpus and created a natural language processing system that agrees with humans on the sentiment of a statement 93 percent of the time, far above the average sentiment analysis tool.
“We’re approximating human level performance from machines, and able to capture things around sarcasm and slang and implicit meaning that the last generation technologies haven’t been able to do,” Key said.
Seth Grimes, founder of analytics strategy consultancy Alta Plana Corp., said ConveyAPI is one of a handful of “second generation” social media analytics platforms that are using machine learning and natural language processing; others include AlchemyAPI and OpenAmplify.
“You do have a first generation that looks for keywords, and that’s very crude,” Grimes said. “You have new products launching with that crude approach. I think it’s fair to call it a second generation.”
Grimes, who has worked as a consultant with Converseon, said the second generation has been emerging for a few years; Converseon’s approach stands out because of their history as a brand research agency.
“They came to building a tool from having deep experience as an agency,” Grimes said. “They weren’t software developers first, first they were working with customers, and that’s a telling difference.”
Susan Etlinger, a research analyst at the Altimeter Group, said these APIs create a data stream that allows social media analysis to break out of its own silo inside an enterprise and start to combine with other data to create the same type of correlations Doak is looking for with his clients at Ford.
“There is definitely an interest in the industry among enterprise business to start integrating social data with enterprise data, like business intelligence or market research,” Etlinger said. “Given that the API is open to develop on, that opens a whole host of options. You can start to tune your (social media analytics) tools to better fit your business, and that’s something that has significant value.”
Email Staff Writer Ian B. Murphy at email@example.com. Follow him on Twitter .