The rise of big data analytics has set off a race to design more powerful tools to handle the demands of data-focused applications. But often the most innovative companies are those that marry data science to a specific business problem and put that technology into the hands of business people, according to experts at a big data conference.
One of the biggest challenges software companies are trying to address is collecting and analyzing data from varied sources, including social media, mobile devices, and machines, such as two-way utility meters or networked medical devices.
The data surge has led to the formation of several new companies around Hadoop (such as Cloudera, Hortonworks and MapR) and a new generation of databases (NuoDB, ParAccel and MemSQL to name a few) designed to handle high volumes of varied data sources or analyze streams of information. Meanwhile, established database companies continue to focus on scalability, security, and performance of the software infrastructure.
But much of the innovation in the big data world now comes from people who can combine these powerful tools with deep vertical industry knowledge, said Chris Lynch, a partner at Atlas Venture and former CEO of database company Vertica Systems (now part of HP).
“Too many people are focused on being platforms and tools companies. It’s really hard (but) the world doesn’t need another NoSQL database,” said Lynch, who spoke at a November 21 conference in Boston on big data and analytics hosted by Xconomy. “What’s better is if you can build an application that’s highly integrated and solves a specific business process.”
For example, a group of people with years of experience in the travel industry formed a startup called Hopper that aims to reinvent the online travel business. (Lynch’s firm is an investor.)
Lynch also sees a lot of potential for services that can bring the skills of hard-to-find data scientists to many companies by integrating with existing business applications. He invested in a company called Nutonian, which seeks to find patterns and relationships in data to improve predictive analytics. “When you truly automate so that you have a marketing analyst or a quant in a box, that’s ambitious,” he said.
Wanted: Bridges to Machine Data
When it comes to machine data, the amount of data is expected to increase as sensors and processors are placed into more and more machines. General Electric, which has an Industrial Internet initiative, now collects and analyzes data from wind turbines, medical devices, and gas turbines at power plants. In general, one of the biggest challenges associated with machine data is access, or simply getting information in a relatively easy way, which can affect the performance of an application.
“Right now, if you look at different domain-specific standards, within some micro areas there is some degree of standardization,” said Stephen Wolfram, the CEO of search company Wolfram Research who presented at the event. “(But) as a whole, every company putting out an API or a device is always going to put some special feature they want to capture and they end up putting in some special thing into the way they communicate with it that isn’t standardized at all.”
For his part, Wolfram is creating connections between cloud-based information services and machine data. At the event, he announced that Wolfram Research software now runs on Raspberry Pi, an inexpensive palm-size computer processor often used by hobbyists. As embedded hardware for devices gets more powerful, software developers will more easily be able to write applications that combine data from devices in the field with data available on the cloud, Wolfram said.
Mobile devices, too, offer the potential for a whole new set of applications, said John Joseph, the president of startup DataGravity, which is developing software to glean insights from corporations’ multiple sources of data. In health care, for instance, consumer mobile devices could be used to monitor people’s health and improve a diagnosis, he said. Using mobile devices, such as smartphones and tablets, for product design and manufacturing, can also greatly speed up product development.
Forecast: Analytics in the Cloud
As technology gets more sophisticated, IT organizations need to focus more on applying analytics tools to specific business problems, he said. “With the onslaught of cloud apps, it’s much less a discussion about infrastructure and more about pushing insight and information to the business user. IT’s role and the definition of its job function is changing rapidly,” he said.
The combination of big data with cloud computing and mobile represents a major transition point, along the lines of the shift from mainframe computing to PCs, said Jit Saxena, the founder of Netezza (now part of IBM) and a board member of data start-ups Hadapt and ParElastic. In the past, Saxena told customers he didn’t care about their business problems because his technology was so much faster than competing systems. But that won’t attitude won’t work any longer because business people have become much work technically savvy and have higher expectations.
“Business users don’t care what technology they’re using. In the future, they will have requirements for the technology that will be provided to them as a service,” he said. “What happens behind the scenes won’t be talked about at all.”
Martin LaMonica is a technology journalist in the Boston area. Follow him on Twitter @mlamonica.