The prevailing view of Hadoop today is that the real value of early Hadoop adoption has yet to be proven among business enterprises. Many businesses have realized the benefits of storing their data within Hadoop, but are limited in their understanding of how to gain the most value from its analytics capabilities. Organizations are at different levels of Hadoop adoption, but their end goals are the same: to gain as much revenue growth and operational insight as possible.
To provide more visibility into how Hadoop is actually being used by businesses, and the challenges and successes within this space, Novetta conducted the Big Data Analytics and Hadoop Survey at the recent Strata+Hadoop World event in San Jose, California. Novetta interviewed attendees from 66 companies to determine the maturity of Hadoop adoption among Strata + Hadoop World visitors and to pinpoint the status of their Hadoop implementation plans and efforts. These individuals came from a variety of backgrounds and with various job titles, including developers, data scientists, data analysts, CIOs, CTOs, enterprise architects, and more.
Respondents provided insight into their Hadoop adoption, the nature of their investment, the business case that drove the investment, the challenges they are facing in using Hadoop, and the sources of data they are analyzing.
The results bode well for Hadoop and its early adopters, with 79 percent of polling organizations successfully investing in Hadoop. This result is in line with a 2014 Gartner report that indicated 73 percent of organizations have invested or plan to invest in big data in the next two years, up from 64 percent in 2013. That same Gartner report stated that organizations are starting to get off the fence about their big data investment: The number of organizations reporting they had no plans for big data investment fell from 31 percent in 2013 to 24 percent in 2014.
Organizations understood in previous years that investing in insights for their big data was important, but did not have a strategy to attack the massive amounts of stored information. The good news is this seems to have transitioned from scary to necessary in just two years.
When it was introduced, Hadoop promised to transform business economics, making data storage and computing more cost-effective. And it seems that it is making good on that promise. According to the Big Data Analytics and Hadoop Survey, 60 percent of businesses say that Hadoop is delivering value.
According to the Institute of Electrical and Electronics Engineers, the varied nature of stored data makes data integration and interoperability challenging for organizations that are deploying big data architectures. And this was reflected in the survey results: 30 percent of respondents reported that Hadoop’s inability to link various data sources is their biggest challenge; 28 percent cited the exploding volume of data as their biggest pain point, and 16 percent blamed the expanding variety of data. Of the various data types, survey respondents ranked their data analysis priorities. Transaction data lead the race, at 59 percent, and customer data and log data closely followed, at 57 percent.
When leveraging Hadoop, organizations have to contend with structured, semi-structured, and unstructured data. To gain true insights, enterprises must determine a means to join all three data types. The new challenge is no longer strictly Hadoop implementation, as adoption is reaching critical mass at this point. The new challenge is integrating disparate data sets within Hadoop. Organizations must deal with unstructured and fragmented data in a sound, repeatable way to extract valuable new insights from big data.
To view additional survey findings, click on the infographic below.
Jennifer Reed is Director of Product Management at Novetta. She is responsible for defining and implementing product strategy for Novetta Entity Analytics; establishing and maintaining relationships with clients, partners, and analysts; seeking new market opportunities; and providing oversight of overall strategy, technical, and marketing aspects of the product. Jennifer joined Novetta after serving as a Senior Product Manager at IBM for InfoSphere MDM. While at IBM, she was responsible for overseeing MDM strategy for big data, including unstructured data correlation, for which she was a co-inventor, and entity resolution on Hadoop. With more than 20 years of technical expertise and background in financial services and government, Jennifer is focused on turning that knowledge into easier-to-implement solutions that solve the highly complex, real-world needs of big data customers.
Subscribe to Data Informed for the latest information and news on big data and analytics for the enterprise.