As more companies explore how to tackle big data and analytics, one of the first (of many) challenges they will face is the dearth of qualified personnel to help guide their efforts.
Most companies do not have the people or technology in place to begin a big data journey, let alone achieve a return on their investment. Because of this, it becomes necessary to find and engage with third parties. There are upsides and downsides to these arrangements that are particular to big data.
For most things that companies hire outside providers for – business process outsourcing (BPO), software-as-a-service, IT outsourcing – there are pre-defined boundaries, rules, and policies, all backed up by precedence that can guide successful engagements and outcomes. Unlike these more traditional business outsourcing arrangements, big data does not come neatly packaged.
Because big data outsourcing is so new, there are few examples to follow. And the very nature of the things that companies are looking for –insights into their customer base, operational efficiencies, cures for diseases, etc. – actually works against placing boundaries on the relationship. The whole point is to discover new things, and this requires a degree of freedom that would not be tolerated in more traditional outsourcing agreements.
“Big data is different from other types of technology and business process outsourcing arrangements,” said Mike Bennett, a partner in the Intellectual Property department at the law firm Edwards Wildman. “Traditional outsourcing tends to be statically defined. But big data is the opposite.”
Big data is also special because of the technologies required to make sense of it. Not all servers, networks, databases, and architectures are created equal. This can have a big impact on how successfully the data in question is handled, parsed, stored, accessed, or otherwise manipulated. Also, big data outsourcing is very dependent on business need: Is it just raw processing power that is required, or do data scientists need to be involved? Does the project require a deep knowledge of the industry from which the data is generated? Are there regulatory and privacy concerns?
“There is no routine yet,” says Gartner Research Director Svetlana Sicular. “It’s all innovation.”
In big data outsourcing, the data itself takes center stage, said Adam Haverson, a senior manager at CapTech Consulting, an IT management consulting and system integration firm. Questions surrounding the data abound: What data will be used? From what sources? If it is less than perfect, will that affect the outcome? What about ETL issues? What is an acceptable amount of compression?
“Compression and ‘lossy’ summarization play a large role in the ability of big data to produce actionable insights in a cost-effective manner,” he said. “These are fundamentally different characteristics than you find with other services which are candidates for outsourcing.”
Haverson points to a simple example: comparing the difference in quality between a non-compressed bitmap file and a JPEG file. They can easily differ in size by a factor of 8x with minimal quality difference. But if you are willing to accept minor degradation in clarity, you can get a 20x or better compression and still preserve the image.
“A parallel exists for big data,” Haverson said. “It can be skillfully aggregated in ways that require less space while making it faster and easier to process. But a high level of skill and trust is required. Not all vendors are capable of this level of nuance, and not all firms want to outsource a function this close to home.”
Then there is the question of what to do with the results, said Wilco Van Ginkel, co-chair of the Cloud Security Alliance’s Big Data Working Group.
“At the end of the day,” said Van Ginkel, “if all that data doesn’t provide me anything new or good, why would I even bother? That is the key question nowadays. The underlying infrastructure and data technologies are maturing, and there is lots of research, investments, and startups going on in that regard.”
Then there is the experience and skill of the company doing the outsourcing. Big data is new. Much of the technology to store, manage, parse, and analyze it is new. And the interpretation of results is often the subject of great debate. So finding the right outsourcing provider isn’t a matter of simply picking one off the first page of search results.
Third-party vendors that specialize in big data still have more expertise than the average company, and finding the right one should really be just a matter of applying tried and true processes for partnering with any contractor or third party.
“While the outsourcing of such big data projects is very different from outsourcing the management of telephone systems or managed desktop services, it is not any greater risk,” said Dev Patel, CEO of BitYota, a provider of third-party data warehouse services. “You can argue it’s a greater risk if an SLA of a finance trader’s desktop is not accessible for a few minutes.”
Like any outsourcing arrangement, the benefit comes from being able to spin-up hard to find resources quickly. Also, outsourcing big data gives companies the opportunity to see if there’s any value to be had. Outsourcers also can serve to fill the gaps as in-house expertise and capabilities are developed, even providing guidance and expertise along the way, said Van Ginkel.
“The advantage is that they have seen different customers and might get a better idea of workable or proven data templates, which tend to work in a specific area or for a specific problem,” he said.
Currently, there are two main use cases for big data outsourcing, Sicular said. The first is data warehouse optimization, when companies upload and store unstructured data so they can run queries against it. These business-extension use cases make up about 80 percent of the market today.
The other 20 percent fall into what Gartner calls “game-changing” use cases. This is when new business models get tried out or predictive and prescriptive analytics get applied to lots of unstructured and structured data to see what shakes out.
The two biggest problems companies will run into when they outsource big data is outsourcing too much (so if the project fails, it fails completely), and not being able to interpret the results.
But with few options and the inability of most companies to deal with big data internally, outsourcing may be the best way to go.
“The bottom line is that BDaaS (big data-as-a-service) can do a good job providing the underlying platform,” said Van Ginkel, “but the real hard problems are still for the company to resolve.”
Now a freelance writer, in a former, not-too-distant life, Allen Bernard was the managing editor of CIOUpdate.com and numerous other technology websites. Since 2000, Allen has written, assigned, and edited thousands of articles that focus on the intersection of technology and business. As well as content marketing and PR, he now writes for Data Informed, Ziff Davis B2B, CIO.com, the Economist Intelligence Unit, and other high-quality publications. Originally from the Boston area, Allen now calls Columbus, Ohio, home. He can be reached at 614-937-2316 or firstname.lastname@example.org. Please follow him on Twitter at @allen_bernard1, on Google+ or on LinkedIn.
Subscribe to Data Informed for the latest information and news on big data and analytics for the enterprise.