Overcome These 5 Challenges to Manage Data Overload

by   |   November 28, 2016 5:30 am   |   1 Comments

Rob Consoli, Chief Revenue Officer, Liaison Technologies

Rob Consoli, Chief Revenue Officer, Liaison Technologies

As companies are increasingly recognizing, data is the new currency in business. Enterprises harness the power of their data and apply it to improve daily operations in many ways. But the more forward-thinking businesses are taking it a step further by using data to drive innovation and disrupt their industries. In this way, companies are moving toward a data-inspired future.

Companies that want to compete and win in a data-driven economy have to find a way to leverage the power of their data and extract maximum value. But that can be a challenge as the volume of data grows and new sources of information come online in a multitude of formats, threatening data overload. Data overload, if not carefully prepared for and mitigated, could prevent an organization from enjoying any of the benefits of a data-driven economy.

There are five potential challenges associated with data overload that an organization needs to be prepared to encounter:

1) Analysis paralysis from too much information and too many sources: There is already an enormous influx of data streaming into the enterprise from multiple sources, but it’s set to increase exponentially. Analysts predict that the Internet of Things (IoT) will comprise 200 billion connected devices by 2020. When IoT information is combined with other data sources, including cloud applications and social media, the volume of information can quickly overwhelm businesses.

It is a well documented fact that too many options can prove paralyzing for consumers, and businesses aren’t immune from that phenomenon. Without a solution that enables them to effectively handle the influx of information and harmonize data from multiple sources, businesses will face disrupted data workflows.

2) Silos created by fragmented data solutions: Big data works when companies can glean insights from a unified data pool. But too often, businesses face data fragmentation. They work with a range of big data tools that each address one part of the operation, including functions like data storage, cleansing, API management, data visualization, and more.

This piecemeal approach to data management results in multiple silos, which make governance and compliance incredibly challenging. Meanwhile data quality, security, and visibility decrease while expenses and inefficiency increase.

3) Data generation and resource disadvantages for small and mid-sized businesses: Big data is expensive; it requires an investment in resources to generate, process, and store all that information. Large companies like big box retailers have the cash and infrastructure to make big data work for them — they have assets like cameras, consumer apps, and point-of-sale software to generate and make sense of data so that they can continuously improve the customer experience.

But small and mid-sized businesses typically don’t have the resources to monitor, influence, and predict customer behavior. And even those that do usually do not have a sufficiently large customer base to generate macro-level insights. Small and mid-sized businesses have to leverage all the innovation and creativity available to them in order to find ways to make the big data revolution work on their budget and with their customer base.

4) Faulty decision management processes: With big data analysis, more decisions are left to machines. That leaves the decision-making process vulnerable to the inclusion of faulty variables. In 2010, there was a stock market crash — the Flash Crash — that occurred due to a faulty decision management process that relied too extensively on algorithms. That incident demonstrated that it is unwise to leave machines completely in charge.

Businesses that use big data must guard against making similar mistakes that can wreak operational havoc downstream. To prevent a Flash Crash-like catastrophe, companies must balance the efficiency of machine algorithms with the superiority of human judgment and make adjustments as necessary.

5) Backward-looking analytics that don’t foster innovation: It’s important to acknowledge that big data algorithms are by nature backward-looking. Users propose a hypothesis, crunch historical data points, and review outcomes that fall into predetermined ranges.

This process can yield incredibly valuable insights, but its focus on historical data points doesn’t readily foster innovative thinking. It merely provides a starting point. Companies that want to facilitate calculated risk-taking to drive disruptive change will usually need to look beyond math-based big data, leveraging it while also using creativity to innovate.

These five challenges present significant barriers for companies that seek to use big data to its full potential. But the good news is that big data practices and technology are maturing, making these barriers easier to overcome.

There are now solutions on the market that enable companies to leverage the cloud for the integration and data management support they need, regardless of business size or in-house expertise. One example of such a solution is a new approach to integration and data management called data Platform as a Service (dPaaS). dPaaS is a cloud integration and data management model named for its ability to provide PaaS functionality at the point of data analysis without encumbering users with the particulars of the underlying data capture, integration, or management mechanics.

Businesses that access such innovative solutions to overcome the five challenges of data overload with a comprehensive integration and management approach will be ready to embrace a data-inspired future.


Rob Consoli is the Chief Revenue Officer for Liaison Technologies. He brings over 25 years of technology industry experience and has a demonstrated track record of successfully building teams and helping growth-oriented companies navigate cultural and process transitions as they expand operations and global reach. In this pivotal role, Rob leads Liason’s North American efforts to strategically position and sell the company’s cloud-based integration and data management solutions, as well as increase its sales to meet the company’s growth objectives. Consoli holds a Master of Science from Southern Methodist University and a Bachelor of Science from Auburn University.


Subscribe to Data Informed for the latest information and news on big data and analytics for the enterprise.

Tags: , , , , , , ,

One Comment

  1. Christine
    Posted November 30, 2016 at 10:59 am | Permalink

    Mr. Consoli, in the article you mention “Companies that want to facilitate calculated risk-taking to drive disruptive change will usually need to look beyond math-based big data, leveraging it while also using creativity to innovate.” Can you explain what that looks like? What non-math-based data can be used?

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>