Adopting an Agile Data Approach: Tips and Challenges

by   |   March 3, 2016 5:30 am   |   1 Comments

Big data is constantly growing and changing, and businesses need to adapt accordingly. To adapt successfully, businesses need agile data management processes. Data agility refers to the ability to adapt to the changing nature of big data successfully. The changing nature of big data reflects the dynamic nature of business. Companies constantly have to re-evaluate and modify strategies according to changing business realities. The situation represents a big shift from when companies could afford to stick with a particular strategy for some time.

Data agility can be likened to physical agility. Physical agility enables you to become swifter in movements, avoid injuries, and respond to rapidly changing situations quickly. Similarly, data agility reflects how well a company is managing the big data it is collecting. This is much more than simply collecting and analyzing data. This is about using the data so that you are able to respond to change quickly and appropriately, which is a constant in today’s business world.

Data agility enables companies to strategize based on insights and to evaluate and modify strategies in real time or as close to real time as possible. For example, instead of structuring data into static schemas, agile data analysis practices can help analysts explore different approaches to the structuring of data so that analysts find more ways of discovering the value hidden in data. This approach is known as schema-on-read. Another example of data agility is the retaining of unused data. Many businesses discard or neglect unused data, a practice that might deprive them of valuable, unearthed insights. The low price points of the Hadoop File System enables companies to retain the unused data.

The following tips can help you adopt data agility and make it work:

Assess the condition of data. First, your data must be ready to be used in agile mode. So begin by assessing the condition of your data. For example, if there is data duplication, too much junk, or outdated data, or if the quantity of data is not significant enough, you must address these issues before embarking on data agility adoption drive.

Related Stories

Rethinking DBMS: The Advantages of an Agile Infrastructure.
Read the story »

The Benefits of Agile Analytics Development and How to Do It Right.
Read the story »

Big Data and the Progression toward Streaming Analytics.
Read the story »

Speed Analytics with a Data First Approach.
Read the story »

Draw from all data sources. You need to document all types of customer data that your business produces from all sources, internal or external. You also need to analyze what types of data you are collecting or not collecting. After that, deploy and unify all the data in a customer data platform so that all data are available centrally, in real time. That platform should be the place where all data gets analyzed and you should be accessing the platform all the time.

Real-time actions. For organizations adopting the data agility model, the old way of data collection and analysis is out. With the old model, data would be collected, stored, and analyzed, and then insights were collected. The entire process would take a lot of time. Now, organizations need to respond to frequent feedback and changes, and take actions almost instantly. The luxury of taking weeks or months to gather and normalize data into a data warehouse is a thing of the past.

Foster a culture of agility. Fostering a culture of agility is as important as having updated tools and processes. You may have the best tools, processes, training programs, and technologies, but you also must have people who are excited about data agility and willing to put things into practice. Different departments need to collaborate and put the processes into practice. For example, the data warehouse team needs to be in sync with the data reporting team so that data additions are reported and analyzed instantly.

Data Agility Adoption Challenges

It goes without saying that organizations face a lot of challenges when adopting data agility. These challenges are multi-faceted: financial, process, human resources, and tools identification.

Financial. Data agility adoption can involve hiring skilled personnel and a significant investment in software tools and technologies, and a lot of training. A lot of companies do not have the budget or financial resources to invest in data agility adoption. While they may do an honest attempt at processes and human resources, arranging finances is a challenge for them.

Process. Existing processes are often incompatible with the standards that data agility requires. It can take a long time to fix the errors in existing processes and make them compatible with an agile data environment. An assessment of existing processes is required to identify necessary changes. The challenge is even bigger in organizations that have a lot of legacy processes. For example, analytics collection policies in the waterfall model that follow a sequential process are incompatible with a data agility model because by the time the data is taken for processing, it has become outdated. An agile data environment requires data to be collected in real-time as much as possible.

Human resources. This is perhaps the biggest challenge of all. Getting your employees to adopt the agile mindset is not easy, especially if something different had been practiced previously. The challenge is more visible in big organizations with a large employee base and legacy processes. Initiating change and maintaining the tempo are two of the biggest challenges. However, with a concerted effort at culture change, things can improve over time.

Identifying the right tools. Organizations need to identify tools that help them adopt data agility in their own context. The choice of tools ideally needs to be based on certain principles. For example, the tools need to be able to process real-time or near real-time data and provide analytics; the framework needs to be flexible to accommodate changes and be platform independent, and the tools need to empower the business users to create analytics based on their unique needs. You also need to have qualified people to handle the tools.

For organizations that need to improve their ability to respond to changing business conditions, conditions, and goals, an agile data approach is all but inevitable. A business environment that is more dynamic than ever demands the ability to use data to make decisions and modify strategies in real time.

Kaushik Pal has more than 16 years of experience as a technical architect and software consultant in enterprise application and product development. He is interested in new technology and innovation areas, as well as technical writing. His main focus area is web architecture, web technologies, Java/J2EE, Open source, big data, cloud, and mobile technologies. You can find more of his work at Email him at

Subscribe to Data Informed
for the latest information and news on big data and analytics for the enterprise.

Anzo Smart Data Lake [Whitepaper]

Tags: , , , , , , , ,

One Comment

  1. Posted March 4, 2016 at 6:37 am | Permalink

    Hi Kaushik – Nice post with a great overview. A lot of companies are struggling with this now. It helps a lot if a company has already spent time to make their environment more agile but we often see that companies take a traditional waterfall approach to creating an agile environment! (so it takes a long time).

Post a Comment

Your email is never published nor shared. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>