Connect with us

Hi, what are you looking for?


Four Steps to Big Data Agility

Can Big Data flourish in an Agile environment? Big Data projects mostly follow the traditional Waterfall approach including Analysis, Requirements Definition, Design, Build and Test phases.

On the face of it, this approach seems time-tested and rational. You assess the current scenario, identify tools and techniques and address the issue. You document your objectives and prepare a list or make a report. You also rope in users to determine how to design each product and you go on to build the documented solution and finally test it to check if it works just as planned.

The main argument behind Big Data projects is that it can provide patterns and pinpoint trends that would otherwise be unavailable. However, there is one important link that can go amiss: that all data is useless unless and until it is converted into intelligence. That is where your Big Data project turns into a Business Intelligence project that helps you gain insights into what is not known.

This knowledge will prompt new questions that you did not know you wanted answers to in the first place. What does this imply for an enterprise that is building software solutions, services or other products? It means you need to build a little to know what you want to build at the end.

This is the reason why some companies are using the Agile approach to run data management projects. Using Agile Data means a joint approach to development and delivery; cross-functional teams comprising members of the business and IT, work in “data labs” which bring reliable insights that allow the company to target business priorities and ensure positive outcomes quickly.

Make Your Big Data Agile in Four Steps

  • Business-Oriented Approach: Identify business use cases for advanced analytics. List the different streams of data required. Make two road-maps: one for digital business objectives: budgets, and time frames and milestones; the other for data requirements to build an effective big data architecture and provide seamless analytics support.
  • Joint-Ownership Attitude: Make the business side physically sit with the IT team, understand data migration and data management protocols to come up with just-in-time data requirements, and validate the business case for the proposed solutions, assuring quality. Employ collaboration and social networking tools amongst dispersed teams.
  • Cross-functional Teams: Build Scrum teams that gather data scientists, engineers, developers, quality control specialists and business information owners in “data labs” to develop and deliver minimally viable data-migration products and processes that may be released, tested, and refined quickly
  • Adopt Emerging Technologies: Say goodbye to traditional data warehouses and introduce “data lakes” – unlocking data from silos of structured and unstructured business information collected from different systems across different functions and locations of a Company; it not only reduces storage costs but helps to retain compatibility with standard discovery tools and can be reconfigured with ease.

Leadership plays a pivotal role to accelerate Big Data transformation in an Agile manner. Reading the DataOps Manifesto could be a starting point.

Written By

Arvind Rongala is Director, Invensis Learning, a global training and certification provider in the IT, technical and business management fields. Under his stellar leadership, Invensis Learning has trained 50,000 plus professionals across the World, and is a trusted partner for Fortune 1000 companies. Associated with the Invensis group of companies for nearly a decade, Arvind is now a recognized name in the learning and continuing education sphere. His views on topics like Agile Project Management, DevOps, IT Security and Governance, Quality Management and Digital Marketing are valued by media, industry peers and the business community at large.

Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

You May Also Like


For even more than a decade, the Internet of Things has been nothing further than a nerdy conception. With the digital age in high...


Big data is rapidly transforming business intelligence. Companies, governments, and education institutions are using it for many purposes including competition, innovation, productivity, consumer insights,...


Photo by Alexander Sinn on Unsplash Big data has affected almost every aspect of people’s life, whether you are booking a car rental in...


Source: Pixabay After the outbreak of the COVID-19 pandemic, the importance of storing and managing online data has become one of the main priorities...