Can Big Data flourish in an Agile environment? Big Data projects mostly follow the traditional Waterfall approach including Analysis, Requirements Definition, Design, Build and Test phases.
On the face of it, this approach seems time-tested and rational. You assess the current scenario, identify tools and techniques and address the issue. You document your objectives and prepare a list or make a report. You also rope in users to determine how to design each product and you go on to build the documented solution and finally test it to check if it works just as planned.
The main argument behind Big Data projects is that it can provide patterns and pinpoint trends that would otherwise be unavailable. However, there is one important link that can go amiss: that all data is useless unless and until it is converted into intelligence. That is where your Big Data project turns into a Business Intelligence project that helps you gain insights into what is not known.
This knowledge will prompt new questions that you did not know you wanted answers to in the first place. What does this imply for an enterprise that is building software solutions, services or other products? It means you need to build a little to know what you want to build at the end.
This is the reason why some companies are using the Agile approach to run data management projects. Using Agile Data means a joint approach to development and delivery; cross-functional teams comprising members of the business and IT, work in “data labs” which bring reliable insights that allow the company to target business priorities and ensure positive outcomes quickly.
Make Your Big Data Agile in Four Steps
- Business-Oriented Approach: Identify business use cases for advanced analytics. List the different streams of data required. Make two road-maps: one for digital business objectives: budgets, and time frames and milestones; the other for data requirements to build an effective big data architecture and provide seamless analytics support.
- Joint-Ownership Attitude: Make the business side physically sit with the IT team, understand data migration and data management protocols to come up with just-in-time data requirements, and validate the business case for the proposed solutions, assuring quality. Employ collaboration and social networking tools amongst dispersed teams.
- Cross-functional Teams: Build Scrum teams that gather data scientists, engineers, developers, quality control specialists and business information owners in “data labs” to develop and deliver minimally viable data-migration products and processes that may be released, tested, and refined quickly
- Adopt Emerging Technologies: Say goodbye to traditional data warehouses and introduce “data lakes” – unlocking data from silos of structured and unstructured business information collected from different systems across different functions and locations of a Company; it not only reduces storage costs but helps to retain compatibility with standard discovery tools and can be reconfigured with ease.
Leadership plays a pivotal role to accelerate Big Data transformation in an Agile manner. Reading the DataOps Manifesto could be a starting point.