Adopting Agile may require a cultural shift and in the beginning can be disruptive to an organization. However, as I mentioned in Part 1 of this blog series, Agile Data Integration holds the promise to increase chances of success, deliver projects faster, and reduce defects. Applying Lean principles within your organization can help ease the transition to Agile Data Integration. Lean is a set of principles first explored in the context of data integration by John Schmidt and David Lyle in their book on Lean Integration. First and foremost Lean recommends an organization focus on eliminating waste and optimizing the data integration process from the customers’ perspective. Agile Data Integration maximizes the business value of projects (e.g. Agile BI, Data Warehousing, Big Data Analytics, Data Migration, etc.) because you can get it right the first time by delivering exactly what the business needs when they need it. Break big projects into smaller more manageable deliverables so that you can incrementally deliver value to the business. Agile Data Integration also recommends the following:
- Use empirical process control – Don’t try to plan everything up front (fixed date, fixed cost, fixed price). Projects are unpredictable by nature. This illusion of predictability is very reassuring and not easy to overcome. It also removes flexibility from the project. Change is imminent so you must be flexible and agile to deal with changes. Use empirical process control instead of trying to plan everything up front due to the unpredictability of DI projects
- Facilitate collaboration – Frequent and effective collaboration between core team members is required. For ongoing projects like DW/BI, teams struggle with changing requirements and complex technologies. Agile requires much more collaboration between management, the customer, and development teams (as opposed to the throw it over the wall and wait-n-see approach). Requirements can emerge as the product emerges or business environment changes. DW/BI projects just like many systems development projects are unpredictable in nature. Empirical process control is best suited for managing complex processes. It requires frequent inspection and adaptive response
- Remove impediments – The project leader (or SCRUM master) is tasked to remove impediments and make sure the project team members are most productive. Provide team members with the best tools to complete their tasks efficiently and effectively. Remove non-value added tasks from their daily routine and continuously optimize the process to meet the needs of the customer.
- Assess projects frequently – Agile gives management direct control of the project with frequent assessment of value, cost, duration, etc. The team needs the tools to understand and measure progress, quality, and what is left to complete. Frequently inspect, assess, and adjust. Demonstrate any piece of user functionality on the selected technology (e.g. Sprint goal in SCRUM).
- Adapt to change – Learn from each iteration and quickly adapt to changes. Business requirements will change, new data sources may emerge. Plan for change and implement a flexible data architecture built on adaptive data services to hide underlying data complexities, insulate projects and applications from change, maximize reuse, and ensure business rules and policies are consistently applied across projects and the enterprise.
- Balance design & production – It’s important to balance design and production so that the project continuously evolves to meet requirements and incrementally deliver value to the business. Too often there is a lot of design with little construction or too much construction without enough design. This separation of design and production results in too many tasks that don’t contribute to creating software (e.g. endless meetings, back-and-forth between stage gates, rework, rubber-stamp code reviews, etc.).