Analysis and application of Agile Methodologies in BI Projects

As a software developer, the purpose of this post will be to analyze how different methodologies can positively impact the development of a project; I will compare the similarities and differences between IT and BI projects to proceed with an approach to the Agile methodology and its impact to BI projects.

For Business Intelligence professionals, with or without experience, I believe that the BI methodology topic should be considered the backbone of their professional development. A lot of professionals adopt methodologies based on Kimball’s industry standards for designing, developing, and deploying data warehouses and business intelligence systems; Likewise, other professionals design their own ‘flavours’ based on those same standards.

The Business Intelligence methodology gives us a better certainty that our project will achieve the proposed solutions within a range of time determined from an IT point of view, but it is very important to understand that BI projects have indeed two perspectives: business and IT.

The use of methodologies can orient us to correctly define the business solutions and project goals, but it won’t guarantee the success of them because of direct and indirect variables that can affect the outcome. For example, selected BI tools, employee’s technical and business backgrounds, budget, etc.

Even though I think it’s not a rule, generally in both IT and BI projects, the methodology is somehow aligned with the technology chosen as solution, in case of BI, the platform can define essential points related to the technology itself, like better practices, process documentation, component’s definition, roles involved, etc. Now, if we think about the standards that should not be affected by technology, we can consider that the business requirements should all look very similar regardless of the methodology.

We know that there are plenty of methodologies for software development. Traditionally, these methodologies focus on process control by establishing phases, tasks, techniques and tools; Reason why they tend to be squared and straightforward and are usually guided through the documentation generated in each of the designed phases.

This approach is not suitable for projects where the system environment is constantly changing — meaning BI projects — or where the nature of the project forces you to reduce the development time but keeping the quality — meaning IT projects —. Reason why, methodologies like Agile emerge as a good option to fulfill that methodology gap.

Dilbert

The impact of applying Agile methodologies to BI projects can be huge if it’s well done, BI applications are challenging by nature; As mentioned above, their requirements constantly change, plus the inevitable need to add indirect variables such as the addition of stakeholders, incremental scope and depth, which by result will directly impact on the project’s complexity. Agile Software Development addresses the challenges described above right on the principles that defines it (http://www.agilemanifesto.org/principles.html). But its success lies on the methodology used to apply those principles.

Agile development methodology attempts to provide many opportunities to assess the direction of a project throughout the development lifecycle. This is achieved through regular cadences of work, known as sprints or iterations, at the end of which teams must present a shippable increment of work. Thus by focusing on the repetition of abbreviated work cycles as well as the functional product they yield, agile methodology could be described as “iterative” and “incremental.” In an agile paradigm, every aspect of development — requirements, design, etc. — is continually revisited throughout the lifecycle. When a team stops and re-evaluates the direction of a project every two weeks, there’s always time to steer it in another direction.

Source: http://agilemethodology.org

The advantages of using Agile methodologies in BI projects become very clear after understanding the definition and principles: by developing a prioritized product features or requirements (Product Backlog) and dividing them into small chunks (Sprint Backlogs) we can deliver working software during frequent meetings with stakeholders, which at the same time will give us the opportunity to verify releases or even re-prioritize requirements.

In essence, I can conclude that the Agile methodology helps companies to build the right product instead of committing to develop a piece of software that hasn’t even been written yet. Agile empowers teams to optimize their release as it’s developed.