How can you reduce development costs and improve software reliability and accuracy at the same time? How can you make IT work together with Business while architect-ing your BI applications? If these goals sound contradictory and difficult to achieve, then Agile development may well fit the bill. Indeed in numerous BI projects, one or the other flavor of Agile is used to attain these very goals.
Defining Agile
There are several Agile development methodologies available:
• eXtreme Programming (XP)
• Scrum
• Kanban
• Feature-Driven Development (FDD)
• Crystal Clear
• Dynamic Systems Development Method (DSDM)
• Adaptive Software Development (ASD) and more…
At the core of any flavor of Agile development methodology is the iteration, which may last from 1 to 4 weeks (one unit of time) to develop a piece of the software. Each iteration is treated as an entire software project with its associated planning, design, coding, testing and documentation tasks.
What is it about Agile development which makes it particularly suitable for data warehousing and business intelligence projects?
* Agile emphasizes on communication be it through meetings (be it through the phone, VOIP, web or IM) over written documents. The idea is to get the user involved much early in the development process and incorporate their feedback, so as to minimize the risk of developing faulty software. For organizations adopting BI, very often users are clueless about the systems to build, the technology to use or even the range of analysis they require. Products are often bought after effective sales pitches from vendors and left to IT to deploy and architect. In such cases, IT can use Agile methodologies like DSDM, SCRUM or ASD to flesh out the requirements and deliver BI which actually provides insight rather than building a monolithic and unreliable data warehouse difficult to query and administer.
* Agile gels well with the evolutionary approach required for a data warehousing / BI lifecycle. Requirements change over time, and the iterations of the Agile methodology (with database re-factoring and evolutionary data modeling ) is more efficient in capturing these changes than the classical waterfall approach.
* Proof of the concept, technology and architecture is crucial to justify continued investment in DW/BI projects, especially on the enterprise scale. This is simpler and easier to do with Agile.
* Agile imbibes every member of the project team with extra responsibilities, making them owners of discrete functions and helps the project manager overcome the ‘taskmaster‘ stereotype and concentrate on being a leader or a visionary.
BI is essentially gaining competitive edge by insight into your business through lagging (measures) and leading (predictive model-based) metrics, which allows feedback cycles and restructuring of processes (Plan-Do-Check-Act Deming cycle). This essentially involves cooperation and teamwork across functions to model and understand the multi-faceted perspectives. Teamwork being the foundation of Agile, it is a natural fit for projects in BI and data warehousing.
~biguru
The Business Intelligence Blog 
I think most BI projects, and even most corporate IT departments, would benefit from a more agile approach to pretty much everything. I highly recommend that every IT professional, developer, sys admin or whatever, read the Pragmatic Programmer.
LikeLike
i think it is great idea ,to use Agile in developing BI applications, i am also intrested in how to use agile in developing DSS…..
LikeLike
I think it is great idea ,to use Agile in developing BI applications, i am also intrested in how to use agile in developing DSS…..
if we initiate Agile methodology for building DSS ,then it will be easy to build BI applications using Agle
LikeLike
Using Agile in developing BI applications is reallu good option, i said option because there are many other options including several Software Development Methodologies, So this require a lot of work in order to prove that Agile is the best for developing BI applications
in my opinionwe first must show how to use agile in developing DSS,then for BI……
LikeLike
Hi !
We’ve used SCRUM for BI-development for 6 months now, running 3 week sprints. We had to make some small modifications in order to make it work. Overall we are delivering more value to the business in less time and with less resources.
LikeLike
Agile methodologies can constructively influence BI projects with in the three major ways you mention. There are SOME Agile concepts that directly and productively apply to BI efforts but many others are problematic.
But Wholesale adoption of Agile methodologies for BI (and anything else if you ask me) is destructive. Sprints appear to help get a project over a certain line to meet a minimum threshold – they do not promote standards, they often force comprises to extensibility and they ignore additional value, benefit and support of business objectives that could be accommodated if folks were not hung up on some arbitrary ittereation with a functional threshold.
How often have you seen someone allocate time to go back and shore up compromises that are in the “completed” work from a sprint?
LikeLike
Great post. Although few people can see the obvious benefit at this point in time, I agree Agile is the way to go for all BI projects – unless of course, you wish to spend too much and get delays with your project 😉
You may be interested in a recent post about the benefits of Agile BI (http://analytical-mind.com/2009/04/27/we-have-a-huge-budget-and-too-much-time-to-complete-our-business-intelligence-bi-project/).
LikeLike
Pingback: Agile pratices and methods applied to BI « Analytical Mind
Pingback: Agile Business Intelligence and Data Warehouse Development | BI Monkey
Forrester actually has a pretty defined set of criteria where they think Agile BI is appropriate. They are saying it does not replace traditional BI delivery methods in all instances. While I hesitate to use the word “all” in any sentence, the reality is that I have trouble seeing a scenario where traditional waterfall methods would work. Maybe you don’t need to go all-out agile with every project, but without at least doing some initial rapid prototyping, I’m not sure how you can expect to avoid those fun “requirements surprises”, massive delays, and ultimately large rework efforts.
Balanced Insight is sponsoring free reprints of the Forrester report on Agile BI if you don’t feel like handing $499 over to Forrester to read it 🙂 See: http://www.balancedinsight.com/forrester/r.php?p=big,m
LikeLike
Pingback: Kadenza+ | Blog | Scrum; de eerste meters