Ever since the first software development methodology was published people have been searching for a silver bullet that will make software projects more engineering practice or science than unpredictable art. The proponents of Waterfall like methodologies focus on planning and documenting the entire product set of requirements up front assuming that subject matter experts will be able to accurately estimate the effort required to develop the system. Waterfall methodology works well when you have a very well defined requirement set, stable budget and a market that is slow to change. When any of those factors are not present it becomes increasingly difficult to meet customer expectations and almost impossible to deliver business value prior to completion of the entire effort. In my experience clients who believe that they have a well-defined set of requirements often end up redefining scope or changing schedule as important business features are discovered. When this occurs project success criteria should change but business stakeholders are often reluctant to acknowledge that these new facts and business priorities have changed what can possibly be delivered given the cost and schedule constraints imposed by the original plan. No stakeholder that I have ever met wants to go back to the their management to ask for more project funding or to the project steering committee to have scope changes approved.
But XP, Lean and SCRUM are not a silver bullet either. One common observation in large organizations adopting Agile/SCRUM is that the development teams have an easier time making the leap than the business stakeholders. The finance and governance people want a specific feature set in a specific time period for a specific price. The business stakeholders want to dictate a feature set that is all high priority, and then go away and possibly view demonstrations at some time interval. The common misconception is that with SCRUM you do not perform detailed planning and the team just codes the functionality described by the user stories.
The truth is that business stakeholders have to be very engaged in this process and that SCRUM employs planning and estimating on an iterative basis balancing the planning effort with the knowledge that it will continually be revisited through the course of the project. The product owner owns these decisions that need to be made and revisited periodically. The business culture will evolve from a schedule-driven to business value driven focus. Mike Cohn in his book, “Agile Estimating and Planning” describes agile planning as being more focused on planning than on the plan. Agile planning encourages change, results in plans that are easily changed and the planning effort is spread throughout the project. He goes on to state that the reason plans fail is that they focus on activities instead of features. The business receives little if any value from the completion of activities.
It is not until the product owner is able to cross the Waterfall//Agile chasm that the team will be able to begin to realize the benefits of a truly agile process. The product owner will drive the focus on features and provide clarity to the priority of these features. This is the first small step needed to begin to adopt SCRUM and start your organization on the path to becoming more Agile.