The Enterprise Software Project Lifecycle
Lots of companies out there are trying to build big enterprise platform systems. Lots of people get involved, requirements are gathered, plans are drawn up, and eventually you end up with an architecture diagram that looks a lot like this:
Then teams are put together to go and implement the entire solution once it has been planned out. The estimated delivery is usually years in the future but milestones are planned along the way to make sure the teams are on track. Development starts with enthusiasm and morale is high. Even months into the project, folks are excited about the possibilities and are even thinking of other things that they will do once the planned work is complete. You come into the office each morning and you see happy teams working together:
Six months go by, and one day you come into the office and notice that the enthusiasm for the project has waned a bit. Folks are still working together, solving problems, but they aren't laughing and smiling like they used to.
Some turnover has started happening, and you have had some setbacks while you try to find people to replace the ones that have left. You have done a lot of great work, but you are still years away from completing your project. You are on budget, but the economic picture has changed and the business is starting to question whether your solution is going to bring in the value that was originally predicted. You know that these kind of questions always come up though, so you get to work and carry on the fight.
Another six months goes by. You have delivered several releases, and your users are able to get some things done now. Not all of it, of course, and they are already changing some of the things that they really want. You have had to abandon parts of your original architectural design and try to incorporate new ones that weren't part of the original picture. While going between meetings, you take a look in the team room and it gives you pause as you notice that the team seems a little stressed.
You ask them what is up, and it turns out there is a major defect in production that they haven't been able to resolve. In fact, it is a problem that has been dogging the team off and on for months. Mostly it hasn't been a big deal because there are workarounds, but now the workaround isn't working for one of the users, and it has been escalated up the chain and has now become the top prioirty to fix. The team has not been able to reproduce the problem in lower environments, so it has been a slow process to try and isolate the problem.
We could follow this scenario further, but I think you get the point. What started as a project that people were excited about has turned into a project that people are just hoping to be done with so that they can move on to the next thing. This desire to wrap things up will become so strong that teams will start cutting corners and dropping features just to keep the deadlines from slipping, but some of this causes the overall quality to go down. A lot of projects that head down this road will either fail, be delivered late, or will not live up to expectations. I'd like to say that this is something that only happens occasionally, but it actually happens all the time. If this isn't the right way of building software, what is? How can we still build a large enterprise project without sacrificing quality, burning out our team, and actually keeping delivery on schedule? It might seem contradictory, but the key is to start small.
Building Big Projects From Small Pieces
There is a transformation going on in the world of software, and chances are that you have heard about it. If you aren't Agile yet and don't have any plans to become Agile, chances are you are going to have lots of projects that follow the lifecycle mentioned in the previous section. Maybe your company has better success at running projects in a waterfall fashion, but compared to a lot of companies that are using Agile, you are falling behind every day. There are companies out there who don't release software monthly, weekly, or even daily. They are releasing new software almost continuously. Customers request features that are delivered within days or weeks instead of months or years. Even more than that, they are able to keep up with changing trends and jump on new technologies almost as soon as they come out.
Does this sound impossible or unrealistic? It might. There are some key differences between what these companies are doing and what companies that fall into the death trap that hampers a lot of projects.
Deliver What Is Most Important First
One thing that often hampers projects is that there is a lot of work going on in parallel to build all the various components that will fit together to build the final solution. The problem is that before you can deliver any one part of it, you have to plan how all these components work and how they work together and how they will be released in a meaningful way. This planning involves a lot of people and it can take a long time before even the first one is started. This means the project will often stretch out for years and that users may not even have something tangible for quite some time.
One of the key aspects of Agile is that it aims to break down really large projects into smaller pieces that can be planned and delivered in shorter amounts of time. In fact, when you break it down far enough, you end up with parts that can be delivered in under 10 days. The parts delivered may not do a lot, but users will be able to see progress much sooner and start providing valuable feedback to the development team. Another benefit is that you can reorder these pieces so that the most important ones are delivered first. The pieces that fall towards the back of the backlog are things that the user doesn't really need immediately or possibly at all, and your team will be more focused on delivering what is actually needed as opposed to what is desired. The folks who plan out software projects can tend to add features that are really just nice-to-haves, and Agile forces them to prioritize things so that these items don't get in the way of the things that really are important.
One last advantage of this is that what your users consider to be most important will change over time. When teams first move to Agile, they can sometimes struggle with what is most important because most users consider what they individually want to be most important. When you prioritize requirements that come from a number of different users, you start to figure out what the real important things are because you can't possibly deliver everything at once, and so some things will win out and some will slip into the background. Users start realizing that things that they thought were important really weren't, and when they actually get functionality delivered, they start to focus on giving you feedback to get the next important thing instead of trying to think of everything they want. They start thinking smaller because they know that if they ask for small things they will get them sooner, and pretty soon you are able to start delivering to them more frequently. Some companies have staggered this so that they can deliver functionality daily or even hourly, and the sooner you get something to a user, the sooner they can give you feedback to adjust your priorities for the next thing to deliver.
It might not sound like it, but failing fast is a key aspect of Agile. A lot of projects will go through long cycles of development only to end up delivering something that really doesn't meet the requirements, isn't really needed any more, or doesn't actually fulfil the needs of the users. Projects that end up in this state are just a waste on your bottom line, and there is a lot of waste going on even for the projects that do make it to the finish line. The key thing is that you want to be able to change direction as soon as you know that you are going the wrong way so that you minimize the waste.
You may find out that you have wasted your time after a few months when funding suddenly dries up or the business changes direction, but being Agile means that you already have other work planned out so you can shift over without having to start a long waterfall process over. No matter whether you are working Agile or Waterfall, a failed project is a failed project. The difference is that with waterfall, you might have to put development teams on hold while a new set of plans are drawn up. Agile teams are continually planning things out, so chances are high that there is plenty of other work in the backlog.
There are also times when nothing has gone wrong with a project, but something happens that suddenly becomes a higher priority. With waterfall projects, dealing with rapid change is much harder. Teams have spent a lot of time planning and changes in the plan often have huge impacts on the product being delivered and when. With Agile, you can sqeeze in different pieces of work into two week periods, and this can help you minimize the impact. Teams are used to getting new work every couple of weeks, and a change in priorities isn't really that big a deal. They just see the stories that have been planned and they execute on them.
The last item I will talk about is continuous improvement. This isn't just about continually improving your product, although that definitely is part of it. What I'm actually talking about is continually improving the way you build and deliver your product. As you go through each iteration of building components, you are going to have some things go right and some things go wrong. The important thing is to keep doing the things that went well and make adjustments to fix the things that didn't. You might have to try a few different approaches, but over time you will find processes that help your teams become more productive.
Remember how the team was burning out as time went along on the project? That is one of the worst things that can happen. Your best people are not going to stick around in a project that is getting to be more work than fun. When they leave, you will have an additional problem of find the folks to replace them while still trying to keep the rest of the team motivated. Finding good talent is not hard, but finding great talent is, and you need to make sure that you are listening to your teams and addressing their needs so that they will want to keep working for you instead of someone else.
Agile embraces the concept of continous improvement and defines simple routines for ensuring that feedback is gathered at regular intervals. Sometimes these retrospectives can turn into sessions where folks are just complaining, but if you keep them focused on not only raising issues but also proposing solutions to the problems, you will find that these sessions can incredibly beneficial. You won't necessarily be able to solve every problem that comes up, but sometimes just letting teams have their say can go a long way for building better relationships among team members.
I know this is a long winded post, but Agile is one of the things that I'm quite passionate about, and it goes hand in hand with how good teams actually keep delivering quality while other teams continue to struggle just to stay afloat. It comes down to breaking down complexity and really starting small. If you do that, you can build almost anything, and you may find a quicker way there. Not to mention that you will most likely enjoy the journey a whole lot more.