Why Waterfall won’t die soon?

Current development techniques are unrecognisable from the days of punch cards and tick-a-tapes and the constraints of yester-year. Over the decades the legacy of these primitive development environments lingers on, not in technological terms but in mentality terms. The Waterfall generation has now matured and they are now running and directing the projects using what they know best – which is why it won’t die soon.

Many decades ago software development was a very time consuming and expensive undertaking. It was difficult. If you programmed a computer using punch cards then you needed to be very clear up front on your requirements. Debugging punch card systems was notoriously difficult and time consuming. Doing a dry run through your deck of punch cards could aid in reducing bugs but it couldn’t prevent badly defined requirements or misinterpretation of requirements. For this reason alone software development used to be very expensive and extra effort was required upfront to really pin down the full detail of all requirements.

Ok, so punch cards may be a bit extreme so what about COBOL or something slightly more recent such as FORTRAN? Well, even these software development languages often required developers to book processor time on mainframes to test and debug their code. In these environments, extra effort in defining everything upfront mitigated risks as the development environments were very primitive by today’s standards.

In the olden days this approach (Waterfall) was engrained into everyone who worked on the software development lifecycle, from Project Managers and developers, to end users. There were no other ways for decades.

Development environments back then were very different to what’s available today and many of the methodologies were born out of the development hurdles and technical constraints of the day.

Now contrast this with current development capabilities:

• Integrated development environments
• Interactive debugging
• Object Orientated Design
• Real-time compilation and the ability to run software on the developers PC to get instant feedback
• Continuous Integration
• Automated Testing
• Automated Deployment
• Advanced release pipelining supporting rapid deployment
• User / Developer collaboration enabling developers to develop systems WITH the end users
• Object Relational Mapping
• Design Patterns
• Feedback loops measured in minutes / hours rather than months.

So, if the landscape is so unrecognisable now from how it used to be when Waterfall was the de-facto standard then why are lots of organisations still following an approach that is completely unnecessary and in most cases counter productive?

BECAUSE – the developers of yester-year are the Project Managers of today! They haven’t moved on from when they used to develop software. I’ve heard this a lot from PM’s who say they used to be developers – usually COBOL developers?

Singling out Project Managers is unfair as it’s more of a cultural thing across many levels of the organisation following conditioning over many decades. Another big contributor is the end users. The end users of yester-year are the senior business users of today and expect detailed definitions of all requirements up-front ‘because that’s the proper way to do it isn’t it?’

I’ve noticed this to be a particular problem in the UK Public Sector where they employ individuals – usually in positions such as ‘Business Assurance’ or ‘Project Governance Officer’ – to ensure project teams are capturing all requirements up-front and ‘doing things properly’. When challenged as to why it’s important to define all requirements in detail up-front, a common response is “because it’s the process” or “it’s the proper way to do it”. For decades these individuals and those around them have been told they must define all requirements up-front and be very detailed in their spec’s to reduce risk, so for them, this is the only way to develop software.

Another common misconception is that by not doing detailed up-front requirements capture we’re trying to avoid doing documentation. The reality is we’re not avoiding the documentation. I’ve regularly found that we’re not actually reducing the amount of documentation we produce - we’re just doing different documentation. Different in that we’re using other, more efficient, mental models to capture and communicate requirements or solutions such as site maps, wireframes, design mock-ups, user journeys, etc.

So, the biggest hurdle to finally putting Waterfall to rest is letting people know that its ‘ok’ to do things differently and developing software is not as difficult as it used to be – relatively speaking that is.

Agile perceptions

I've just presented at a conference titled "Agile in the public sector". I was there to represent the NHS Information Centre to talk about our experiences - good & bad - of introducing Agile practices into our area of the public sector.

There were only a couple of similar case study presentations from other speakers who were actually doing this kind of stuff on the ground - in the public sector. The rest were what I class as Evangelists who talk about the ideals of Agile and basically regurgitate the usual Agile idealism's. Apologies to those guys if this sounds disrespectful. I'm sure all of them have had a lot of experience of introducing Agile practices but it seemed too distant and abstract from the usual day to day experiences.

During the other speaker sessions I heard phrases like 'Agile is a journey' and 'Agile is a state of mind'. ok, great it's a state of mind but what can someone in the public sector do when returning to the office tomorrow to adopt this 'state of mind'? I really don't think the Agile community helps itself by presenting Agile as a mystical black art. Yes, they are right - it is a different mind set but where can individuals start when faced with introducing new practices into their organisation?

I also felt there were a few mixed messages coming out of the session. The session was supposed to be about Agile in the public sector and yet the audience were faced with DSDM, Scrum, XP, DSDM, and DSDM. One of the audience made the point that it's confusing for those entering the Agile space to know which 'flavour' of Agile is right for them? I'm not surprised and I'm probably going to confuse matters further by saying they're all right and they're all wrong.

Study all methodologies individually and take out and use the bits that best fit the needs of your project. Don't be afraid to experiment.

Agile by Stealth

I don’t know how many other people have witnessed the scenario where someone goes to senior management clutching a bunch of books and papers on Agile development, asking if they can do Agile on the next project? The usual response from senior management (who aren’t usually tech savvy) is a disguised fob off. In most cases they don’t even know what Agile development is so find it easier to dismiss. The types of organisations I’m referring to here are usually well entrenched in the bowels of PRINCE2 methodology. And no, this isn’t a dig at PRINCE2 as it has its place, and it can coexist with Agile approaches.

Trying to release these strongholds from the shackles of PRINCE2 is extremely challenging. Many training and consultancy companies who evangelise Agile development talk about organisational transformation to enable Agile to happen. This can only work if cascaded down from the upper Escalon’s of the company and is perceived to be a big undertaking for most organisations.

If you are lucky enough to work for a forward thinking organisation that values its employee’s opinions and is willing to embrace change then go for it and adopt an Agile approach to software development. Do it properly though and bring in external help if the size of your organisation warrants it. If however, you’ve tried to introduce Agile approaches to your organisation and you feel like you’ve been fobbed off then there is another way, Agile by Stealth.


Why do people ask permission to do Agile? Agile shouldn’t be taken as an all or nothing, or a one size fits all. Agile provides you with an extensive toolbox of approaches that you need to pick and mix from to suit the needs of your project.

What I find is that if you announce in an organisation that you’re going to do Agile, you are instantly met with a lot of resistance. Why is this? Are they scared of the unknown or just resistant to change? Remember, most people don’t know what they like, they like what they know so any attempt to take them out of their comfort zone will be met with resistance.

As I said earlier, look at Agile as an extensive toolkit and pick out what’s best for your project. You don’t have to jump straight into hardcore SCRUM, XP, DSDM, RUP, or any other methodology for that matter and go for a purist approach, which is likely to fail if you do. One bit of advice I’d give though is to make sure you read up on and understand the principles of Lean Software development. With such an extensive toolkit you risk introducing a lot of unnecessary wastage to your development life cycle, so only adopt what you need.

I’ve listed below some of the practices that development teams can introduce from the Agile toolkit without the need for any management approval. These can sit very comfortably within a PRINCE2 stronghold and will drastically improve software quality and project success, even if you’re not doing Agile in the purest sense:

Automated Testing – when I suggest this to development teams the first reaction is “we already do unit testing”, but unit testing is only one part of automated testing. Unit, functional testing using products like Selenium, security testing, and performance testing are other methods of testing to automate. This blog post doesn’t attempt to go into any detail on these approaches so go fire up Google and get exploring. If you’re asking yourself “why is automated testing important” or even “what is automated testing”, then again, go look it up on Google. It’s too big a subject to cover here.

Continuous Integration – this should be one of your high priorities if you’ve got multiple developers on your project and CI can drastically reduce development time by removing the need for a final integration phase of a development cycle. It also weeds out all those annoying integration issues that usually result in an element of rework by facing up to the problems at the point they occur and still fresh in the developers mind. Doing a build upon check-in is an opportunity to run your automated suit of tests (see above) and alerting your development team when the build is broken.

Daily Build (Nightly Build) – Many projects I work on have both Continuous Integration as well as a daily build in place. The reason for having both is that some automated tests can be extensive and take quite a bit of time to execute. For example, you wouldn’t want to run a performance test which lasts an hour or so for each check-in made to your source repository. The daily build is an opportunity to make a deployment to a staging server for stakeholders to view, preferably using an automated deployment process.

Short Iterations – When defining Work Packages within the PRINCE2 framework, just define them as short Sprints but with a different label to keep it within the PRINCE2 framework. Other terms regularly used within PRINCE2 and MSP are staged deliveries.

Kanban, Burn up, and Burn Down Charts – Project Management Offices and Senior Management constantly demand progress and status reports. Quite often this produces friction between developers and management due to the burden of collecting data. A great way to stem the tide of status requests and progress reports is to set up a series of Information Radiators. The Kanban system is a great way for multiple developers on a project to self-manage and to show exactly who’s working on what and what the status is. Burn-up and burn-down charts are great tools to represent the likelihood of the current development phase coming in to land on time. These methods have the distinct advantages over traditional time recording systems that they don’t put an admin burden on developers, tend to be more accurate, and are much more affective at communicating progress.

Test Driven Development – the first mistake most development teams make with TDD is that they write unit tests for absolutely everything! Always keep Lean principles at the forefront of your mind and only write unit tests for complex or high risk areas of your code base. You can easily check for adequate code coverage by using code coverage tools such as nCover. Constantly look at ways of automating the testing of code and maximising test coverage. TDD isn’t just about writing test cases in nUnit before producing production code. It’s an approach that starts upstream with your Business Analyst. If your Business Analyst can learn to articulate requirements as a series of test cases then you’re well on your way.

Prioritisation of Requirements – One of the core principles of Agile development methodologies is the ability to respond to change and actually embrace change. This is through short iterations and re-prioritisation of requirements at each stage boundary in close consultation with the customer. This usually results in developing the high priority requirements first to a releasable state and then take stock to either add or remove requirements before commencing with the next development iteration. Adopting a phased or staged approach to your project will open up the opportunity to revisit the requirements and priorities at the end of each stage. Most PRINCE2 projects that I’ve worked on already have a pragmatic phased delivery plan, but quickly lose agility when constrained by Change Control Boards and bloated governance. This is probably one of the more common areas of incompatibility between PRINCE2 and Agile. If you are in a position to establish and write your projects’ change control procedure then this is easy to fix. ;)

Communication & close collaboration – having your key users or stakeholders working with your development team is another pillar of Agile. Good communication and close stakeholder management is a pre-requisite for any methodology. I always set up a staging server on every project and make sure the automated daily build deploys a working piece of software to the staging server so that all stakeholders can see the work in progress. This results in no surprises at the end of the development phase and ensures expectations are managed daily throughout the phase. If you’re not as brave as this (it does require a lot of caveats) then you should aim to review the software as its being built at least once a week with your key users and stakeholders.

At the end of the day, delivering software is not easy and there’s no magic formula. No individual methodology can guarantee you success so it’s important to increase the size of your toolkit by learning about other approaches to software development, enabling you to adapt to the needs and circumstances of individual projects.

The key message is this, no where in the PRINCE2 or MSP manual does it say “you MUST NOT use any other methodology, approach or tool with PRINCE2”. So, stop asking to ‘do agile’ and get on with it.

Time Recording and Wooden Dollars

Let’s look at a typical project in an agency / software house that has adopted time recording. During this project a number of things happen:

• The developers are constantly pulled off the project to support legacy apps, attend company meetings, and tend to various admin tasks. Only the time they actually spend on the project is recorded on the timesheet. All other tasks are booked to non-project codes.
• Its winter time and a lot of time is lost due to high absence and sickness levels within the team (snivel). Again, only the time they work on the project is booked to it.
• There are performance issues with a developer, resulting in quality issues and slow delivery. The PM makes a case with the project office that it would be unfair to have all this developers time booked to the project as he hadn’t received “value for money”. The PM wins the case and not all of the time spent on the project is booked to it.
• Part of the project (as sold by the sales team) involves a developer getting up to speed with a new piece of technology which is critical to the delivery of the project. It could be considered R&D. The PM instructs the developer not to book the time spent on this task to the project as its “Training” and should be booked to the general company training code.

The project is delivered late, BUT on paper the project is shown to have been delivered on budget and made a profit. The only problem is, in all of the cases above, the company is still absorbing costs so the fact that the time isn’t booked to the project means the project is dealing in wooden dollars. As this project is deemed to have been a financial success, this pattern is repeated on future projects. Scale this up to companies running many projects concurrently and this issue is hard to detect but massively amplified. Eventually, the company goes bust. A list of specific companies this has happened to and the evidence to back it up is available upon request.

Time recording is a necessary evil in most software houses or agencies. A public sector client of mine is currently debating the need to introduce a time recording system to help with managing 60+ concurrent projects. I’ve come across the time recording debate many times and there isn’t a simple answer to whether you should or shouldn’t do time recording. The key thing to understand and be clear on is your purpose for introducing time recording to your organisation. Here are some reasons for doing time recording that I've heard some organisations use:

• We want a way to quantify and record the amount of effort that goes into a specific task so we can improve future estimates. (excellent)
• We want to accurately measure the P&L of individual projects. (difficult)
• We want to use time recording to track progress. (valid but a very poor way to measure progress)
• We need time recording to maximise staff utilisation. (very, VERY bad. See below)
• We bill our clients on a time and materials basis so the time we enter goes straight into an invoice for the client. (good work if you can get it)
• Time recording will assist us with resource planning. (possible, but better tools available to do this)

Introducing time recording to your organisation will have a huge cultural impact. Below I’ve listed some of the good and bad points about time recording that should help you in forming a judgement on whether it’s right for your organisation:

Bad

- You have to police it resulting in staff feeling a squeeze from two different angles:

• PM’s police what time is booked to their projects and fight like mad to stop time being booked to protect the budget. They have become very creative in their methods!
• Line managers want to maximise the amount of billable time (wooden dollars) booked to projects and resist time being booked to non-project codes such as Training, Admin, Induction, Team Briefings, etc. (see utilisation below)

- Line managers waste a lot of time chasing individuals to fill in their timesheets that could be better utilised elsewhere.

- It requires a lot of discipline from staff to update their timesheet daily. Daily updates keep the data more accurate and timely. If you have staff updating timesheets weekly then it’s highly unlikely that the data will be accurate.

- You may end up employing someone just to manage and police the time recording system.
Burden – time recording introduces a huge burden to your organisation and those who work within that environment. Any additional burden kills morale. If you constantly hound staff to complete timesheets then you will destroy their morale.

- If there’s a lot of resistance to time recording you may find some staff purposely incorrectly book time to undermine the validity of the system.

- Inaccurate filing and recording of time:

• Some time recording systems I’ve seen don’t allow you to enter more than 37.5hrs per week. So how do you record overtime? This is a great way for some unscrupulous project managers to hide project overruns.
• I’ve seen PM’s entering time on behalf of their team and miraculously making the figures fit.
• Most people just make the time fit into what they feel are appropriate allocations they can get away with. They try booking it all to projects before having to justify why they’ve booked it to non-project codes.

I’ve even come across PM’s who have stopped people from working on their project because “we don’t have the budget for you”. In these specific cases, the people offering help had no other work to do and could add real value. The real cost to the organisation would be the same as they’re paying for them anyway. The project would have been delivered earlier. The only cost to the project would have been in wooden dollars as the time recording system would report the project being over budget after they had recorded their time against it.

- Many KPIs (e.g. utilisation) within organisations are calculated using data from time recording systems. Given all of the above, this results in management decisions being made on wildly inaccurate data.

Good

• Split charging of time for shared resources. If a developer is pulled of your project to resolve a support issue for a previous project then it’s useful to bill the time accordingly.
• Time recording can help in measuring how long specific tasks take when used in the context of historical analysis and estimation of future work.
• Time recording, if used correctly, has the potential to separate effort from schedule (requires a lot of discipline). In other words, if a task took 4 hours of effort but the elapsed time was 2 days then both of these metrics are valuable. This difference between effort and schedule is extremely common and yet rarely recorded. The elapsed time is usually recorded which means the project accounts for team meetings, sickness, or any other issues not normally accounted for in project estimates.
• Time recording generates data. Data is great for producing lots of reports and pretty looking management dashboards.
• It’s very easy to measure staff utilisation with a time recording system…..or so it seems.

Utilisation

Most managers are trained and actively encouraged to maximise the utilisation of their staff. It makes sense right, as you don’t want staff sat around with nothing to do? It makes perfect sense that the more value we extract from our staff maximises the bottom line? Managers are often given a big fat pat on the back for reaching company utilisation targets, regularly in excess of 70%. Something that seems so obviously good actually has the direct opposite effect.

Maximising utilisation is bad. Just in case you didn’t get that, MAXIMISING UTILISATION IS VERY BAD.

To survive and be profitable in a software house the key is throughput. The more software you can get out of the door the more you can invoice clients. This is particularly relevant to fixed price contracts. I’ve come across very few outsourced software development projects charged on a T&M basis. The more you maximise utilisation the less chance you’ll have of having the right resource available when you need it. This incurs delays to the project or additional costs in bringing in contract staff or freelancers to cover. Many agencies in this situation experience the ripple effect where they start to borrow resources from other projects to get a project closed and invoiced. This cascades through all other projects. I’m not saying you should over resource so you can throw more bodies at a project, what I’m saying is you should over resource so you can improve your chances of having the right people available at the right time.

Maximising utilisation impacts heavily on staff morale. Staff feel pressured to book all time to projects and not to morale boosting activities such as personal development, training, team meetings, etc. It stifles creativity, communication, and innovation.

There’s an excellent book called Slack which covers the subject of utilisation in much more detail. The analogies it provides, which I quote far too often are:

• If you run a motorway network at 100% - you get traffic jams
• If you run a network connection at 100% utilisation – everything slows down
• If you maximise utilisation within your agency – you print less invoices and go bust! (ok, this last quote is mine)

I’m sure there are many project accountants out there who may pick fault with what I’ve written here and quote various accounting theories and approaches. I genuinely welcome their comment along with others as this has proven to be a highly contentious issue within organisations. Feel free to comment.

www.openPDP.com

What can you do with a daily commute of 1hr 20mins on a train? Well, initially I read many books, listened to technology podcasts, and talked to some very interesting strangers (people avoid making eye contact with me now for some reason?). I even downloaded a Commodore 64 emulator and burnt many hours of nostalgic game play. I even did a bit of peeking and poking to produce a sprite bouncing around the screen. Once I'd downloaded a macro assembler I knew I'd hit rock bottom and needed to do something more constructive with my time.

So this is what I did. I decided to knock up a website that would assist individuals in their personal/professional development. 

I've been frustrated over the years with the lack of investment organisations make in their staff - particularly developers. This is normally due to time and resource constraints on the required breathing space needed to set a framework up. 

www.openPDP.com aims to directly address a number of issues facing individuals and organisations. As an individual you can explore an industry wide source of development objectives. As an organisation you can mentor your staff in a simple, structured, manor - with zero entry costs.

openPDP is completely free to use. Its there to provide a mentoring framework for anyone who takes the initiative in their personal development. If you have an active interest in your own development then you don't have to wait for your boss or organisation to get started. Simply log on to openPDP and get cracking!

To keep this post inline with the theme of this blog, here's some metrics on the solution:

• Mixture of C# and VB due to cut & paste off Google (am I a bad developer?)
  
• Source lines of code = 946
  
• Max. cyclomatic complexity = 6
  
• Max. depth = 8
  
• Train journeys from Warrington to Leeds = 10
  
• Train journeys from Leeds to Warrington = 10
  
• Test coverage = 52%
  

Total cost of development = £17.62. The 17 quid was to cover the cost of the domain name.

www.openPDP.com

Basic Software Metrics

I have a confession. I’ve been hiding a secret for many years now… I’m obsessed with software metrics!

I’ve been quietly collecting software metrics from many different project teams and companies and I get really excited when I get to run my tools on new code bases. Even basic code growth graphs tell a story of project woes. I’ve managed to generate code growth profiles for really successful projects and compare these to profiles for projects that can only be described as horrific. It’s very easy to spot a project that’s been through turmoil just from the code growth graph.

The following graph shows the comparison of various projects from different organisations. When you contrast Project 2 with Project 3 you can clearly see that the team working on Project 2 are clearly thrashing around and struggling to deliver anything.


The reason for the disastrous Project 2 was nothing to do with the development teams’ abilities - it was down to boardroom politics!

Project 3 was the most successful project I’ve ever come across. The biggest surprise is a distributed development team, half in the UK and half in India delivered it.

The number of software houses who don’t collect software metrics, or don’t even know what software metrics are or what they can be used for staggers me. Once you get into it, it seems like an obvious thing to do and yet not many people do it.

Often I’ll ask a developer how big their code base is in Source Lines Of Code (SLOC). In most cases the response is “I have no idea”. Try it for yourself and see their response when you tell them how big their project is. I recently asked a developer on a project and they estimated about 20,000 SLOC. The actual SLOC count was 123,000. This developer had been working with the code base for over a year!

Software metrics are useful for a lot of things but I predominantly use them for two things, firstly for estimation, and secondly for progress tracking.

Metrics for Estimation

Historical project data is critical for good estimation. Whole books have been dedicated to software estimation and it would be impossible for me to do the subject any justice in a short article such as this, so I’ll just give you a flavour.

Imagine the scenario where a customer gives you a set of requirements. In most cases, a seasoned software house would probably have done a project like this before. Wouldn’t it be nice if you could simply look down a list of past projects and find a few that seem to be similar. From this shortlist you can then see exactly how big the code bases were, how big a team worked on it, how much effort was required, and how long it took to complete. Is it possible then that given similar requirements your new project will take just as long?

People have said to me in the past that they can remember how long projects take and use their own “gut feel” to estimate projects. It’s amazing how quickly people forget about the bad points in a past project and can screen them out. You can’t argue with the data though.

Metrics for Progress Tracking

Developers are often consumed in backend coding and as such can’t demonstrate tangible progress to project managers by way of working software. The usual way for project managers to measure progress in these situations is to take the developers pessimistic appraisal of how complete the work package is. Project Managers can still measure accurate progress (or lack of) by looking at the code growth graph for the project.

Code diff metrics as well as code growth metrics are important in seeing progress.

Code growth metrics are useful in that you can sometimes trend the development velocity against your delivery dates and predict the final code size. Correlate this with code base sizes from historical project data and you are in with a good chance of accurately measuring progress.

You can also see when the growth profile flat-lines which is usually an indication that the developers have run into a problem such as interpretation of requirements or short changed upstream activities.

Code Diff metrics are probably more important as you can see when the developers are consumed in refractoring activities. Refactoring is generally a good thing but when your team are spending most of their time changing existing lines of code, with no code growth, and no tangible progress, then it means one of two things:

1) Your developers are engaged in re-work, fixing major floors in the design of the application, or

2) Your developers are in the stabilisation phase of your project lifecycle.

You will certainly know which one of the above you fit into by simply looking at the software on screen and forming your own judgement on functional completeness.

This is just the tip of the iceberg on the subject of software metrics. Once you start to research this subject you’ll discover just how big a subject it is. I’d recommend you start with some real simple metrics such as Code growth and code diff. It’s really easy to set up and pays massive dividends.

The book is nearly finished!

Apologies for not posting in a while. I've been busy writing a book! I didn't mean to write a book, but I suddenly realised I'd built up a huge amount of potential book material over the past decade and finally decided to put it into some kind of structure. It's not the kind of stuff you'd write in a blog so hence not publishing it here. I've got about another month to go until the first draft is ready. If you'd like to review it for me then please email me on ian@solutioneers.co.uk.

There is one warning though. If you've worked for me as a developer then there's a good chance you feature in the book so be warned. I don't hold back!