Agile by Stealth

I don’t know how many other people have witnessed the scenario where someone goes to senior management clutching a bunch of books and papers on Agile development, asking if they can do Agile on the next project? The usual response from senior management (who aren’t usually tech savvy) is a disguised fob off. In most cases they don’t even know what Agile development is so find it easier to dismiss. The types of organisations I’m referring to here are usually well entrenched in the bowels of PRINCE2 methodology. And no, this isn’t a dig at PRINCE2 as it has its place, and it can coexist with Agile approaches.

Trying to release these strongholds from the shackles of PRINCE2 is extremely challenging. Many training and consultancy companies who evangelise Agile development talk about organisational transformation to enable Agile to happen. This can only work if cascaded down from the upper Escalon’s of the company and is perceived to be a big undertaking for most organisations.

If you are lucky enough to work for a forward thinking organisation that values its employee’s opinions and is willing to embrace change then go for it and adopt an Agile approach to software development. Do it properly though and bring in external help if the size of your organisation warrants it. If however, you’ve tried to introduce Agile approaches to your organisation and you feel like you’ve been fobbed off then there is another way, Agile by Stealth.


Why do people ask permission to do Agile? Agile shouldn’t be taken as an all or nothing, or a one size fits all. Agile provides you with an extensive toolbox of approaches that you need to pick and mix from to suit the needs of your project.

What I find is that if you announce in an organisation that you’re going to do Agile, you are instantly met with a lot of resistance. Why is this? Are they scared of the unknown or just resistant to change? Remember, most people don’t know what they like, they like what they know so any attempt to take them out of their comfort zone will be met with resistance.

As I said earlier, look at Agile as an extensive toolkit and pick out what’s best for your project. You don’t have to jump straight into hardcore SCRUM, XP, DSDM, RUP, or any other methodology for that matter and go for a purist approach, which is likely to fail if you do. One bit of advice I’d give though is to make sure you read up on and understand the principles of Lean Software development. With such an extensive toolkit you risk introducing a lot of unnecessary wastage to your development life cycle, so only adopt what you need.

I’ve listed below some of the practices that development teams can introduce from the Agile toolkit without the need for any management approval. These can sit very comfortably within a PRINCE2 stronghold and will drastically improve software quality and project success, even if you’re not doing Agile in the purest sense:

Automated Testing – when I suggest this to development teams the first reaction is “we already do unit testing”, but unit testing is only one part of automated testing. Unit, functional testing using products like Selenium, security testing, and performance testing are other methods of testing to automate. This blog post doesn’t attempt to go into any detail on these approaches so go fire up Google and get exploring. If you’re asking yourself “why is automated testing important” or even “what is automated testing”, then again, go look it up on Google. It’s too big a subject to cover here.

Continuous Integration – this should be one of your high priorities if you’ve got multiple developers on your project and CI can drastically reduce development time by removing the need for a final integration phase of a development cycle. It also weeds out all those annoying integration issues that usually result in an element of rework by facing up to the problems at the point they occur and still fresh in the developers mind. Doing a build upon check-in is an opportunity to run your automated suit of tests (see above) and alerting your development team when the build is broken.

Daily Build (Nightly Build) – Many projects I work on have both Continuous Integration as well as a daily build in place. The reason for having both is that some automated tests can be extensive and take quite a bit of time to execute. For example, you wouldn’t want to run a performance test which lasts an hour or so for each check-in made to your source repository. The daily build is an opportunity to make a deployment to a staging server for stakeholders to view, preferably using an automated deployment process.

Short Iterations – When defining Work Packages within the PRINCE2 framework, just define them as short Sprints but with a different label to keep it within the PRINCE2 framework. Other terms regularly used within PRINCE2 and MSP are staged deliveries.

Kanban, Burn up, and Burn Down Charts – Project Management Offices and Senior Management constantly demand progress and status reports. Quite often this produces friction between developers and management due to the burden of collecting data. A great way to stem the tide of status requests and progress reports is to set up a series of Information Radiators. The Kanban system is a great way for multiple developers on a project to self-manage and to show exactly who’s working on what and what the status is. Burn-up and burn-down charts are great tools to represent the likelihood of the current development phase coming in to land on time. These methods have the distinct advantages over traditional time recording systems that they don’t put an admin burden on developers, tend to be more accurate, and are much more affective at communicating progress.

Test Driven Development – the first mistake most development teams make with TDD is that they write unit tests for absolutely everything! Always keep Lean principles at the forefront of your mind and only write unit tests for complex or high risk areas of your code base. You can easily check for adequate code coverage by using code coverage tools such as nCover. Constantly look at ways of automating the testing of code and maximising test coverage. TDD isn’t just about writing test cases in nUnit before producing production code. It’s an approach that starts upstream with your Business Analyst. If your Business Analyst can learn to articulate requirements as a series of test cases then you’re well on your way.

Prioritisation of Requirements – One of the core principles of Agile development methodologies is the ability to respond to change and actually embrace change. This is through short iterations and re-prioritisation of requirements at each stage boundary in close consultation with the customer. This usually results in developing the high priority requirements first to a releasable state and then take stock to either add or remove requirements before commencing with the next development iteration. Adopting a phased or staged approach to your project will open up the opportunity to revisit the requirements and priorities at the end of each stage. Most PRINCE2 projects that I’ve worked on already have a pragmatic phased delivery plan, but quickly lose agility when constrained by Change Control Boards and bloated governance. This is probably one of the more common areas of incompatibility between PRINCE2 and Agile. If you are in a position to establish and write your projects’ change control procedure then this is easy to fix. ;)

Communication & close collaboration – having your key users or stakeholders working with your development team is another pillar of Agile. Good communication and close stakeholder management is a pre-requisite for any methodology. I always set up a staging server on every project and make sure the automated daily build deploys a working piece of software to the staging server so that all stakeholders can see the work in progress. This results in no surprises at the end of the development phase and ensures expectations are managed daily throughout the phase. If you’re not as brave as this (it does require a lot of caveats) then you should aim to review the software as its being built at least once a week with your key users and stakeholders.

At the end of the day, delivering software is not easy and there’s no magic formula. No individual methodology can guarantee you success so it’s important to increase the size of your toolkit by learning about other approaches to software development, enabling you to adapt to the needs and circumstances of individual projects.

The key message is this, no where in the PRINCE2 or MSP manual does it say “you MUST NOT use any other methodology, approach or tool with PRINCE2”. So, stop asking to ‘do agile’ and get on with it.

Time Recording and Wooden Dollars

Let’s look at a typical project in an agency / software house that has adopted time recording. During this project a number of things happen:

• The developers are constantly pulled off the project to support legacy apps, attend company meetings, and tend to various admin tasks. Only the time they actually spend on the project is recorded on the timesheet. All other tasks are booked to non-project codes.
• Its winter time and a lot of time is lost due to high absence and sickness levels within the team (snivel). Again, only the time they work on the project is booked to it.
• There are performance issues with a developer, resulting in quality issues and slow delivery. The PM makes a case with the project office that it would be unfair to have all this developers time booked to the project as he hadn’t received “value for money”. The PM wins the case and not all of the time spent on the project is booked to it.
• Part of the project (as sold by the sales team) involves a developer getting up to speed with a new piece of technology which is critical to the delivery of the project. It could be considered R&D. The PM instructs the developer not to book the time spent on this task to the project as its “Training” and should be booked to the general company training code.

The project is delivered late, BUT on paper the project is shown to have been delivered on budget and made a profit. The only problem is, in all of the cases above, the company is still absorbing costs so the fact that the time isn’t booked to the project means the project is dealing in wooden dollars. As this project is deemed to have been a financial success, this pattern is repeated on future projects. Scale this up to companies running many projects concurrently and this issue is hard to detect but massively amplified. Eventually, the company goes bust. A list of specific companies this has happened to and the evidence to back it up is available upon request.

Time recording is a necessary evil in most software houses or agencies. A public sector client of mine is currently debating the need to introduce a time recording system to help with managing 60+ concurrent projects. I’ve come across the time recording debate many times and there isn’t a simple answer to whether you should or shouldn’t do time recording. The key thing to understand and be clear on is your purpose for introducing time recording to your organisation. Here are some reasons for doing time recording that I've heard some organisations use:

• We want a way to quantify and record the amount of effort that goes into a specific task so we can improve future estimates. (excellent)
• We want to accurately measure the P&L of individual projects. (difficult)
• We want to use time recording to track progress. (valid but a very poor way to measure progress)
• We need time recording to maximise staff utilisation. (very, VERY bad. See below)
• We bill our clients on a time and materials basis so the time we enter goes straight into an invoice for the client. (good work if you can get it)
• Time recording will assist us with resource planning. (possible, but better tools available to do this)

Introducing time recording to your organisation will have a huge cultural impact. Below I’ve listed some of the good and bad points about time recording that should help you in forming a judgement on whether it’s right for your organisation:

Bad

- You have to police it resulting in staff feeling a squeeze from two different angles:

• PM’s police what time is booked to their projects and fight like mad to stop time being booked to protect the budget. They have become very creative in their methods!
• Line managers want to maximise the amount of billable time (wooden dollars) booked to projects and resist time being booked to non-project codes such as Training, Admin, Induction, Team Briefings, etc. (see utilisation below)

- Line managers waste a lot of time chasing individuals to fill in their timesheets that could be better utilised elsewhere.

- It requires a lot of discipline from staff to update their timesheet daily. Daily updates keep the data more accurate and timely. If you have staff updating timesheets weekly then it’s highly unlikely that the data will be accurate.

- You may end up employing someone just to manage and police the time recording system.
Burden – time recording introduces a huge burden to your organisation and those who work within that environment. Any additional burden kills morale. If you constantly hound staff to complete timesheets then you will destroy their morale.

- If there’s a lot of resistance to time recording you may find some staff purposely incorrectly book time to undermine the validity of the system.

- Inaccurate filing and recording of time:

• Some time recording systems I’ve seen don’t allow you to enter more than 37.5hrs per week. So how do you record overtime? This is a great way for some unscrupulous project managers to hide project overruns.
• I’ve seen PM’s entering time on behalf of their team and miraculously making the figures fit.
• Most people just make the time fit into what they feel are appropriate allocations they can get away with. They try booking it all to projects before having to justify why they’ve booked it to non-project codes.

I’ve even come across PM’s who have stopped people from working on their project because “we don’t have the budget for you”. In these specific cases, the people offering help had no other work to do and could add real value. The real cost to the organisation would be the same as they’re paying for them anyway. The project would have been delivered earlier. The only cost to the project would have been in wooden dollars as the time recording system would report the project being over budget after they had recorded their time against it.

- Many KPIs (e.g. utilisation) within organisations are calculated using data from time recording systems. Given all of the above, this results in management decisions being made on wildly inaccurate data.

Good

• Split charging of time for shared resources. If a developer is pulled of your project to resolve a support issue for a previous project then it’s useful to bill the time accordingly.
• Time recording can help in measuring how long specific tasks take when used in the context of historical analysis and estimation of future work.
• Time recording, if used correctly, has the potential to separate effort from schedule (requires a lot of discipline). In other words, if a task took 4 hours of effort but the elapsed time was 2 days then both of these metrics are valuable. This difference between effort and schedule is extremely common and yet rarely recorded. The elapsed time is usually recorded which means the project accounts for team meetings, sickness, or any other issues not normally accounted for in project estimates.
• Time recording generates data. Data is great for producing lots of reports and pretty looking management dashboards.
• It’s very easy to measure staff utilisation with a time recording system…..or so it seems.

Utilisation

Most managers are trained and actively encouraged to maximise the utilisation of their staff. It makes sense right, as you don’t want staff sat around with nothing to do? It makes perfect sense that the more value we extract from our staff maximises the bottom line? Managers are often given a big fat pat on the back for reaching company utilisation targets, regularly in excess of 70%. Something that seems so obviously good actually has the direct opposite effect.

Maximising utilisation is bad. Just in case you didn’t get that, MAXIMISING UTILISATION IS VERY BAD.

To survive and be profitable in a software house the key is throughput. The more software you can get out of the door the more you can invoice clients. This is particularly relevant to fixed price contracts. I’ve come across very few outsourced software development projects charged on a T&M basis. The more you maximise utilisation the less chance you’ll have of having the right resource available when you need it. This incurs delays to the project or additional costs in bringing in contract staff or freelancers to cover. Many agencies in this situation experience the ripple effect where they start to borrow resources from other projects to get a project closed and invoiced. This cascades through all other projects. I’m not saying you should over resource so you can throw more bodies at a project, what I’m saying is you should over resource so you can improve your chances of having the right people available at the right time.

Maximising utilisation impacts heavily on staff morale. Staff feel pressured to book all time to projects and not to morale boosting activities such as personal development, training, team meetings, etc. It stifles creativity, communication, and innovation.

There’s an excellent book called Slack which covers the subject of utilisation in much more detail. The analogies it provides, which I quote far too often are:

• If you run a motorway network at 100% - you get traffic jams
• If you run a network connection at 100% utilisation – everything slows down
• If you maximise utilisation within your agency – you print less invoices and go bust! (ok, this last quote is mine)

I’m sure there are many project accountants out there who may pick fault with what I’ve written here and quote various accounting theories and approaches. I genuinely welcome their comment along with others as this has proven to be a highly contentious issue within organisations. Feel free to comment.

Software Development Resources

I finally got around to pulling together many of the documents, templates, and tools I use on a regular basis into a static resource page. You can view this list of resources by clicking the following link:

Software Development Resources Page

I've got a lot more to add over the coming weeks but hopefully this will be a big enough list to be useful.

Flow Time

ok, so Flow Time isn't a new concept. It's been around for a very long time now. First published in 1987, Peopleware by Timothy Lister & Tom DeMarco completely changed my approach to managing software developers.

Yet, Flow Time does seem to be a new concept to many development managers I speak to (and hence the reason for this post). As soon as I tell them what Flow Time is they instantly get it. It's not a difficult concept to grasp and most of them admit to interrupting developer Flow Time regularly.

If you take a look around your development team - probably located in an open plan office - count how many of them are wearing headphones. A large percentage no doubt. What they're doing is getting into Flow Time.

So if you still don't know what Flow Time is it's really simple. Have you ever been happily coding away only to realise that you've completely forgotten about the passage of time and it's now way later than you thought? Congratulations, you've experienced Flow Time. In Peopleware, the authors say it takes a minimum of 15 minutes to get into a Flow but takes seconds to come out of it.

This means that every time you as a manager stops by a developers desk to "get a progress update", or anyone else telephones, emails, or interrupts them in any way, you're actually knocking the hell out of productivity. For every interruption it'll take them at least 15 minutes to get back to being productive.

As a development manager I tell all of my developers to respect the Flow Time of others, i.e. if the headphones are on, do not disturb. I keep all team and project meetings to the beginning or end of the day. I encourage my team to check their emails at times to maximise Flow Time, i.e. beginning/middle/end of day. I tell them it's ok to turn on voicemail and respond at the end of the day. I also don't agree with hunt groups on phone systems or developers being interrupted to pickup someone else's phone.

Go and buy a copy of Peopleware and when you've read it, sit back and observe your development team for a short period of time. I can guarantee you will make changes.

Home-shoring, Near-shoring, and Off-shoring - tips for technical leads with distributed development teams

First of all, a bit of definition - I define home-shoring as developers who are probably freelancers working from home as part of a geographically distributed team. Near-shoring is developers who are within a couple of hours of your timezone, and off-shoring is developers who are further afield. This blog post is NOT about Outsourcing.

I've been managing and delivering software projects successfully using distributed teams for 5 years now. I've managed distributed teams from UK, Romania, North India (Deli), South India (Trevandum), and China (Beijing) on various types and sizes of projects. On the flip-side, I've been a developer in distributed teams on several projects, working remotely from my home in the UK and one project in particular where it was being run from Spain. With this experience I offer the following tips to other technical leads who may already be working with, or considering working with distributed teams from home or abroad.

• Make sure all team members can contact each other at all times. As a manager I insist all team members appear online and can be contacted instantly via VOIP. Very invasive yes, but I can walk up to team members if they're based in my office location so it shouldn't be any different for remote workers.
• Use Skype or an Instant Messenger application to see when other team members are online. Verbal communication is very important, don't let language barriers stop you from speaking.
• It's very unwise (or even suicidal) to attempt a project using a distributed team without adequate source control. One other tip whilst on the subject of source control - locate the source repository closer to the physical location of the majority of developers. I saw a company once with the source repository in the UK but all their developers were in India?!
• Use continuous integration - along with automated testing as much as possible. Ensure all developers are on the email list for when the build breaks.
• Ensure all developers check in code regularly.
• Invest some time at project inception on training all the team on how to work together effectively. Show them the desktop sharing tools, show them how to use Skype properly, make sure they understand the source control system and the automated build. You can do this easily using Webex or other conferencing / collaboration tools.
• If you can, realign the working hours of the developers to synchronise the timezones. In other words, get the on-shore team to work early's and the off-shore team to work lates. It's important to maximise the amount of true collaborative working time to ease communication.
• Give all team members a proper brief. See my earlier post on how to brief developers properly. Once you've written a work package, send it to the remote developer. Give them at least half a day to digest it. The next step is to do a desktop share and verbally step through the work package with them, line by line. During this desktop share session you'll flush out most of the questions and answers. Once they start development on the work package, make sure you review their progress every day via desktop share, reviewing their code and looking at the working software on their PC.
• Here's some tools that I use with remote teams - Lotus Sametime Unyte, Webex, Skype. Actually, one of the coolest tools I've seen recently is MPK20 from Sun Microsystems. I'm not sure how effective it will be for remote teams but I can't wait to try it out on my next project!
• If possible, bring the off-shore developers over to spend the first two weeks of the project on-shore working along side other project team members. This will serve as an excellent induction to the project which they can then take back with them.
• If you have a significant number of off-shore developers on your project, bring one of them over to your offices to act as technical liaison for the duration of the project.

I'm very keen to hear other peoples experiences of working with remote teams both as developers and as managers. Please post any comments on this topic so others can share your experiences.

Continuous Transparency - a new best practice?

Continuous Transparency is the practice of giving your customer or key stakeholders direct access to the output of your daily build or continuous integration environment, in other words, your customer is able to see the work in progress (warts and all).

For web applications you simply get your build script to automatically deploy the build to an internet facing server and send the URL to your customer. For thick client applications you give VNC or RDC access to a demo PC, usually located in your corporate DMZ, and send your customer the connection details.

When you adopt this practice for the first time you must communicate to all stakeholders the following in relation to having direct access to the build:

• What you will see in the build is work in progress. It is normal for untested software that is in mid-development to crash or give unexpected results. Please do not log any calls or be alarmed if the software crashes or is partially unavailable or completely unavailable when you try to access it.
• You must not use the build for demonstration purposes within your organisation as it may change significantly before the final release and may not be available when you come to demonstrate it. You also run the risk of setting the wrong expectations should the software fail or change in the future.
• Placeholders will often be used where content or functionality is not yet available to provide “scaffold” for the application.
• What you see in the daily build does NOT constitute an official release.
• All previous data is removed each time the build is deployed.

Advantages of Continuous Transparency

• Builds an enormous amount of confidence with the customer and creates a very strong relationship as they can see tangible progress and feel constantly engaged.
• Enables you to share problems with the customer in real-time. The most common problem is having an overly optimistic (unachievable) schedule. The customer can usually gauge the overall progress of the development and realise that you can’t complete all functionality on time. They actively help you to de-scope the solution to hit their deadline.
• Focuses the developers’ minds when they know all stakeholders are watching their creations on a daily basis and improves quality upstream.
• Entirely removes the risk of any surprises to the customer when the first release is made, where they would have traditionally only seen the software after many months of development.
• Flushes out change requests or fundamental requirements issues earlier on in the development life cycle. Better to find out earlier before project budget is burnt.
  

Disadvantages of Continuous Transparency

• If your project has major resourcing issues (I’m not talking about the odd day of sickness or annual leave) then your customer will be able to see an unexplainable lack of progress.

I have operated Continuous Transparency on all projects I’ve run over the past 4 years and it’s always been a great success.

Everyone I’ve introduced it to automatically gives me a look of horror followed by "showing the customer untested software or airing our bugs in public is ridiculous", or "but doesn’t it give the customer a chance to sneak in changes?" But once they try it and reap the rewards of success they say they will never go back to how they used to do software. Most importantly though, is all my customers have said “why don’t all our suppliers operate like this.”

I’m interested to hear comments from anyone who’s already operating this approach or is concerned or sceptical about using it.

Briefing Developers

For some time now, I’ve been meaning to write a post on “briefing developers properly”. The difficulty I’ve had is articulating it in such a way that it provides some real tangible advice that can be of immediate use to Technical Leads and Project Managers. The more I think about it, the more I realise I could write an entire book on the subject. There isn’t simply a one-size fits all Developer Briefing Template.doc that you simply fill in the blanks.

Many attributes affect which approach you use to brief a developer. Project type, project size, team size, and schedule are all attributes you need to think about when considering the most appropriate approach. For example, a small Flash based multimedia project would probably be best suited to a simple story board. A web site should have a site map as a minimum. A web application could require a more detailed work package incorporating UML.

What ever approach you take here’s some key points to consider when writing a brief:

• A verbal brief isn’t a brief. It must be in some kind of a written form.


• Piecing together the brief from a large email thread is the wrong type of written brief!


• Complementing the written brief with verbal communication is acceptable.


• Get the developer to play back the brief to you to give you the confidence they’ve understood it.


• Don’t be too prescriptive in the brief as you still want developers to use their expertise in coming up with solutions.


• Reduce the risk of over-engineering or misinterpretation of specs by putting extra effort into the test & acceptance criteria, and review their work regularly.


• Don’t just give them the document to get on with. Step them through it and get them to explain it to you.


• Don’t assume that something that is obvious to you is obvious to them. Explain everything, even if you risk insulting their intelligence.


• Keep documentation lightweight. Remember, “a document isn’t finished when you can’t think of anything else to put in it, it’s finished when you can’t take anything else out of it.” (Steve McConnell).


• Avoid writing a paragraph containing multiple business rules that may result in some rules becoming overlooked. Separate out each business rule into an itemised list that can be ticked off when completed.


• Pull together a work package that not only contains the brief, but also contains any assets or test data the developer needs to complete the work.


• Get the developer to provide you with a work breakdown structure for the task at hand, along with estimates for each task. When the developer goes through this exercise it shows they’ve really thought about what they’ve got to do.
  

Test & Acceptance Criteria
I can’t emphasise enough how important it is to define test & acceptance criteria for the developer. You can do this in a number of ways but I tend to generate some test data for “input” into the system and specify how it should result in its “output” state. If you do have a test team, get them to contribute to this section.

Your T&A criteria should exercise the non-functional requirements as well as the functional requirements. So, if you’ve specified 100,000 records as the warrantable limit of your system then you need to specify this in the T&A. You should also provide this test dataset in the work package for the developer.

Work Packages
In many organisations I’ve worked for, it’s very rare for a developer to do everything in the software lifecycle themselves. The organisations are usually made up of teams of Business Analysts, Graphic Designers, Producers, PM’s, Developers, and Testers. This results in a collection of documents looking at the project from different perspectives that the developers have to implement.

My personal preference when briefing developers is to create what I call a Work Package. This work package pulls together the relevant sections from the collection of documentation to deliver a specific piece of functionality. The following diagram should give you some ideas as to what could be included in a work package:


I often get asked how big a work package should be. This obviously depends on the size of the project, but from the point at which the developer starts the work package to the point where it’s finished (see previous post on what is classed as finished!), should be a minimum of 1 week. I’ve written many work packages that take 20-30 days to complete so there are no strict guidelines on how big it should be. I have found though that writing a detailed work package for small tasks of a few hours simply doesn’t work. Quite often in these situations I find a whiteboard or flipchart perfectly adequate, but the important point is that it’s being written down.