Skip to main content

Getting More Agile - 2009 in Review

One of the reason I like blogging is that writing forces me to gather insights on what I do. It's kind of like a personal retrospective. In my current situation, I'm now at a bit of a stand-still, where we (our team at work) have achieved what we set out to do last year, and now we are a bit unsure on what to do next.

So, before we move on to have a look at our future goals, let's do a little year-in-review, 2009:

I started working at IP Labs in the beginning of the year. Part of my role in the team was to do some agile coaching. Why did they need this? I think they were feeling some frustrations which are typical for growing teams. They come from a situation where they were small enough to master chaos, but over time, as the code-base and team grows, they need some protocols, rituals and routines to help them keep on top of the mud. In short, agile practices.

There were times where I got really frustrated because things were moving to slow, and the team did not adopt some practices as vigurously as I would have wanted. Now, more than a year after, I'm really proud of the team, and I think we've done really well and achiveved a lot.

Below, I'll iterate through the practices we've been having a go at, and I'll link up to the Cantara agile wiki where appropriate. Here's a small TOC to start off with, as the post got a bit long:

  1. Continuous Integration
  2. Continuous Deployment
  3. Centralized artifact management
  4. Metrics
  5. Stand-ups
  6. Knowledge meetings and Retrospectives
  7. Team structure
  8. Agile methodologies
  9. Summary and wishes for the future

Continuous Integration


Before:

There was a nightly cron-job that build the whole shebang, running unit-tests. If the build failed, the team received a mail with the console-output of the build. Build-breakage during the day was discovered by random SCM updates, followed by random office visits and team mails to get it fixed again. I also think there was a habit of only performing an update early morning.

Measures:

We replaced the cron-job with Hudson. We increased the frequency of the build to poll SCM every five minutes and extended the test-suite to include a full database-teardown/setup, so we also discover inconsistencies in our schema (we used to have one centralized developer database with the according havoc, but now everyone can set up a local database in seconds). We also set up commit-mails, so developers get instant notification when someone does a commit.

After:

We now discover build breakage within 30 minutes of a commit, along with a ready list of commits for the build to track down and fix the break. We added more machines to the Hudson cluster to increase build speed, so the different branches of our product can be built in parallel.

Continuous Deployment


Before:

We had a set of dedicated test servers in the various datacenters we have around the world. These servers were deployed manually on demand. In order to try out some new functionality or verify a bug, you had to get a developer to find an available test-server, manually deploy and then communicate the deployment back.

Measures:

We set up internal nightly deployment jobs in Hudson, one for each branch of our product. If the build is not successful, nothing is deployed. After a successful build, we run web-based integration tests to make sure things are up and running again. We also set up a weekly build, which is deployed early monday morning.

After:

Each morning, QA, managers, designers, etc. can find the latest greatest version of our product running on an internal server with no effort needed from developers. The weekly deployment proves helpful because we have an earlier version of our product we can compare with if we find something wrong, and is also a somewhat more stable environment for our testers, etc.

Centralized artifact management (with Maven)


Before:

Every library was built with Ant and manually committed into our product's libs folder. I think that this lead to less modularization, because it was a bit of a pain to externalize a module from the product. I've blogged extensively about this subject before.

Measures:

We started using Maven for building our libraries, and are currently undergoing an experiment to mavenize the build of our whole product. We set up Nexus as an artifact repository.

After:

We have gotten as far as making a parallell build system, so now we've got both our Maven build *and Ant build under continuous integration. It's a little bit of overhead maintaining both dependency sets, but we're hoping that after a few months, we can convince the whole team to make the switch. We just need to get our whole build/deployment cycle working with Maven first, and make sure that the IDE usage does not suffer (testing m2eclipse extensively these days).

Metrics

Before:

We had a coding standard including Eclipse warnings. Violating our compilation-warnings was a no-go (zero-warning-policy). Beyond this, we didn't measure much in our code base.

Measures:


After:

With our mavenization efforts, we were recently able to make use of Sonar (which rocks, btw) instead of the old, more primitive (boring) Ant reports in Hudson. We're still not so good at making use of these reports, but perhaps putting some numbers on our complexity and test coverage will help make it clear to the team where the weak spots in our code base lies, and how it can steadily get better/worse depending on our efforts.

Stand-ups


There were no standups, only the odd rush meeting when needed.

Measures:

After creating our Java lounge, we started having standup meetings every morning. With 18 people, it quickly got boring. There was simply too many people to have a productive standup, in my opinion. We also had very little news that were worth communicating to all of us on a day-to-day basis, so after some weeks, we reduced the frequency to twice a week: Monday morning (what will we do this week), and Friday morning (what did we do this week) in combination with our weekly knowledge meeting (see below).

After:

Having a bi-weekly standup is much nicer, I think, although you have to support it with *something else*. Having the Friday standup sitting down in the knowledge meeting gives much more room for productive discussion, and it's not the end of the world if we go beyond our designated 10 minute round.

Knowledge meetings and Retrospectives

Before:

Every Wednesday, we had a one hour long presentation by one of the team members, either about a module or feature in the product, or some general technology (Spring, Maven, etc). There was never much retrospectives to speak of.

Measures:

We've kept great practice of the knowledge meetings, although moved it to Friday. We've also started using them for release retrospectives and more internal discussions.

After:

We've only had one real retrospective so far, but it was really awesome, so much feedback and opinions from the team about how we can work differently. Having our knowledge meeting on Fridays also gives a more relaxed, week-in-review kind of tone, I think. Wednesdays were also bad because they collided with our deployments. One outcome of our first retrospective at the end of last year is that we put some sub-structure into the team (see below).

Structure of the teams

Before:

One big team with 18 developers. Individuals, or some time pairs, were appointed responsibility for a certain new feature or refactoring (some times even more people, depending on the size of the task). You knew (well, it's still a lot like this) that Mr. X is responsible for component Y.

Measures:

Based on said retrospective, we split the team into four sub-teams.

After:

We still do standups together, but leave the sub-teams to do their own standups, situps, whatever. The sub-teams are completely self-organized, and can choose whatever practices they choose. My team tried Scrum for a while, but dropped it for Kanban, which two other teams have now picked up as well. The fourth team is really small and focused, so I think they manage fine without any method to speak of. Today, new features and tasks are more assigned on a sub-team basis, and less on an individual basis. It's easier to bring up things for discussions in the sub-team, easier to get someone to pair-program with, and over all more ownership, I think.

Agile Methodologies

Before:

There was kind of dynamic/chaotic flow in the team. People knew who did what, and who to turn to in order to get a feature. The team manager tried to stay on top of what everyone was doing in relation to which customer/manager's demands.

Measures:

I tried hard to introduce Scrum on the team, but it never really took off. I now think that management-wize, the team already had much of the agility it needed and Scrum didn't add anything extra to the mix. The team was self-organized, and mostly well shielded from the business side. Even though our Scrum prototype-project was successfully delivered, Scrum failed to charm my colleagues (it was more popular with the business/customer side). The daily standups, the estimation, the planning, the burndown, it was all too much overkill for a team used to "just doing it".

After:

With the organizing of the sub-teams, we now leave leave the decision to Scrum or not to the sub teams. Three of our teams have adopted Kanban boards to provide transparency of work-in-progress, although we still don't have any mechanisms in place for communicating progress to the business side.

Summary

Wow, that's actually a big list. Like I said, I'm really proud of us for how far we've come with getting more agile in just a year. Now, this presents us with the question, what should we aim for now?

Some things I would like for us to achieve in the future:

  • More pair-programming. I've tried to do this, but I'm not so good at it myself, and its not wildly popular with the team either (yet).
  • Coding dojos. I've wanted to introduce this for a long while, but hey, let's face it, programming in front of everybody is straight out scary. Hope I can summon the courage some time the next few months.
  • More test coverage. We're now at about 12% unit-test coverage (excluding the tests that use database). I'd like it to grow, say to 50% over the next couple of years.
  • More integration tests. We still depend on too much manual testing before we can make a release. I would like it if 99% of our bugs were discovered by automatic tests way ahead of the release date.
  • More transparency to the business side. I want them to be able to see what we're doing without having to ask us about it.
  • More frequent releases. Well, who doesn't want this. We currently do a bugfix release every two weeks, and a major release about every quarter. I would like to tear down the difference between these two kinds, and increase frequency, automate more of the process.

That's it. Please comment with any ideas or feedback. Would be nice to hear how you have done it at your place :)

Popular posts from this blog

Encrypting and Decrypting with Spring

I was recently working with protecting some sensitive data in a typical Java application with a database underneath. We convert the data on its way out of the application using Spring Security Crypto Utilities. It "was decided" that we'd be doing AES with a key-length of 256, and this just happens to be the kind of encryption Spring crypto does out of the box. Sweet!

The big aber is that whatever JRE is running the application has to be patched with Oracle's JCE in order to do 256 bits. It's a fascinating story, the short version being that U.S. companies are restricted from exporting various encryption algorithms to certain countries, and some countries are restricted from importing them.

Once I had patched my JRE with the JCE, I found it fascinating how straight forward it was to encrypt and decrypt using the Spring Encryptors. So just for fun at the weekend, I threw together a little desktop app that will encrypt and decrypt stuff for the given password and sa…

Managing dot-files with vcsh and myrepos

Say I want to get my dot-files out on a new computer. Here's what I do:

# install vcsh & myrepos via apt/brew/etc
vcsh clone https://github.com/tfnico/config-mr.git mr
mr update

Done! All dot-files are ready to use and in place. No deploy command, no linking up symlinks to the files. No checking/out in my entire home directory as a Git repository. Yet, all my dot-files are neatly kept in fine-grained repositories, and any changes I make are immediately ready to be committed:

config-atom.git
    -> ~/.atom/*

config-mr.git
    -> ~/.mrconfig
    -> ~/.config/mr/*

config-tmuxinator.git  
    -> ~/.tmuxinator/*

config-vim.git
    -> ~/.vimrc
    -> ~/.vim/*

config-bin.git   
    -> ~/bin/*

config-git.git          
    -> ~/.gitconfig

config-tmux.git  
    -> ~/.tmux.conf    

config-zsh.git
    -> ~/.zshrc

How can this be? The key here is to use vcsh to keep track of your dot-files, and its partner myrepos/mr for operating on many repositories at the same time.

I discovere…

Always use git-svn with --prefix

TLDR: I've recently been forced back into using git-svn, and while I was at it, I noticed that git-svn generally behaves a lot better when it is initialized using the --prefix option.

Frankly, I can't see any reason why you would ever want to use git-svn without --prefix. It even added some major simplifications to my old git-svn mirror setup.

Update: Some of the advantages of this solution will disappear in newer versions of Git.

For example, make a standard-layout svn clone:

$ git svn clone -s https://svn.company.com/repos/project-foo/

You'll get this .git/config:

[svn-remote "svn"]
        url = https://svn.company.com/repos/
        fetch = project-foo/trunk:refs/remotes/trunk
        branches = project-foo/branches/*:refs/remotes/*
        tags = project-foo/tags/*:refs/remotes/tags/*

And the remote branches looks like this (git branch -a):
    remotes/trunk
    remotes/feat-bar

(Compared to regular remote branches, they look very odd because there is no remote name i…

The Best Log Viewer Ever

This is what it looks like when I want to have a look through the logfile, to see what a user did on one of our machines one day:


Read the whole story about how it works on the Viaboxx Systems blog (and upvote on DZone!).

Microsoft ups their Git efforts another notch

This week Microsoft announced first class Git support embedded in the coming version of Visual Studio.

Now, it's not completely shocking. We could have seen it coming since Microsoft started offering Git repos on CodePlex, and more recently offering a Git client for TFS. In any case, these are some big news. Scott Hanselman weighs on some features and some more background here.

For those who are a bit unaware of what the Git situation on Windows looks like these days, I've dotted down these notes:
Some explanation on these:

msysGit has long been The Way to use Git on Windows. It's basically a port of Git itself, so it's a command-line tool.GitExtensions (includes Visual Studio integration), TortoiseGit, Git Shell, posh-git and most other tools are powered by msysGit.libgit2 is a native library for doing Git stuff. It is developed completely separate from Git itself. The above tools could (and should) probably use libgit2 instead of hooking onto and around msysGit.Github…