Skip to main content

Expose your test reports

I started off this post with the title "Our Quest for Agility" but it kind of trailed off into being about our latest customer-dashboard tool. I'll do another post about our agile quest later.

The Greenback project

Before the summer, I was issued with the task of setting up a customer-management portfolio tool, which I named Greenback (as in, money, because that's what we want from our customers). The first goal was to provide our customer-managers with an overview of which customers exists, where they are installed, which applications they are running and other configuration details. Greenback is basically a web application with a big table with our customers and some filters to see the customers for a single manager, or with a particular configuration. Our customer-managers can also edit, add and comment all the entries, naturally.

We have a rather large portfolio of customers (200+), and each of these are hosted with their own application of our software. Through development, we strive to keep all these customers in a working state for testing, but it's not always easy. Developing a feature for one customer will some times break the functionality in many others. Maintaining the test-data for every customer is also a challenge.

Automatic deployment and web testing


At the same time, I started pondering about how we could continuously test all these customers' applications. First thing, we got automatic deployment set up. I then made a simple web-test that basically tried loading the front page of a customer's application on the nightly deployment. I then looped this test over every customer, every morning (in our CI-environment/Hudson).

At the same time, we made a big overhaul of our test data to get every customer's test-data into working condition. It didn't take long before the first customer's application broke again on the nightly build. I started wondering: How can we keep every customer in a working state? We need some kind of proper reporting that people will respond to, because scrolling through a console output to see which applications are broken simply does not work over time.

Synergy

Then it occurred to me. Why not report the results of the web-tests back to the Greenback overview? I put together a little call-back at the end of our customer-test that wrote the result into Greenback, merely as a traffic light: Red meant that the setup scripts for the customer were not working at all, yellow meant that the application was working, but the front page wasn't working properly, and green meant everything is OK. I then made a new column in our customer table with the color of the test-result, and a link to the nightly deployment for those who want to investigate any problems, or test a particular application.

Later on, I also added a little test-history so customer-managers and developers alike could track when a certain application stopped working (we can't always drop what we're doing and fix test data every time we see a red light). Next in line is to produce a feed that people can subscribe to, to see any changes in the state of applications.

The great advantage of this is that the customer-managers themselves can take ownership in making sure that their applications are working. It was all in all pretty easy to set up (a few weeks of work, all in all), and I hope others with similar needs will find some inspiration in this.

Comments

Popular posts from this blog

Open source CMS evaluations

I have now seen three more or less serious open source CMS reviews. First guy to hit the field was Matt Raible ( 1 2 3 4 ), ending up with Drupal , Joomla , Magnolia , OpenCms and MeshCMS being runner-ups. Then there is OpenAdvantage that tries out a handful ( Drupal , Exponent CMS , Lenya , Mambo , and Silva ), including Plone which they use for their own site (funny/annoying that the entire site has no RSS-feeds, nor is it possible to comment on the articles), following Matt's approach by exluding many CMS that seem not to fit the criteria. It is somewhat strange that OpenAdvantage cuts away Magnolia because it "Requires J2EE server; difficult to install and configure; more of a framework than CMS", and proceed to include Apache Lenya in the full evaluation. Magnolia does not require a J2EE server. It runs on Tomcat just like Lenya does (maybe it's an idea to bundle Magnolia with Jetty to make it seem more lightweight). I'm still sure that OpenAdvant

What I've Learned After a Month of Podcasting

So, it's been about a month since I launched   GitMinutes , and wow, it's been a fun ride. I have gotten a lot of feedback, and a lot more downloads/listeners than I had expected! Judging the numbers is hard, but a generous estimate is that somewhere around 2000-3000 have listened to the podcast, and about 500-1000 regularly download. Considering that only a percentage of my target audience actively listen to podcasts, these are some pretty good numbers. I've heard that 10% of the general population in the western world regularly listen to podcasts (probably a bit higher percentage among Git users), so I like to think I've reached a big chunk of the Git pros out there. GitMinutes has gathered 110 followers on Twitter, and 63, erm.. circlers on Google+, and it has received 117 +'es! And it's been flattr'ed twice :) Here are some of the things I learned during this last month: Conceptually.. Starting my own sandbox podcast for trying out everythin

Considerations for JavaScript in Modern (2013) Java/Maven Projects

Disclaimer: I'm a Java developer, not a JavaScript developer. This is just what I've picked up the last years plus a little research the last days. It's just a snapshot of my current knowledge and opinions on the day of writing, apt to change over the next weeks/months. We've gone all modern in our web applications, doing MVC on the client side with AngularJS or Ember , building single-page webapps with REST backends. But how are we managing the growing amount of JavaScript in our application? Yeoman 's logo (not necessarily the conclusion of this blog post) You ain't in Kansas anymore So far we've just been doing half-random stuff. We download some version of a library and throw it into our src/main/webapp/js/lib , or we use it from a CDN , which may be down or unreachable when we want to use the application.. Some times the JS is minified, other times it's not. Some times we name the file with version number, other times without. Some

Managing dot-files with vcsh and myrepos

Say I want to get my dot-files out on a new computer. Here's what I do: # install vcsh & myrepos via apt/brew/etc vcsh clone https://github.com/tfnico/config-mr.git mr mr update Done! All dot-files are ready to use and in place. No deploy command, no linking up symlinks to the files . No checking/out in my entire home directory as a Git repository. Yet, all my dot-files are neatly kept in fine-grained repositories, and any changes I make are immediately ready to be committed: config-atom.git     -> ~/.atom/* config-mr.git     -> ~/.mrconfig     -> ~/.config/mr/* config-tmuxinator.git       -> ~/.tmuxinator/* config-vim.git     -> ~/.vimrc     -> ~/.vim/* config-bin.git        -> ~/bin/* config-git.git               -> ~/.gitconfig config-tmux.git       -> ~/.tmux.conf     config-zsh.git     -> ~/.zshrc How can this be? The key here is to use vcsh to keep track of your dot-files, and its partner myrepos/mr for o

Git Stash Blooper (Could not restore untracked files from stash)

The other day I accidentally did a git stash -a , which means it stashes *everything*, including ignored output files (target, build, classes, etc). Ooooops.. What I meant to do was git stash -u , meaning stash modifications plus untracked new files. Anyhows, I ended up with a big fat stash I couldn't get back out. Each time I tried, I got something like this: .../target/temp/dozer.jar already exists, no checkout .../target/temp/core.jar already exists, no checkout .../target/temp/joda-time.jar already exists, no checkout .../target/foo.war already exists, no checkout Could not restore untracked files from stash No matter how I tried checking out different revisions (like the one where I actually made the stash), or using --force, I got the same error. Now these were one of those "keep cool for a second, there's a git way to fix this"situation. I figured: A stash is basically a commit. If we look at my recent commits using   git log --graph --