Monday, April 13, 2009

Reason 2: Clean up your JAR-files

Update: Added a summary section at the end of this post.

This post is a part of a tiny series I'm doing on why we use Maven, and you should too.
Previous posts:
Some background

Back a couple of months, I got the rewarding task of cleaning up our project's lib folder. You know the one: Crammed with JAR-files of various versions of the various dependencies your project has..
fizz-project
\
fizz-core
fizz-web
fizz-libs
 \
   junit.jar
   spring.jar
   common.jar
   lots and lots of others...
   ...

Yup, good old fizz-libs. It needs to be regularly cleaned up to reduce software rot. Over time, the developers try out new open source libraries and remove the use of old ones, but they seldom take care to clean out the libs-folder, because they don't know if there could be any hidden effects from removing JAR files.

Now, at our place we use Eclipse's .classpath file for specifying each module's dependencies. Each module is explicitly configured with which JAR-files (within fizz-libs) it depends on.

Unfortunately, Eclipse's .classpath file does not support any automatic reporting for analyzing and discovery of unused JAR-files. So I grabbed Jar Analyzer and set it loose on our libs folder (by the way, the author of the tool, Kirk, has a blog full of good thoughts on JAR-design, lately taking a humanly readable take on OSGi, recommended).

Jar Analyzer scans for compile dependencies, meaning that it can create a tree of which JAR-files are needed to compile which JAR-files that are needed to compile these JAR-files, and so on. You get a very nice report/graph which shows you all the JAR-files and why they are in there.

You can also see the JAR-files that don't have any connection to your code, remove them and their children. What I found in our libs folder was that about 20% of the 150 JAR files in our libs folder were unused at compile time, and these were potential JARs to be removed.

The big aber is that you don't get any hint on which JAR-files are used only at runtime by means of discovery and reflection. And this is where the real work begins.

The only way to find out whether a JAR file is used at runtime is basically to take it out, start up your application and test every functionality. If you have an application of moderate size, performing a 100% regression test takes many hours. So in practice, I ended up doing alot of guessing, quick and dirty testing, and asking around to find out which of the runtime dependencies were actually in use.

In the end, after two days of researching and testing, I ended up removing half of the compile-time-unused dependencies, crossing my fingers I didn't break anything in the process (which I did anyway).

The experience left something to be desired in the way we define dependencies: Eclipse's .classpath file simply does not allow you to express how and why which dependencies are in your project. You need something else: A tool that can define scope, version and transitivity of dependencies.

Scope


The total classpath for our running application is expressed in a dependencies module, which is an Eclipse project that solely exists to be used by Ant to build up which JAR-files should be part of our deployable WAR-file.

This dependencies module is alot like our web module, but it excludes references to JAR-files which are already available in the runtime of our application server. You could say that these JAR-files belong to a certain scope: they are not needed at compile time, and they're not needed at testing-time. Their scope is limited to runtime.

Another scope we often talk about is testing. These JAR-files are only needed for running and compiling tests. JUnit and mocking toolkits are typical examples. A good reason for keeping JAR-files with this scope seperate is that you do not want these JAR-files deployed along with the rest of your application.

So, let's take another look at our libs folder:

   fizz-libs
 \
   junit.jar  (test scope)
   mock.jar   (test scope)
   spring.jar (compile scope)
   common.jar (compile scope)
   jboss.jar  (runtime dependency)
So of these, only spring.jar and common.jar need to be brought along when we are deploying.

Instead of hacking together a special deploy-classpath configuration for Ant, Maven does exactly this when building WAR-files out of the box.

Transitive dependencies

Let us take a look at a couple of our modules (mea:
fizz-project
\
fizz-core
fizz-web
If fizz-web depends on fizz-core, and fizz-core depends on spring.jar for compilation, you can be pretty sure that fizz-web also depends on spring.jar indirectly. We say that fizz-web has a transitive dependence to spring.jar, or spring.jar is a transitive dependency of fizz-web.

In these terms, all dependencies in Eclipse are non-transitive until they are configured as being exported. This is the nearest thing Eclipse gets to having scoping on its dependencies.

Naturally, keeping track of transitive dependencies is imperative. Lack of control on this leads to missing class definitions at runtime, as well as unneeded JAR-files in your lib-folder that need to be cleaned up periodically (like I did).

Versions on dependencies

Now, add into this mix that spring.jar also depends on other JAR-files again, perhaps apache-commons or something else. And then you have to remember that you need to know which versions of these 3rd party library depends on which versions of their transitive deps, and
so on. This problem is expressed pretty well by Jason Van Zyl in his blog post Why Maven uses JAR names with versions, so I'm not going to write more on it right here. Basically, having an explicit notion of which version is an important part of controlling your dependencies.


Summary: Keeping order in your dependencies is easier with Maven
Since Maven makes us express the dependencies, their versions and their scope in the POM, an XML file dedicated to this purpose, we have a much easier task of maintaining our JAR-files. The easiest way to show this is to demonstrate the maven-dependency-plugin on a tiny example application I had lying around:

[INFO] [dependency:analyze]
[WARNING] Used undeclared dependencies found:
[WARNING] com.opensymphony:xwork:jar:2.0.4:compile
[WARNING] Unused declared dependencies found:
[WARNING] org.springframework:spring-mock:jar:2.0.5:test
[WARNING] org.springframework:spring-core:jar:2.0.5:test
[WARNING] javax.servlet:servlet-api:jar:2.4:provided
[WARNING] javax.servlet:jsp-api:jar:2.0:provided


The result of running mvn dependency:analyze is a report saying:

(a) which dependencies this project has that are unused (and can be removed),
(b) which dependencies this project has that are used at runtime, and
(c) which undeclared transitive dependencies are sucked in, but not declared in the pom.xml as it should be.

This is all the information I need to do the cleanup I spent two days on (given that all dependencies are correctly configured in our project).

I hope next time I'll be able to express why it's a good thing to publish/subscribe dependencies instead of pushing them into projects.