Skip to main content

Reason 2: Clean up your JAR-files

Update: Added a summary section at the end of this post.

This post is a part of a tiny series I'm doing on why we use Maven, and you should too.
Previous posts:
Some background

Back a couple of months, I got the rewarding task of cleaning up our project's lib folder. You know the one: Crammed with JAR-files of various versions of the various dependencies your project has..
fizz-project
\
fizz-core
fizz-web
fizz-libs
 \
   junit.jar
   spring.jar
   common.jar
   lots and lots of others...
   ...

Yup, good old fizz-libs. It needs to be regularly cleaned up to reduce software rot. Over time, the developers try out new open source libraries and remove the use of old ones, but they seldom take care to clean out the libs-folder, because they don't know if there could be any hidden effects from removing JAR files.

Now, at our place we use Eclipse's .classpath file for specifying each module's dependencies. Each module is explicitly configured with which JAR-files (within fizz-libs) it depends on.

Unfortunately, Eclipse's .classpath file does not support any automatic reporting for analyzing and discovery of unused JAR-files. So I grabbed Jar Analyzer and set it loose on our libs folder (by the way, the author of the tool, Kirk, has a blog full of good thoughts on JAR-design, lately taking a humanly readable take on OSGi, recommended).

Jar Analyzer scans for compile dependencies, meaning that it can create a tree of which JAR-files are needed to compile which JAR-files that are needed to compile these JAR-files, and so on. You get a very nice report/graph which shows you all the JAR-files and why they are in there.

You can also see the JAR-files that don't have any connection to your code, remove them and their children. What I found in our libs folder was that about 20% of the 150 JAR files in our libs folder were unused at compile time, and these were potential JARs to be removed.

The big aber is that you don't get any hint on which JAR-files are used only at runtime by means of discovery and reflection. And this is where the real work begins.

The only way to find out whether a JAR file is used at runtime is basically to take it out, start up your application and test every functionality. If you have an application of moderate size, performing a 100% regression test takes many hours. So in practice, I ended up doing alot of guessing, quick and dirty testing, and asking around to find out which of the runtime dependencies were actually in use.

In the end, after two days of researching and testing, I ended up removing half of the compile-time-unused dependencies, crossing my fingers I didn't break anything in the process (which I did anyway).

The experience left something to be desired in the way we define dependencies: Eclipse's .classpath file simply does not allow you to express how and why which dependencies are in your project. You need something else: A tool that can define scope, version and transitivity of dependencies.

Scope


The total classpath for our running application is expressed in a dependencies module, which is an Eclipse project that solely exists to be used by Ant to build up which JAR-files should be part of our deployable WAR-file.

This dependencies module is alot like our web module, but it excludes references to JAR-files which are already available in the runtime of our application server. You could say that these JAR-files belong to a certain scope: they are not needed at compile time, and they're not needed at testing-time. Their scope is limited to runtime.

Another scope we often talk about is testing. These JAR-files are only needed for running and compiling tests. JUnit and mocking toolkits are typical examples. A good reason for keeping JAR-files with this scope seperate is that you do not want these JAR-files deployed along with the rest of your application.

So, let's take another look at our libs folder:

   fizz-libs
 \
   junit.jar  (test scope)
   mock.jar   (test scope)
   spring.jar (compile scope)
   common.jar (compile scope)
   jboss.jar  (runtime dependency)
So of these, only spring.jar and common.jar need to be brought along when we are deploying.

Instead of hacking together a special deploy-classpath configuration for Ant, Maven does exactly this when building WAR-files out of the box.

Transitive dependencies

Let us take a look at a couple of our modules (mea:
fizz-project
\
fizz-core
fizz-web
If fizz-web depends on fizz-core, and fizz-core depends on spring.jar for compilation, you can be pretty sure that fizz-web also depends on spring.jar indirectly. We say that fizz-web has a transitive dependence to spring.jar, or spring.jar is a transitive dependency of fizz-web.

In these terms, all dependencies in Eclipse are non-transitive until they are configured as being exported. This is the nearest thing Eclipse gets to having scoping on its dependencies.

Naturally, keeping track of transitive dependencies is imperative. Lack of control on this leads to missing class definitions at runtime, as well as unneeded JAR-files in your lib-folder that need to be cleaned up periodically (like I did).

Versions on dependencies

Now, add into this mix that spring.jar also depends on other JAR-files again, perhaps apache-commons or something else. And then you have to remember that you need to know which versions of these 3rd party library depends on which versions of their transitive deps, and
so on. This problem is expressed pretty well by Jason Van Zyl in his blog post Why Maven uses JAR names with versions, so I'm not going to write more on it right here. Basically, having an explicit notion of which version is an important part of controlling your dependencies.


Summary: Keeping order in your dependencies is easier with Maven
Since Maven makes us express the dependencies, their versions and their scope in the POM, an XML file dedicated to this purpose, we have a much easier task of maintaining our JAR-files. The easiest way to show this is to demonstrate the maven-dependency-plugin on a tiny example application I had lying around:

[INFO] [dependency:analyze]
[WARNING] Used undeclared dependencies found:
[WARNING] com.opensymphony:xwork:jar:2.0.4:compile
[WARNING] Unused declared dependencies found:
[WARNING] org.springframework:spring-mock:jar:2.0.5:test
[WARNING] org.springframework:spring-core:jar:2.0.5:test
[WARNING] javax.servlet:servlet-api:jar:2.4:provided
[WARNING] javax.servlet:jsp-api:jar:2.0:provided


The result of running mvn dependency:analyze is a report saying:

(a) which dependencies this project has that are unused (and can be removed),
(b) which dependencies this project has that are used at runtime, and
(c) which undeclared transitive dependencies are sucked in, but not declared in the pom.xml as it should be.

This is all the information I need to do the cleanup I spent two days on (given that all dependencies are correctly configured in our project).

I hope next time I'll be able to express why it's a good thing to publish/subscribe dependencies instead of pushing them into projects.

Comments

Popular posts from this blog

Open source CMS evaluations

I have now seen three more or less serious open source CMS reviews. First guy to hit the field was Matt Raible ( 1 2 3 4 ), ending up with Drupal , Joomla , Magnolia , OpenCms and MeshCMS being runner-ups. Then there is OpenAdvantage that tries out a handful ( Drupal , Exponent CMS , Lenya , Mambo , and Silva ), including Plone which they use for their own site (funny/annoying that the entire site has no RSS-feeds, nor is it possible to comment on the articles), following Matt's approach by exluding many CMS that seem not to fit the criteria. It is somewhat strange that OpenAdvantage cuts away Magnolia because it "Requires J2EE server; difficult to install and configure; more of a framework than CMS", and proceed to include Apache Lenya in the full evaluation. Magnolia does not require a J2EE server. It runs on Tomcat just like Lenya does (maybe it's an idea to bundle Magnolia with Jetty to make it seem more lightweight). I'm still sure that OpenAdvant

What I've Learned After a Month of Podcasting

So, it's been about a month since I launched   GitMinutes , and wow, it's been a fun ride. I have gotten a lot of feedback, and a lot more downloads/listeners than I had expected! Judging the numbers is hard, but a generous estimate is that somewhere around 2000-3000 have listened to the podcast, and about 500-1000 regularly download. Considering that only a percentage of my target audience actively listen to podcasts, these are some pretty good numbers. I've heard that 10% of the general population in the western world regularly listen to podcasts (probably a bit higher percentage among Git users), so I like to think I've reached a big chunk of the Git pros out there. GitMinutes has gathered 110 followers on Twitter, and 63, erm.. circlers on Google+, and it has received 117 +'es! And it's been flattr'ed twice :) Here are some of the things I learned during this last month: Conceptually.. Starting my own sandbox podcast for trying out everythin

Encrypting and Decrypting with Spring

I was recently working with protecting some sensitive data in a typical Java application with a database underneath. We convert the data on its way out of the application using Spring Security Crypto Utilities . It "was decided" that we'd be doing AES with a key-length of 256 , and this just happens to be the kind of encryption Spring crypto does out of the box. Sweet! The big aber is that whatever JRE is running the application has to be patched with Oracle's JCE  in order to do 256 bits. It's a fascinating story , the short version being that U.S. companies are restricted from exporting various encryption algorithms to certain countries, and some countries are restricted from importing them. Once I had patched my JRE with the JCE, I found it fascinating how straight forward it was to encrypt and decrypt using the Spring Encryptors. So just for fun at the weekend, I threw together a little desktop app that will encrypt and decrypt stuff for the given password

The academical approach

Oops, seems I to published this post prematurely by hitting some Blogger keyboard shortcut. I've been sitting for some minutes trying to figure out how to approach the JavaZone talk mentioned in my previous blog-post. Note that I have already submitted an abstract to the comittee, and that I won't publish the abstract here in the blog. Now of course the abstract is pretty detailed on what the talk is going to be about, but I've still got some elbow room on how to "implement" the talk. I will use this blog as a tool to get my aim right on how to present the talk, what examples to include, what the slides should look like, and how to make it most straightforward and understandable for the audience. Now in lack of having done any presentations at a larger conference before, I'm gonna dig into what I learned at the University, which wasn't very much, but they did teach me how to write a research paper, a skill which I will adapt into creating my talk: The one

Git Stash Blooper (Could not restore untracked files from stash)

The other day I accidentally did a git stash -a , which means it stashes *everything*, including ignored output files (target, build, classes, etc). Ooooops.. What I meant to do was git stash -u , meaning stash modifications plus untracked new files. Anyhows, I ended up with a big fat stash I couldn't get back out. Each time I tried, I got something like this: .../target/temp/dozer.jar already exists, no checkout .../target/temp/core.jar already exists, no checkout .../target/temp/joda-time.jar already exists, no checkout .../target/foo.war already exists, no checkout Could not restore untracked files from stash No matter how I tried checking out different revisions (like the one where I actually made the stash), or using --force, I got the same error. Now these were one of those "keep cool for a second, there's a git way to fix this"situation. I figured: A stash is basically a commit. If we look at my recent commits using   git log --graph --