Skip to main content

Reason 2: Clean up your JAR-files

Update: Added a summary section at the end of this post.

This post is a part of a tiny series I'm doing on why we use Maven, and you should too.
Previous posts:
Some background

Back a couple of months, I got the rewarding task of cleaning up our project's lib folder. You know the one: Crammed with JAR-files of various versions of the various dependencies your project has..
fizz-project
\
fizz-core
fizz-web
fizz-libs
 \
   junit.jar
   spring.jar
   common.jar
   lots and lots of others...
   ...

Yup, good old fizz-libs. It needs to be regularly cleaned up to reduce software rot. Over time, the developers try out new open source libraries and remove the use of old ones, but they seldom take care to clean out the libs-folder, because they don't know if there could be any hidden effects from removing JAR files.

Now, at our place we use Eclipse's .classpath file for specifying each module's dependencies. Each module is explicitly configured with which JAR-files (within fizz-libs) it depends on.

Unfortunately, Eclipse's .classpath file does not support any automatic reporting for analyzing and discovery of unused JAR-files. So I grabbed Jar Analyzer and set it loose on our libs folder (by the way, the author of the tool, Kirk, has a blog full of good thoughts on JAR-design, lately taking a humanly readable take on OSGi, recommended).

Jar Analyzer scans for compile dependencies, meaning that it can create a tree of which JAR-files are needed to compile which JAR-files that are needed to compile these JAR-files, and so on. You get a very nice report/graph which shows you all the JAR-files and why they are in there.

You can also see the JAR-files that don't have any connection to your code, remove them and their children. What I found in our libs folder was that about 20% of the 150 JAR files in our libs folder were unused at compile time, and these were potential JARs to be removed.

The big aber is that you don't get any hint on which JAR-files are used only at runtime by means of discovery and reflection. And this is where the real work begins.

The only way to find out whether a JAR file is used at runtime is basically to take it out, start up your application and test every functionality. If you have an application of moderate size, performing a 100% regression test takes many hours. So in practice, I ended up doing alot of guessing, quick and dirty testing, and asking around to find out which of the runtime dependencies were actually in use.

In the end, after two days of researching and testing, I ended up removing half of the compile-time-unused dependencies, crossing my fingers I didn't break anything in the process (which I did anyway).

The experience left something to be desired in the way we define dependencies: Eclipse's .classpath file simply does not allow you to express how and why which dependencies are in your project. You need something else: A tool that can define scope, version and transitivity of dependencies.

Scope


The total classpath for our running application is expressed in a dependencies module, which is an Eclipse project that solely exists to be used by Ant to build up which JAR-files should be part of our deployable WAR-file.

This dependencies module is alot like our web module, but it excludes references to JAR-files which are already available in the runtime of our application server. You could say that these JAR-files belong to a certain scope: they are not needed at compile time, and they're not needed at testing-time. Their scope is limited to runtime.

Another scope we often talk about is testing. These JAR-files are only needed for running and compiling tests. JUnit and mocking toolkits are typical examples. A good reason for keeping JAR-files with this scope seperate is that you do not want these JAR-files deployed along with the rest of your application.

So, let's take another look at our libs folder:

   fizz-libs
 \
   junit.jar  (test scope)
   mock.jar   (test scope)
   spring.jar (compile scope)
   common.jar (compile scope)
   jboss.jar  (runtime dependency)
So of these, only spring.jar and common.jar need to be brought along when we are deploying.

Instead of hacking together a special deploy-classpath configuration for Ant, Maven does exactly this when building WAR-files out of the box.

Transitive dependencies

Let us take a look at a couple of our modules (mea:
fizz-project
\
fizz-core
fizz-web
If fizz-web depends on fizz-core, and fizz-core depends on spring.jar for compilation, you can be pretty sure that fizz-web also depends on spring.jar indirectly. We say that fizz-web has a transitive dependence to spring.jar, or spring.jar is a transitive dependency of fizz-web.

In these terms, all dependencies in Eclipse are non-transitive until they are configured as being exported. This is the nearest thing Eclipse gets to having scoping on its dependencies.

Naturally, keeping track of transitive dependencies is imperative. Lack of control on this leads to missing class definitions at runtime, as well as unneeded JAR-files in your lib-folder that need to be cleaned up periodically (like I did).

Versions on dependencies

Now, add into this mix that spring.jar also depends on other JAR-files again, perhaps apache-commons or something else. And then you have to remember that you need to know which versions of these 3rd party library depends on which versions of their transitive deps, and
so on. This problem is expressed pretty well by Jason Van Zyl in his blog post Why Maven uses JAR names with versions, so I'm not going to write more on it right here. Basically, having an explicit notion of which version is an important part of controlling your dependencies.


Summary: Keeping order in your dependencies is easier with Maven
Since Maven makes us express the dependencies, their versions and their scope in the POM, an XML file dedicated to this purpose, we have a much easier task of maintaining our JAR-files. The easiest way to show this is to demonstrate the maven-dependency-plugin on a tiny example application I had lying around:

[INFO] [dependency:analyze]
[WARNING] Used undeclared dependencies found:
[WARNING] com.opensymphony:xwork:jar:2.0.4:compile
[WARNING] Unused declared dependencies found:
[WARNING] org.springframework:spring-mock:jar:2.0.5:test
[WARNING] org.springframework:spring-core:jar:2.0.5:test
[WARNING] javax.servlet:servlet-api:jar:2.4:provided
[WARNING] javax.servlet:jsp-api:jar:2.0:provided


The result of running mvn dependency:analyze is a report saying:

(a) which dependencies this project has that are unused (and can be removed),
(b) which dependencies this project has that are used at runtime, and
(c) which undeclared transitive dependencies are sucked in, but not declared in the pom.xml as it should be.

This is all the information I need to do the cleanup I spent two days on (given that all dependencies are correctly configured in our project).

I hope next time I'll be able to express why it's a good thing to publish/subscribe dependencies instead of pushing them into projects.

Comments

Popular posts from this blog

Open source CMS evaluations

I have now seen three more or less serious open source CMS reviews. First guy to hit the field was Matt Raible ( 1 2 3 4 ), ending up with Drupal , Joomla , Magnolia , OpenCms and MeshCMS being runner-ups. Then there is OpenAdvantage that tries out a handful ( Drupal , Exponent CMS , Lenya , Mambo , and Silva ), including Plone which they use for their own site (funny/annoying that the entire site has no RSS-feeds, nor is it possible to comment on the articles), following Matt's approach by exluding many CMS that seem not to fit the criteria. It is somewhat strange that OpenAdvantage cuts away Magnolia because it "Requires J2EE server; difficult to install and configure; more of a framework than CMS", and proceed to include Apache Lenya in the full evaluation. Magnolia does not require a J2EE server. It runs on Tomcat just like Lenya does (maybe it's an idea to bundle Magnolia with Jetty to make it seem more lightweight). I'm still sure that OpenAdvant

Git Stash Blooper (Could not restore untracked files from stash)

The other day I accidentally did a git stash -a , which means it stashes *everything*, including ignored output files (target, build, classes, etc). Ooooops.. What I meant to do was git stash -u , meaning stash modifications plus untracked new files. Anyhows, I ended up with a big fat stash I couldn't get back out. Each time I tried, I got something like this: .../target/temp/dozer.jar already exists, no checkout .../target/temp/core.jar already exists, no checkout .../target/temp/joda-time.jar already exists, no checkout .../target/foo.war already exists, no checkout Could not restore untracked files from stash No matter how I tried checking out different revisions (like the one where I actually made the stash), or using --force, I got the same error. Now these were one of those "keep cool for a second, there's a git way to fix this"situation. I figured: A stash is basically a commit. If we look at my recent commits using   git log --graph --

Leaving eyeo

Thirteen blog posts later, this one notes my departure from eyeo after 4 years and 3 months. I joined eyeo around the headcount of 80 employees, and now I think there's just over 250 people there. My role coming in was as operations manager, doing a mix of infrastructure engineering and technical project management. I later on took on organizational development to help the company deal with its growing pains . We introduced cross-functional teams, departments (kind of like guilds), new leadership structures, goal-setting frameworks, onboarding processes and career frameworks.  And all of this in a rapidly growing distributed company. I'm proud and happy that for a long time I knew every employee by name and got to meet every single new-hire through training them on company structure and processes.  At some point, we had enough experienced leaders and organizational developers that I could zoom back in on working in one team, consulting them on  Git and continuous integration

Managing dot-files with vcsh and myrepos

Say I want to get my dot-files out on a new computer. Here's what I do: # install vcsh & myrepos via apt/brew/etc vcsh clone https://github.com/tfnico/config-mr.git mr mr update Done! All dot-files are ready to use and in place. No deploy command, no linking up symlinks to the files . No checking/out in my entire home directory as a Git repository. Yet, all my dot-files are neatly kept in fine-grained repositories, and any changes I make are immediately ready to be committed: config-atom.git     -> ~/.atom/* config-mr.git     -> ~/.mrconfig     -> ~/.config/mr/* config-tmuxinator.git       -> ~/.tmuxinator/* config-vim.git     -> ~/.vimrc     -> ~/.vim/* config-bin.git        -> ~/bin/* config-git.git               -> ~/.gitconfig config-tmux.git       -> ~/.tmux.conf     config-zsh.git     -> ~/.zshrc How can this be? The key here is to use vcsh to keep track of your dot-files, and its partner myrepos/mr for o

Considerations for JavaScript in Modern (2013) Java/Maven Projects

Disclaimer: I'm a Java developer, not a JavaScript developer. This is just what I've picked up the last years plus a little research the last days. It's just a snapshot of my current knowledge and opinions on the day of writing, apt to change over the next weeks/months. We've gone all modern in our web applications, doing MVC on the client side with AngularJS or Ember , building single-page webapps with REST backends. But how are we managing the growing amount of JavaScript in our application? Yeoman 's logo (not necessarily the conclusion of this blog post) You ain't in Kansas anymore So far we've just been doing half-random stuff. We download some version of a library and throw it into our src/main/webapp/js/lib , or we use it from a CDN , which may be down or unreachable when we want to use the application.. Some times the JS is minified, other times it's not. Some times we name the file with version number, other times without. Some