Sunday, April 26, 2009

Reason 3: Don't build everything all the time

This post is a part of a tiny series I'm doing on why we use Maven, and you should too.

Previous posts:
Huge projects are hard to work with
Many projects that have grown over time will become too large to easily work with. The workspace gets too many classes, the IDE slows down and the build takes too long. If the developers focus on separating the concerns of the classes into packages of functionality, these packages can be organized into modules. As the contract and concern of a module becomes more defined, its rate of change will fall to a lower frequency than the rest of the project.

Once a module's activity has slowed down sufficiently, you can retire it into an external project, build a JAR-file from it and use this ready-built artifact as a classpath element for building and running the software instead. If changes occur later on, you can re-build the JAR-file.

Examples on where this might be appropriate:
  • Low level libraries and utilities, much like Java's own java.util package only changes every other year or so.
  • Contracts or adapters around external services, because integration with 3rd parties require stable data-formats.
What Maven does
Maven eases the work of maintaining such dependencies. As projects "subscribe" to their dependencies, they pull in the latest appropriate JAR-files automatically (snapshots) or by choice of versioned artifact. In an environment with a continuous integration server and a maven repository, the library developers need only commit their changes, and the artifact will be automatically built, deployed, and pulled into the depending project. A good term to describe this setup is Enterprise Maven Infrastructure (see this link for more information on how to actually implement this),

Again, understand that it is better to subscribe to your dependencies than to push them in.
  • Library developers do not have to deploy their artifacts to all clients
  • Clients keep the control of which libraries they use, in which version
  • Easier to maintain transitive dependencies (see my previous post)
  • Getting newer versions is easy
Just to clarify the last point: When you want a new version of a library, you change the version number in the pom.xml. If you want the latest/greatest build (handy for an adjacent project, or a library under heavy development), you make it a snapshot version number, and suck in the latest JAR file from your maven repository on every build.

I could go on on how bad it is to work on a project with a single huge build, but I imagine most developers out there are either struggling with it right now, or have done so some time in the past. There are many reasons why you want to break out modules of your project for architectural and code-design/quality reasons, but in this post I chose just to focus on the basic reasons. I came across one video from SonaType that illustrates this situation very well, so if you didn't get it from reading this post, please take a look at it (although the fancy staging functionality of Nexus is not that important, nor the point of this post).

Monday, April 13, 2009

Reason 2: Clean up your JAR-files

Update: Added a summary section at the end of this post.

This post is a part of a tiny series I'm doing on why we use Maven, and you should too.
Previous posts:
Some background

Back a couple of months, I got the rewarding task of cleaning up our project's lib folder. You know the one: Crammed with JAR-files of various versions of the various dependencies your project has..
fizz-project
\
fizz-core
fizz-web
fizz-libs
 \
   junit.jar
   spring.jar
   common.jar
   lots and lots of others...
   ...

Yup, good old fizz-libs. It needs to be regularly cleaned up to reduce software rot. Over time, the developers try out new open source libraries and remove the use of old ones, but they seldom take care to clean out the libs-folder, because they don't know if there could be any hidden effects from removing JAR files.

Now, at our place we use Eclipse's .classpath file for specifying each module's dependencies. Each module is explicitly configured with which JAR-files (within fizz-libs) it depends on.

Unfortunately, Eclipse's .classpath file does not support any automatic reporting for analyzing and discovery of unused JAR-files. So I grabbed Jar Analyzer and set it loose on our libs folder (by the way, the author of the tool, Kirk, has a blog full of good thoughts on JAR-design, lately taking a humanly readable take on OSGi, recommended).

Jar Analyzer scans for compile dependencies, meaning that it can create a tree of which JAR-files are needed to compile which JAR-files that are needed to compile these JAR-files, and so on. You get a very nice report/graph which shows you all the JAR-files and why they are in there.

You can also see the JAR-files that don't have any connection to your code, remove them and their children. What I found in our libs folder was that about 20% of the 150 JAR files in our libs folder were unused at compile time, and these were potential JARs to be removed.

The big aber is that you don't get any hint on which JAR-files are used only at runtime by means of discovery and reflection. And this is where the real work begins.

The only way to find out whether a JAR file is used at runtime is basically to take it out, start up your application and test every functionality. If you have an application of moderate size, performing a 100% regression test takes many hours. So in practice, I ended up doing alot of guessing, quick and dirty testing, and asking around to find out which of the runtime dependencies were actually in use.

In the end, after two days of researching and testing, I ended up removing half of the compile-time-unused dependencies, crossing my fingers I didn't break anything in the process (which I did anyway).

The experience left something to be desired in the way we define dependencies: Eclipse's .classpath file simply does not allow you to express how and why which dependencies are in your project. You need something else: A tool that can define scope, version and transitivity of dependencies.

Scope


The total classpath for our running application is expressed in a dependencies module, which is an Eclipse project that solely exists to be used by Ant to build up which JAR-files should be part of our deployable WAR-file.

This dependencies module is alot like our web module, but it excludes references to JAR-files which are already available in the runtime of our application server. You could say that these JAR-files belong to a certain scope: they are not needed at compile time, and they're not needed at testing-time. Their scope is limited to runtime.

Another scope we often talk about is testing. These JAR-files are only needed for running and compiling tests. JUnit and mocking toolkits are typical examples. A good reason for keeping JAR-files with this scope seperate is that you do not want these JAR-files deployed along with the rest of your application.

So, let's take another look at our libs folder:

   fizz-libs
 \
   junit.jar  (test scope)
   mock.jar   (test scope)
   spring.jar (compile scope)
   common.jar (compile scope)
   jboss.jar  (runtime dependency)
So of these, only spring.jar and common.jar need to be brought along when we are deploying.

Instead of hacking together a special deploy-classpath configuration for Ant, Maven does exactly this when building WAR-files out of the box.

Transitive dependencies

Let us take a look at a couple of our modules (mea:
fizz-project
\
fizz-core
fizz-web
If fizz-web depends on fizz-core, and fizz-core depends on spring.jar for compilation, you can be pretty sure that fizz-web also depends on spring.jar indirectly. We say that fizz-web has a transitive dependence to spring.jar, or spring.jar is a transitive dependency of fizz-web.

In these terms, all dependencies in Eclipse are non-transitive until they are configured as being exported. This is the nearest thing Eclipse gets to having scoping on its dependencies.

Naturally, keeping track of transitive dependencies is imperative. Lack of control on this leads to missing class definitions at runtime, as well as unneeded JAR-files in your lib-folder that need to be cleaned up periodically (like I did).

Versions on dependencies

Now, add into this mix that spring.jar also depends on other JAR-files again, perhaps apache-commons or something else. And then you have to remember that you need to know which versions of these 3rd party library depends on which versions of their transitive deps, and
so on. This problem is expressed pretty well by Jason Van Zyl in his blog post Why Maven uses JAR names with versions, so I'm not going to write more on it right here. Basically, having an explicit notion of which version is an important part of controlling your dependencies.


Summary: Keeping order in your dependencies is easier with Maven
Since Maven makes us express the dependencies, their versions and their scope in the POM, an XML file dedicated to this purpose, we have a much easier task of maintaining our JAR-files. The easiest way to show this is to demonstrate the maven-dependency-plugin on a tiny example application I had lying around:

[INFO] [dependency:analyze]
[WARNING] Used undeclared dependencies found:
[WARNING] com.opensymphony:xwork:jar:2.0.4:compile
[WARNING] Unused declared dependencies found:
[WARNING] org.springframework:spring-mock:jar:2.0.5:test
[WARNING] org.springframework:spring-core:jar:2.0.5:test
[WARNING] javax.servlet:servlet-api:jar:2.4:provided
[WARNING] javax.servlet:jsp-api:jar:2.0:provided


The result of running mvn dependency:analyze is a report saying:

(a) which dependencies this project has that are unused (and can be removed),
(b) which dependencies this project has that are used at runtime, and
(c) which undeclared transitive dependencies are sucked in, but not declared in the pom.xml as it should be.

This is all the information I need to do the cleanup I spent two days on (given that all dependencies are correctly configured in our project).

I hope next time I'll be able to express why it's a good thing to publish/subscribe dependencies instead of pushing them into projects.

Reason 1: Get your libs-folder out of SVN

This post is a part of a tiny series I'm doing on why we use Maven, and you should too.

Previous posts:


The libs folder


We have a libs-module in Subversion. When you check out the source code for our main product, this is one of the modules you get. It contains a hundred-and-some JAR-files. These are the dependencies for the sum of our modules:

fizz-project
\
fizz-core
fizz-web
fizz-libs
\
junit.jar
spring.jar
common.jar
Downloading all these MB's of JAR files is something you'll have to do no matter which build system you use, so bandwith cost isn't an argument. Subversion also uses binary diffs, so copying or changing a JAR file doesn't increase repository size significantly.

The problem arises when you want more than one product. As soon as you want to split your product into two applications (or services), each project needs its own libs-folder:

fizz-project
\
fizz-core
fizz-web
fizz-libs
\
junit.jar
spring.jar
common.jar
woop-project
\
woop-core
woop-web
woop-libs
\
junit.jar
spring.jar
common.jar
By doing this, we almost efficiently duplicate our dependencies. While each project can manage its own dependencies without fearing for breaking functionality in the other project, efforts to maintain the libs is duplicated. All upgrading, figuring out bugs, transitive dependencies will have to be done in two places.

The alternative is to introduce a global libs folder:
fizz-project
\
fizz-core
fizz-web
woop-project
\
woop-core
woop-web
global-libs
\
junit.jar
spring.jar
common.jar
... but unfortunately this would over time become a giant heap of libs of different JAR files in different versions, and for even the tiniest project you would need to check out the entire global-libs folder. Because you have no hard linking to which project a lib belongs, you can not actively do cleanups and get rid of unused JARs. Over time, it would grow to be hundreds of MB, perhaps even GB, containing every JAR file of very version all the products of the company ever used.

So neither of these approaches are particularly tasty for an organization with a large codebase and number of projects. The first approach enforces all-source-code-in-one-build with one gigantic sluggish workspace in your Eclipse, while the second enforces one gigantic global-libs JAR folder that everyone hates to download and deal with.

Maven's solution to this: There is one global JAR-repository like in the second approach, but download is done lazily. Each project has meta-data for which JAR-files it needs, and only upon building a particular project are the needed JAR-files downloaded from the company JAR-repository. Also, JAR-files are cached and referenced in a local Maven repository on each machine, so if you are working on three projects that use junit3.jar, you only have one junit3.jar in your ~/.m2/repository directory.

Additionally, Maven support inheriting dependency management. You can define parent projects that define dependencies, and have different projects inherit these.

This post is already long enough for publishing, so I'll continue next week with Reason 2: Clean up your JAR-files.

Sunday, April 05, 2009

Why you should use Maven

Update: I'm doing some work to split this blog post into some more fine grained posts in a series. Please, bear with me if these changes are giving you lots of noise in your feed-reader.

I used to work as a consultant. My job was to improve our clients' ways of software development. Among the most important things I tried to teach them were:

o Good code style and object orientation
o Practical use of good open source tools (don't re-invent the wheel)
o Automating tests of their software, and continuous integration
o Splitting code into modules, dependency-relationships and versions

The last point is perhaps the trickiest. It's hard to understand why it's needed in an isolated project, it is hard to explain how it should be done, and most importantly, there is no de-facto way or standard for managing modules, their relationships, versions and scope.

This is why me and my consultant buddies embraced Maven many years ago. After years of pushing and evangelizing at our clients', presentations at javaBin and JavaZone, Maven has at least become the way for managing and building Norwegian Java projects. There are still old Ant projects around, but I haven't heard of any new ones being started without Maven, or any old projects where a Ant/Maven migration path hasn't been thought of.

Here at IP Labs we still have an Ant build for our main project, and it works. (We have also started using Maven for a bunch of ad-hoc projects). There is a small, co-located and tightly communicating Java team that all use Eclipse with identical setup. In order to build our product, you check it out of Subversion and run "ant build-all", and that runs pretty similar to your average "mvn install": compiling, running tests and building binaries.

So here's the million dollar question: When our build system works perfectly fine, why should we go to the trouble of converting to Maven?

And instead of answering with some fluffy modular design theory, let me get hands-on, and give some real practical reasons in these following blog-posts: