Skip to main content

Broken Snapshots in Java Builds

Recently I've done a lot of thinking about build tools, especially in regards to Maven, Grails and Gradle, and how they play into release management and versioning with Git. This is just a post to get some of those thoughts off my chest. I'll come back to Gradle in future posts, as I build some more experience with it at work.

A few months ago, I wrote an article on our company blog about Grails' broken snapshot dependency mechanism.
Even though Grails (up onto, and including Grails 2) support snapshot dependencies, the feature is flawed in a way that makes it unusable for us. This will be fixed in Grails 3, but we couldn't wait that long, so we ended up hacking together a workaround. This article describes why and how we did it. (cont)
Now I've done a lot of modularization of huge builds over the years, and I've come to really like Maven's snapshot dependencies as an enabler for balancing between externalizing a library, and keeping it as part of the build.

So when a build tool comes along and claims that it supports snapshots, I expect it to fully support it, all the way, with local and remote repositories, time-stamping, update-policies, the lot.

So which build tools properly grok snapshots?


Which do not?

  • Ivy
  • Grails, and Griffon, because they're based on Ivy, but they will switch to Gradle next year
  • SBT uses Ivy, and thereby the Play 2.0 framework is affected as well.
I'd just like to emphasize how terrifying I find it that relatively fresh projects like SBT and Play base themselves on the muddy foundations of Ivy, instead of building something on top of Aether.

I haven't properly tested Lein, but I'll give it a run soon and update this post with the results. If anyone knows already, please comment and I'll sort it in.

What was the point of this blog post again?

The reason I'm bringing this up again is actually that there were some responses to my workaround on Twitter. Graeme Rocher, Grails project lead, responded with an alternative solution I figured it'd be nice to post in full, as his solution might work for some.

My tweet announcing the post.

Graeme:
With Grails 2.0 why didn't you just re-order the repositories as per the docs?
We're in fact still stuck on the old Grails 1.3.6, but anyhoo, I replied:
Because we want the freshest snapshot, whether it is from remote or local. See https://github.com/alkemist/grails-snapshot-dependencies-fix/issues/1
Graeme:
Ok, but surely a command line / system property to switch the repo order would have solved that for you?
Me:
TBH, that didn't occur to me. It would be a bit annoying for local dev though: Need to invoke grails twice to update deps.
Graeme:
More annoying than maintaining the hack? It would be "grails -Duselocal=true run-app". You could alias it to another cmd even :-)
In retrospect, my own hack has withstood the test of time pretty well. We haven't made any further Grails upgrades though, and I'm not sure if we will before Grails 3.0.

Some reflections on the approach Graeme suggests:

  • CI builds could always run with useLocal=false, but we would have to always deploy up-stream dependencies through the central maven repo. We pretty much always do this though, so this would work fine. Your build might be taking some shortcuts on this involving the local maven repo.
  • Developers would have to make a conscious choice when they would want to use locally built snapshots, and then run with the switch. This would work fine for us, as this is happening less as our snapshot deps are currently very stable, but I can imagine a build where you want the newest of both remote and local very often: You would then first have to run a build to get the remote ones, and then run a second to overwrite with your locally built snapshots.
  • His workaround (and mine) could be ported to other build tools (like SBT) as well.

In the end..

I suppose most Grails project developers don't care, and avoid using snapshots. This implies that
the modules that they do externalize must be released in a new version for every change that they want  to include in the downstream Grails application.

This is a hassle, but OK for slow-moving modules. If you have modules with a lot of development in them, chances are you'll just keep them part of the Grails application, and your build grow bigger and bigger, as well as maybe duplicating libraries you'd rather want to re-use in other applications.

Well, enough rambling. I hope this post might help out some Grails users, and make people more aware of problems with build tools based on Ivy..

Comments

  1. Well you are not alone with the problem :). We also suffered fro this as well as from Grails dependency management in Grails in general. What we've done is:
    - use only maven repository, it is remote corporate repository (Nexus) where snapshots and no-snapshots are stored
    - use SNAPSHOT jar modules a little as possible and using Grails plugins instead that are included into each application as inline plugins

    It means that SNAPSHOT versions of jars are build on CI server and if you need them you need to commit the changes, build it on CI server and run/test/etc. your application that will download SNAPSHOT. That's slows down development from time to time, but that is a seldom case. Most of the things are already in the plugins (inline ones)!

    ReplyDelete
  2. Thanks for commenting Alexey. I'm glad to hear that we're not the only ones annoyed by this.

    We've developed some Grails plugins, but we can't use that for most of our snapshot dependencies, because they were used in other non-Grails projects as well. Thanks for the advice though!

    ReplyDelete

Post a Comment

Popular posts from this blog

Open source CMS evaluations

I have now seen three more or less serious open source CMS reviews. First guy to hit the field was Matt Raible ( 1 2 3 4 ), ending up with Drupal , Joomla , Magnolia , OpenCms and MeshCMS being runner-ups. Then there is OpenAdvantage that tries out a handful ( Drupal , Exponent CMS , Lenya , Mambo , and Silva ), including Plone which they use for their own site (funny/annoying that the entire site has no RSS-feeds, nor is it possible to comment on the articles), following Matt's approach by exluding many CMS that seem not to fit the criteria. It is somewhat strange that OpenAdvantage cuts away Magnolia because it "Requires J2EE server; difficult to install and configure; more of a framework than CMS", and proceed to include Apache Lenya in the full evaluation. Magnolia does not require a J2EE server. It runs on Tomcat just like Lenya does (maybe it's an idea to bundle Magnolia with Jetty to make it seem more lightweight). I'm still sure that OpenAdvant

Encrypting and Decrypting with Spring

I was recently working with protecting some sensitive data in a typical Java application with a database underneath. We convert the data on its way out of the application using Spring Security Crypto Utilities . It "was decided" that we'd be doing AES with a key-length of 256 , and this just happens to be the kind of encryption Spring crypto does out of the box. Sweet! The big aber is that whatever JRE is running the application has to be patched with Oracle's JCE  in order to do 256 bits. It's a fascinating story , the short version being that U.S. companies are restricted from exporting various encryption algorithms to certain countries, and some countries are restricted from importing them. Once I had patched my JRE with the JCE, I found it fascinating how straight forward it was to encrypt and decrypt using the Spring Encryptors. So just for fun at the weekend, I threw together a little desktop app that will encrypt and decrypt stuff for the given password

What I've Learned After a Month of Podcasting

So, it's been about a month since I launched   GitMinutes , and wow, it's been a fun ride. I have gotten a lot of feedback, and a lot more downloads/listeners than I had expected! Judging the numbers is hard, but a generous estimate is that somewhere around 2000-3000 have listened to the podcast, and about 500-1000 regularly download. Considering that only a percentage of my target audience actively listen to podcasts, these are some pretty good numbers. I've heard that 10% of the general population in the western world regularly listen to podcasts (probably a bit higher percentage among Git users), so I like to think I've reached a big chunk of the Git pros out there. GitMinutes has gathered 110 followers on Twitter, and 63, erm.. circlers on Google+, and it has received 117 +'es! And it's been flattr'ed twice :) Here are some of the things I learned during this last month: Conceptually.. Starting my own sandbox podcast for trying out everythin

The academical approach

Oops, seems I to published this post prematurely by hitting some Blogger keyboard shortcut. I've been sitting for some minutes trying to figure out how to approach the JavaZone talk mentioned in my previous blog-post. Note that I have already submitted an abstract to the comittee, and that I won't publish the abstract here in the blog. Now of course the abstract is pretty detailed on what the talk is going to be about, but I've still got some elbow room on how to "implement" the talk. I will use this blog as a tool to get my aim right on how to present the talk, what examples to include, what the slides should look like, and how to make it most straightforward and understandable for the audience. Now in lack of having done any presentations at a larger conference before, I'm gonna dig into what I learned at the University, which wasn't very much, but they did teach me how to write a research paper, a skill which I will adapt into creating my talk: The one

Git Stash Blooper (Could not restore untracked files from stash)

The other day I accidentally did a git stash -a , which means it stashes *everything*, including ignored output files (target, build, classes, etc). Ooooops.. What I meant to do was git stash -u , meaning stash modifications plus untracked new files. Anyhows, I ended up with a big fat stash I couldn't get back out. Each time I tried, I got something like this: .../target/temp/dozer.jar already exists, no checkout .../target/temp/core.jar already exists, no checkout .../target/temp/joda-time.jar already exists, no checkout .../target/foo.war already exists, no checkout Could not restore untracked files from stash No matter how I tried checking out different revisions (like the one where I actually made the stash), or using --force, I got the same error. Now these were one of those "keep cool for a second, there's a git way to fix this"situation. I figured: A stash is basically a commit. If we look at my recent commits using   git log --graph --