Skip to main content

Git tools for keeping patches on top of moving upstreams

At work, we maintain patches for some pretty large open source repositories that regularly release new versions, forcing us to update our patches to match.

So far, we've been using basic Git operations to transplant our modifications from one major version of the upstream to the next.

Every time we make such a transplant, we simply squash together the modifications we made in the previous version, and land it as one big commit into the next version.

Those who are used to very stringent keeping of Git history may wrinkle their nose at this, but it is a pragmatic choice. Maintaining modifications on top of the rapidly changing upstream is a lot of work, and so far we haven't had the opportunity to figure out a more clever way to do it. Nor have we really suffered any consequences of not having an easy to read history of our modifications - it's a relatively small amount of patches, after all.

With a recent boost in team size, we may have that opportunity. Also the need for better history-keeping increases with our capability do to it well.

From the days where I was more active in the Git community, I've chatted with some people dealing with similar problems:

First up was Johannes Schindelin, now at Microsoft, the maintainer of Git for Windows. He has been keeping his own set of scripts for keeping the modifications needed in Git itself to make it work well on Windows. I'm not sure whether his scripts will be easy to re-use out of the box, but it is somehow reassuring that others have managed this kind of operation.

Another one was Josh Triplett at Intel, who developed git-series to help maintain patch-series as they were evolving and awaiting acceptance upstream (sometimes a patch series waiting for acceptance into the linux kernel can go for many months, and undergo dozens of iterations before finally being accepted, if ever). Now, based on what I remember, git-series could be just the thing for us. But perhaps there are other tools.

git-series takes inspiration from Quilt, which is the classic tool for maintaining patches. Quilt is not tied to a particular VCS.

Going from there, there is the similarly named Guilt, which does tie into Git, persisting the managed patches as Git commits rather than patch files. And there's a similar tool called Stacked Git or StGit that does the same. Both of these are heavily inspired by Mercurial's MQ extension.

Another person into these challenges was Adam Spiers at SUSE, the author of git-deps, a very nifty tool for analyzing dependencies between Git commits. He also authored git-explode, which could be useful for us breaking our modification blob into smaller patches. Adam later pointed my attention to a tool called TopGit, which seems to be developed for exactly our kind of use-case, although Adam did have some qualms about it. Similar to Johannes, he has developed some scripts of his own, which he was tempted to turn into a proper project.

And a more random find: I found several of the above tools mentioned in Git Rev News #54, where a new tool called git-revise was promoted.

Inspired by that, I took to searching the Git mailing list for some of the above keywords, and found that Stefan Xanos has discussed and proposed designs of an evolve tool for Git. The topic was brought up again at the latest Git contributor summit. So maybe Git itself will offer more support for these kinds of use-cases some day.

So, an overview of the tools I could find:

 Tool Tech
 Not based on Git
 Git for Windows'
 Garden shears
 Sh May be very specific to the Git
repository itself
 Rust Irregular activity (only Josh
appears to be developing it)
 Guilt Sh MQ like. Last commit was a
couple of years ago.
 StGit Python
 MQ like. Actively developed.
 git-revise Python Circumvents the slowness of
rebase by working on the index
directly. Very new.
 TopGit Sh 11 years of development and
still active! Mighty, almost
intimidating feature set.

To sum it up, it seems many people have related problems, but there are different pain points and workflows that seem to invite their own solutions. The tools may be either opinionated/tailored to specific use-cases.

So we'll have to try them out, see what works, and perhaps even write our own scripts if that works best for us.

I'd be happy to hear any feedback, additional tools or details I got wrong above.


  1. Anonymous29/9/20 16:36

    This comment has been removed by a blog administrator.


Post a Comment

Popular posts from this blog

Open source CMS evaluations

I have now seen three more or less serious open source CMS reviews. First guy to hit the field was Matt Raible ( 1 2 3 4 ), ending up with Drupal , Joomla , Magnolia , OpenCms and MeshCMS being runner-ups. Then there is OpenAdvantage that tries out a handful ( Drupal , Exponent CMS , Lenya , Mambo , and Silva ), including Plone which they use for their own site (funny/annoying that the entire site has no RSS-feeds, nor is it possible to comment on the articles), following Matt's approach by exluding many CMS that seem not to fit the criteria. It is somewhat strange that OpenAdvantage cuts away Magnolia because it "Requires J2EE server; difficult to install and configure; more of a framework than CMS", and proceed to include Apache Lenya in the full evaluation. Magnolia does not require a J2EE server. It runs on Tomcat just like Lenya does (maybe it's an idea to bundle Magnolia with Jetty to make it seem more lightweight). I'm still sure that OpenAdvant

Considerations for JavaScript in Modern (2013) Java/Maven Projects

Disclaimer: I'm a Java developer, not a JavaScript developer. This is just what I've picked up the last years plus a little research the last days. It's just a snapshot of my current knowledge and opinions on the day of writing, apt to change over the next weeks/months. We've gone all modern in our web applications, doing MVC on the client side with AngularJS or Ember , building single-page webapps with REST backends. But how are we managing the growing amount of JavaScript in our application? Yeoman 's logo (not necessarily the conclusion of this blog post) You ain't in Kansas anymore So far we've just been doing half-random stuff. We download some version of a library and throw it into our src/main/webapp/js/lib , or we use it from a CDN , which may be down or unreachable when we want to use the application.. Some times the JS is minified, other times it's not. Some times we name the file with version number, other times without. Some

Managing dot-files with vcsh and myrepos

Say I want to get my dot-files out on a new computer. Here's what I do: # install vcsh & myrepos via apt/brew/etc vcsh clone mr mr update Done! All dot-files are ready to use and in place. No deploy command, no linking up symlinks to the files . No checking/out in my entire home directory as a Git repository. Yet, all my dot-files are neatly kept in fine-grained repositories, and any changes I make are immediately ready to be committed: config-atom.git     -> ~/.atom/* config-mr.git     -> ~/.mrconfig     -> ~/.config/mr/* config-tmuxinator.git       -> ~/.tmuxinator/* config-vim.git     -> ~/.vimrc     -> ~/.vim/* config-bin.git        -> ~/bin/* config-git.git               -> ~/.gitconfig config-tmux.git       -> ~/.tmux.conf     config-zsh.git     -> ~/.zshrc How can this be? The key here is to use vcsh to keep track of your dot-files, and its partner myrepos/mr for o

What I've Learned After a Month of Podcasting

So, it's been about a month since I launched   GitMinutes , and wow, it's been a fun ride. I have gotten a lot of feedback, and a lot more downloads/listeners than I had expected! Judging the numbers is hard, but a generous estimate is that somewhere around 2000-3000 have listened to the podcast, and about 500-1000 regularly download. Considering that only a percentage of my target audience actively listen to podcasts, these are some pretty good numbers. I've heard that 10% of the general population in the western world regularly listen to podcasts (probably a bit higher percentage among Git users), so I like to think I've reached a big chunk of the Git pros out there. GitMinutes has gathered 110 followers on Twitter, and 63, erm.. circlers on Google+, and it has received 117 +'es! And it's been flattr'ed twice :) Here are some of the things I learned during this last month: Conceptually.. Starting my own sandbox podcast for trying out everythin

Git Stash Blooper (Could not restore untracked files from stash)

The other day I accidentally did a git stash -a , which means it stashes *everything*, including ignored output files (target, build, classes, etc). Ooooops.. What I meant to do was git stash -u , meaning stash modifications plus untracked new files. Anyhows, I ended up with a big fat stash I couldn't get back out. Each time I tried, I got something like this: .../target/temp/dozer.jar already exists, no checkout .../target/temp/core.jar already exists, no checkout .../target/temp/joda-time.jar already exists, no checkout .../target/foo.war already exists, no checkout Could not restore untracked files from stash No matter how I tried checking out different revisions (like the one where I actually made the stash), or using --force, I got the same error. Now these were one of those "keep cool for a second, there's a git way to fix this"situation. I figured: A stash is basically a commit. If we look at my recent commits using   git log --graph --