Skip to main content

Git-SVN Mirror without the annoying update-ref

This post is part of a series on Git and Subversion. To see all the related posts, screencasts and other resources, please click here

So no sooner than I had done my git-svn presentation at JavaZone, I got word of a slightly different Git-SVN mirror setup that makes it a bit easier to work with:

In short, my old recipe includes an annoying git update-ref step to keep the git-svn remote reference up to date with the central bare git repo. This new recipe avoids this, so we can simply use git svn dcommit  directly.

So, longer version, with the details. My original recipe is laid out in five steps:

  1. Clone a fresh Git repo from Subversion. This will be our fetching repo.
  2. Set up a bare repo.
  3. Configure pushing from the fetching repo to bare repo
  4. In the shoes of a developer, clone the repo
  5. Set up an SVN remote in the developer's repo
In the new approach, we redefine those last two steps:

(See the original post for how to do the first three.)

4. Clone the bare repo

Note the -o parameter that specifies that we want to call the origin remote "mirror":

tfnico@flint:~/sources/git/>git clone -o mirror ~/git-repos/websites.git/

We can now track the remote branches, and pull the latest changes as they come in to the bare repo.

tfnico@flint:~/sources/git/websites/>git checkout -t mirror/yksi
tfnico@flint:~/sources/git/websites/>git pull --rebase

(Remember to use --rebase to avoid merge commits in case you have local commits.)

When the time comes to push commits back to Subversion, we have to complete the last step.

5. Set up an SVN remote in the developer's repo

We'll do an svn-init with the same parameters as we cloned in the beginning, adding a prefix that matches the name of the remote "mirror":

tfnico@flint:~/sources/git/websites/>git svn init --prefix=mirror/ -s file:///svn-repos/company-repo/websites/

The point of the mirror/ prefix is that whenever we do a pull, the git-svn reference will be updated along with the normal remote reference. So now you can do git svn dcommit without any further ado.

Comments

  1. Great, I came to a similar conclusions earlier - but haven't seen it written down and forgot it in the meantime as well (so kept using update-ref to keep the two pointers refs/remotes/trunk refs/svn/trunk refs/remotes/mirror/svn/trunk in sync).

    Have you run ran into any negative side-effects/gotchas with this --prefix setup since?

    ReplyDelete
  2. Hi Gin, thanks for your comment!

    Generally, no, I haven't seen any real problems with the prefix setup.

    There is one tiny detail: If you accidentally push directly to the bare bare repo (instead of doing a dcommit), and you need to revert that by doing a forced push to remote/svn, you don't have any local reference any more to which commit the SVN HEAD is at. However, this is not really a problem if you are (a) careful, or (b) take measures to make the bare repo non-writable for developers.

    ReplyDelete
  3. Good point, thanks Thomas. I'd probably want to ensure that trouble doesn't strike (most our devs are new to git and might do mistakes). However, I'd prefer not making the whole mirror repo read-only (some of as are pushing pure-git branches there as well..), so will take a look at the gitolite setup: my impression is that it may be able to protect specific branches.

    ReplyDelete
  4. Gin: Perhaps a more practical way to avoid accidents is to have each developer disable the push configuration in their local repositories:

    http://stackoverflow.com/questions/527833/how-to-configure-git-to-avoid-accidental-git-push

    ReplyDelete
  5. Here's a nice one-liner for the above setup:

    git config remote.mirror.pushurl WARNING_USE_SVN_DCOMMIT_INSTEAD_OF_PUSH

    This will cause developers to see an error message when trying to push to the bare repo (mirror).

    ReplyDelete
  6. Good idea - thanks for that too!
    I'd still prefer to do it once (on the server), rather than setting it up on the growing numbers of developer boxes. I guess there is not easy way to share repo configuration (apart from nasty tricks to commit them too) - I'd be happy to be wrong one this.

    Also, do you think it's easy and fast to migrate to this -synced-prefix setup? (the repo is big)
    I guess I could do a 'git remote rename oldmirror svn' , where 'svn' is default; but I'd prefer that one to 'mirror' instead.. can't see anything obvious on man git-svn. Any ideas? thanks again.

    ReplyDelete
  7. Hmm, I think that git-svn needs to be re-initialized (with the --prefix parameter), and to avoid any mess, I've done this in clean clones (so far).

    However, this all takes place on the developer's machines. There's no need to change anything with the fetching/bare repos. So the developers can clone their old repo locally (fastest), add the proper mirror remote, and do git svn init with the --prefix=mirror/.

    For me most of the work was rebuilding the IDE configuration files that were scattered around in the repository work tree.

    ReplyDelete
  8. It would be great if a simple 'git svn init' worked. I was just concerned about the metadata (.git/svn/{.metadata,refs}) and stuff that might take a lot of time to rebuild. In contrast, the pure git-counterpart 'git remote rename' feels fast and safe.

    If a simple 'git-svn init --prefix..' followed by a 'git svn rebase' will rebuild the metadata quickly and safely, that's great. Why shall I use "clean clones"? What kind of mess are referring to: e.g old stuff in .git/svn dir, or anything else?

    ReplyDelete
  9. @Gin: To be honest, I've never tried svn init twice in the same repository. It could be that it works fine, and the old configuration is properly overwritten. Maybe you have to remove the old svn-remote first, and/or the .git/svn folder. Try it out, and let me know. (but if you're worried about corrupting your working git-svn clone, make a copy of it first).

    ReplyDelete
  10. I tried this method a few times today and kept getting snagged when cloning the bare repo. Kept getting "warning: remote HEAD refers to nonexistent ref, unable to checkout."

    Inside the bare repo I found that HEAD pointed to refs/heads/master but the master branch did not exist. So I edited HEAD to use refs/heads/trunk and was able to proceed. Everything is looking fine so far.

    This was on MacOSX 10.6.8 with git 1.7.8.3.

    ReplyDelete
  11. After a little more reading and playing around it looks like I could just as well done the following to fix up the bare repo, to create the master branch and use trunk to set the start-point.

    $git branch master trunk

    ReplyDelete
  12. Hi James, thanks for commenting!

    Yes, after creating the bare repo, the "default branch" has to be set to be trunk like this:

    git symbolic-ref HEAD refs/heads/trunk

    I'm not really sure why I didn't include that step in this guide. I think I did do it in my JavaZone presentation, so I must have discovered that after I wrote the original guide. I'll see if I get a chance to work that in. Maybe write a completely overhauled guide some day :)

    ReplyDelete
  13. Anyone following along here may find this snippet useful:

    test $(git svn find-rev $(git log --all --oneline -n 1 |awk '{print $1}')) -ne $(svn info $(git config --local svn-remote.svn.url) |awk '/^Revision:/ {print $2}')

    I am using it with the Jenkins ScriptTrigger plugin to poll for when the svn Revision no longer matches the latest in the fetching repo, in order to automate the mirroring.

    Of course, if you have admin access to the svn repo you should probably just set up a commit hook there.

    FYI, the output looks like this:
    +++ git log --all --oneline -n 1
    +++ awk '{print $1}'
    ++ git svn find-rev 90fcd3c
    ++ awk '/^Revision:/ {print $2}'
    +++ git config --local svn-remote.svn.url
    ++ svn info file:///Users/james/src/svn-test
    + test 77 -ne 77

    ReplyDelete
  14. Tanks for the tip, James! If I ever get around to doing the big re-write of the big git-svn guide, I'll include this.

    ReplyDelete
  15. James Stansell21/1/12 16:15

    I'm glad you like it! Feel free to suggest improvements. In particular on MS/Win it would need cygwin, whereas rewriting in groovy could make it easier to use there.

    Thanks again for the very helpful git-svn guide!

    ReplyDelete
  16. James Stansell25/1/12 09:31

    Just an update about my jenkins job to fetch from svn and push to the bare repo. The snippet above was interesting but I think duplicated (poorly) what git svn fetch already does. So now I have moved the git svn fetch into the trigger script.

    cd /Users/james/src/git-svn/git-test-fetch
    fetchinfo="$(git svn fetch)"
    [[ $? -ne 0 || ${#fetchinfo} -eq 0 ]] && exit 1 #the fetch either failed or found no change
    read d <<< $(echo "$fetchinfo" |awk '/^r/ {print $1}') #description
    cat <<- _EOD
    < cause$z>< pre>
    $(git svn log --oneline --show-commit --remotes -${d// *r/:})

    $fetchinfo
    < /pre>< /cause>
    < description$z>$d< /description>
    _EOD

    ReplyDelete
  17. @tfnico, a few items you might want to consider for the big re-write of the big git-svn guide:

    1. Use a local svn repository if possible. Yes, svnsync is your friend. Splitting up the network transfer and the svn->git migration into two separate steps will greatly speed up the entire process, even for medium-size projects.

    2. Use an authors.txt file. Your git log will thank you!

    3. When you don't have admin access to the svn server, jenkins can be used to poll for changes, as I wrote above.

    Also, some related projects which are currently active:

    * reposurgeon: http://esr.ibiblio.org/?p=4071

    * svn-dump-fast-export: https://plus.google.com/u/0/106045714513828999745/posts/DAZjRkgJusw

    ReplyDelete

Post a Comment

Popular posts from this blog

Open source CMS evaluations

I have now seen three more or less serious open source CMS reviews. First guy to hit the field was Matt Raible ( 1 2 3 4 ), ending up with Drupal , Joomla , Magnolia , OpenCms and MeshCMS being runner-ups. Then there is OpenAdvantage that tries out a handful ( Drupal , Exponent CMS , Lenya , Mambo , and Silva ), including Plone which they use for their own site (funny/annoying that the entire site has no RSS-feeds, nor is it possible to comment on the articles), following Matt's approach by exluding many CMS that seem not to fit the criteria. It is somewhat strange that OpenAdvantage cuts away Magnolia because it "Requires J2EE server; difficult to install and configure; more of a framework than CMS", and proceed to include Apache Lenya in the full evaluation. Magnolia does not require a J2EE server. It runs on Tomcat just like Lenya does (maybe it's an idea to bundle Magnolia with Jetty to make it seem more lightweight). I'm still sure that OpenAdvant

Encrypting and Decrypting with Spring

I was recently working with protecting some sensitive data in a typical Java application with a database underneath. We convert the data on its way out of the application using Spring Security Crypto Utilities . It "was decided" that we'd be doing AES with a key-length of 256 , and this just happens to be the kind of encryption Spring crypto does out of the box. Sweet! The big aber is that whatever JRE is running the application has to be patched with Oracle's JCE  in order to do 256 bits. It's a fascinating story , the short version being that U.S. companies are restricted from exporting various encryption algorithms to certain countries, and some countries are restricted from importing them. Once I had patched my JRE with the JCE, I found it fascinating how straight forward it was to encrypt and decrypt using the Spring Encryptors. So just for fun at the weekend, I threw together a little desktop app that will encrypt and decrypt stuff for the given password

What I've Learned After a Month of Podcasting

So, it's been about a month since I launched   GitMinutes , and wow, it's been a fun ride. I have gotten a lot of feedback, and a lot more downloads/listeners than I had expected! Judging the numbers is hard, but a generous estimate is that somewhere around 2000-3000 have listened to the podcast, and about 500-1000 regularly download. Considering that only a percentage of my target audience actively listen to podcasts, these are some pretty good numbers. I've heard that 10% of the general population in the western world regularly listen to podcasts (probably a bit higher percentage among Git users), so I like to think I've reached a big chunk of the Git pros out there. GitMinutes has gathered 110 followers on Twitter, and 63, erm.. circlers on Google+, and it has received 117 +'es! And it's been flattr'ed twice :) Here are some of the things I learned during this last month: Conceptually.. Starting my own sandbox podcast for trying out everythin

The academical approach

Oops, seems I to published this post prematurely by hitting some Blogger keyboard shortcut. I've been sitting for some minutes trying to figure out how to approach the JavaZone talk mentioned in my previous blog-post. Note that I have already submitted an abstract to the comittee, and that I won't publish the abstract here in the blog. Now of course the abstract is pretty detailed on what the talk is going to be about, but I've still got some elbow room on how to "implement" the talk. I will use this blog as a tool to get my aim right on how to present the talk, what examples to include, what the slides should look like, and how to make it most straightforward and understandable for the audience. Now in lack of having done any presentations at a larger conference before, I'm gonna dig into what I learned at the University, which wasn't very much, but they did teach me how to write a research paper, a skill which I will adapt into creating my talk: The one

Git Stash Blooper (Could not restore untracked files from stash)

The other day I accidentally did a git stash -a , which means it stashes *everything*, including ignored output files (target, build, classes, etc). Ooooops.. What I meant to do was git stash -u , meaning stash modifications plus untracked new files. Anyhows, I ended up with a big fat stash I couldn't get back out. Each time I tried, I got something like this: .../target/temp/dozer.jar already exists, no checkout .../target/temp/core.jar already exists, no checkout .../target/temp/joda-time.jar already exists, no checkout .../target/foo.war already exists, no checkout Could not restore untracked files from stash No matter how I tried checking out different revisions (like the one where I actually made the stash), or using --force, I got the same error. Now these were one of those "keep cool for a second, there's a git way to fix this"situation. I figured: A stash is basically a commit. If we look at my recent commits using   git log --graph --