Skip to main content

Git-SVN Mirror without the annoying update-ref

This post is part of a series on Git and Subversion. To see all the related posts, screencasts and other resources, please click here. 

So no sooner than I had done my git-svn presentation at JavaZone, I got word of a slightly different Git-SVN mirror setup that makes it a bit easier to work with:

In short, my old recipe includes an annoying git update-ref step to keep the git-svn remote reference up to date with the central bare git repo. This new recipe avoids this, so we can simply use git svn dcommit  directly.

So, longer version, with the details. My original recipe is laid out in five steps:

  1. Clone a fresh Git repo from Subversion. This will be our fetching repo.
  2. Set up a bare repo.
  3. Configure pushing from the fetching repo to bare repo
  4. In the shoes of a developer, clone the repo
  5. Set up an SVN remote in the developer's repo
In the new approach, we redefine those last two steps:

(See the original post for how to do the first three.)

4. Clone the bare repo

Note the -o parameter that specifies that we want to call the origin remote "mirror":

tfnico@flint:~/sources/git/>git clone -o mirror ~/git-repos/websites.git/

We can now track the remote branches, and pull the latest changes as they come in to the bare repo.

tfnico@flint:~/sources/git/websites/>git checkout -t mirror/yksi
tfnico@flint:~/sources/git/websites/>git pull --rebase

(Remember to use --rebase to avoid merge commits in case you have local commits.)

When the time comes to push commits back to Subversion, we have to complete the last step.

5. Set up an SVN remote in the developer's repo

We'll do an svn-init with the same parameters as we cloned in the beginning, adding a prefix that matches the name of the remote "mirror":

tfnico@flint:~/sources/git/websites/>git svn init --prefix=mirror/ -s file:///svn-repos/company-repo/websites/

The point of the mirror/ prefix is that whenever we do a pull, the git-svn reference will be updated along with the normal remote reference. So now you can do git svn dcommit without any further ado.
Labels:

Comments

  1. Anonymous14/11/11 14:04

    Great, I came to a similar conclusions earlier - but haven't seen it written down and forgot it in the meantime as well (so kept using update-ref to keep the two pointers refs/remotes/trunk refs/svn/trunk refs/remotes/mirror/svn/trunk in sync).

    Have you run ran into any negative side-effects/gotchas with this --prefix setup since?

    ReplyDelete
  2. Thomas Ferris Nicolaisen 14/11/11 14:54

    Hi Gin, thanks for your comment!

    Generally, no, I haven't seen any real problems with the prefix setup.

    There is one tiny detail: If you accidentally push directly to the bare bare repo (instead of doing a dcommit), and you need to revert that by doing a forced push to remote/svn, you don't have any local reference any more to which commit the SVN HEAD is at. However, this is not really a problem if you are (a) careful, or (b) take measures to make the bare repo non-writable for developers.

    ReplyDelete
  3. Anonymous14/11/11 15:20

    Good point, thanks Thomas. I'd probably want to ensure that trouble doesn't strike (most our devs are new to git and might do mistakes). However, I'd prefer not making the whole mirror repo read-only (some of as are pushing pure-git branches there as well..), so will take a look at the gitolite setup: my impression is that it may be able to protect specific branches.

    ReplyDelete
  4. Thomas Ferris Nicolaisen 14/11/11 15:24

    Gin: Perhaps a more practical way to avoid accidents is to have each developer disable the push configuration in their local repositories:

    http://stackoverflow.com/questions/527833/how-to-configure-git-to-avoid-accidental-git-push

    ReplyDelete
  5. Thomas Ferris Nicolaisen 14/11/11 15:32

    Here's a nice one-liner for the above setup:

    git config remote.mirror.pushurl WARNING_USE_SVN_DCOMMIT_INSTEAD_OF_PUSH

    This will cause developers to see an error message when trying to push to the bare repo (mirror).

    ReplyDelete
  6. Anonymous14/11/11 15:40

    Good idea - thanks for that too!
    I'd still prefer to do it once (on the server), rather than setting it up on the growing numbers of developer boxes. I guess there is not easy way to share repo configuration (apart from nasty tricks to commit them too) - I'd be happy to be wrong one this.

    Also, do you think it's easy and fast to migrate to this -synced-prefix setup? (the repo is big)
    I guess I could do a 'git remote rename oldmirror svn' , where 'svn' is default; but I'd prefer that one to 'mirror' instead.. can't see anything obvious on man git-svn. Any ideas? thanks again.

    ReplyDelete
  7. Thomas Ferris Nicolaisen 14/11/11 15:52

    Hmm, I think that git-svn needs to be re-initialized (with the --prefix parameter), and to avoid any mess, I've done this in clean clones (so far).

    However, this all takes place on the developer's machines. There's no need to change anything with the fetching/bare repos. So the developers can clone their old repo locally (fastest), add the proper mirror remote, and do git svn init with the --prefix=mirror/.

    For me most of the work was rebuilding the IDE configuration files that were scattered around in the repository work tree.

    ReplyDelete
  8. Anonymous14/11/11 16:13

    It would be great if a simple 'git svn init' worked. I was just concerned about the metadata (.git/svn/{.metadata,refs}) and stuff that might take a lot of time to rebuild. In contrast, the pure git-counterpart 'git remote rename' feels fast and safe.

    If a simple 'git-svn init --prefix..' followed by a 'git svn rebase' will rebuild the metadata quickly and safely, that's great. Why shall I use "clean clones"? What kind of mess are referring to: e.g old stuff in .git/svn dir, or anything else?

    ReplyDelete
  9. Thomas Ferris Nicolaisen 14/11/11 20:12

    @Gin: To be honest, I've never tried svn init twice in the same repository. It could be that it works fine, and the old configuration is properly overwritten. Maybe you have to remove the old svn-remote first, and/or the .git/svn folder. Try it out, and let me know. (but if you're worried about corrupting your working git-svn clone, make a copy of it first).

    ReplyDelete
  10. James Stansell11/1/12 06:22

    I tried this method a few times today and kept getting snagged when cloning the bare repo. Kept getting "warning: remote HEAD refers to nonexistent ref, unable to checkout."

    Inside the bare repo I found that HEAD pointed to refs/heads/master but the master branch did not exist. So I edited HEAD to use refs/heads/trunk and was able to proceed. Everything is looking fine so far.

    This was on MacOSX 10.6.8 with git 1.7.8.3.

    ReplyDelete
  11. James Stansell11/1/12 06:35

    After a little more reading and playing around it looks like I could just as well done the following to fix up the bare repo, to create the master branch and use trunk to set the start-point.

    $git branch master trunk

    ReplyDelete
  12. Thomas Ferris Nicolaisen 13/1/12 10:35

    Hi James, thanks for commenting!

    Yes, after creating the bare repo, the "default branch" has to be set to be trunk like this:

    git symbolic-ref HEAD refs/heads/trunk

    I'm not really sure why I didn't include that step in this guide. I think I did do it in my JavaZone presentation, so I must have discovered that after I wrote the original guide. I'll see if I get a chance to work that in. Maybe write a completely overhauled guide some day :)

    ReplyDelete
  13. James Stansell21/1/12 08:07

    Anyone following along here may find this snippet useful:

    test $(git svn find-rev $(git log --all --oneline -n 1 |awk '{print $1}')) -ne $(svn info $(git config --local svn-remote.svn.url) |awk '/^Revision:/ {print $2}')

    I am using it with the Jenkins ScriptTrigger plugin to poll for when the svn Revision no longer matches the latest in the fetching repo, in order to automate the mirroring.

    Of course, if you have admin access to the svn repo you should probably just set up a commit hook there.

    FYI, the output looks like this:
    +++ git log --all --oneline -n 1
    +++ awk '{print $1}'
    ++ git svn find-rev 90fcd3c
    ++ awk '/^Revision:/ {print $2}'
    +++ git config --local svn-remote.svn.url
    ++ svn info file:///Users/james/src/svn-test
    + test 77 -ne 77

    ReplyDelete
  14. Thomas Ferris Nicolaisen 21/1/12 12:15

    Tanks for the tip, James! If I ever get around to doing the big re-write of the big git-svn guide, I'll include this.

    ReplyDelete
  15. James Stansell21/1/12 16:15

    I'm glad you like it! Feel free to suggest improvements. In particular on MS/Win it would need cygwin, whereas rewriting in groovy could make it easier to use there.

    Thanks again for the very helpful git-svn guide!

    ReplyDelete
  16. James Stansell25/1/12 09:31

    Just an update about my jenkins job to fetch from svn and push to the bare repo. The snippet above was interesting but I think duplicated (poorly) what git svn fetch already does. So now I have moved the git svn fetch into the trigger script.

    cd /Users/james/src/git-svn/git-test-fetch
    fetchinfo="$(git svn fetch)"
    [[ $? -ne 0 || ${#fetchinfo} -eq 0 ]] && exit 1 #the fetch either failed or found no change
    read d <<< $(echo "$fetchinfo" |awk '/^r/ {print $1}') #description
    cat <<- _EOD
    < cause$z>< pre>
    $(git svn log --oneline --show-commit --remotes -${d// *r/:})

    $fetchinfo
    < /pre>< /cause>
    < description$z>$d< /description>
    _EOD

    ReplyDelete
  17. James Stansell26/1/12 04:27

    @tfnico, a few items you might want to consider for the big re-write of the big git-svn guide:

    1. Use a local svn repository if possible. Yes, svnsync is your friend. Splitting up the network transfer and the svn->git migration into two separate steps will greatly speed up the entire process, even for medium-size projects.

    2. Use an authors.txt file. Your git log will thank you!

    3. When you don't have admin access to the svn server, jenkins can be used to poll for changes, as I wrote above.

    Also, some related projects which are currently active:

    * reposurgeon: http://esr.ibiblio.org/?p=4071

    * svn-dump-fast-export: https://plus.google.com/u/0/106045714513828999745/posts/DAZjRkgJusw

    ReplyDelete

Post a Comment

Popular posts from this blog

Open source CMS evaluations

I have now seen three more or less serious open source CMS reviews. First guy to hit the field was Matt Raible ( 1 2 3 4 ), ending up with Drupal , Joomla , Magnolia , OpenCms and MeshCMS being runner-ups. Then there is OpenAdvantage that tries out a handful ( Drupal , Exponent CMS , Lenya , Mambo , and Silva ), including Plone which they use for their own site (funny/annoying that the entire site has no RSS-feeds, nor is it possible to comment on the articles), following Matt's approach by exluding many CMS that seem not to fit the criteria. It is somewhat strange that OpenAdvantage cuts away Magnolia because it "Requires J2EE server; difficult to install and configure; more of a framework than CMS", and proceed to include Apache Lenya in the full evaluation. Magnolia does not require a J2EE server. It runs on Tomcat just like Lenya does (maybe it's an idea to bundle Magnolia with Jetty to make it seem more lightweight). I'm still sure that OpenAdvant...
6 comments
Read more

Encrypting and Decrypting with Spring

I was recently working with protecting some sensitive data in a typical Java application with a database underneath. We convert the data on its way out of the application using Spring Security Crypto Utilities . It "was decided" that we'd be doing AES with a key-length of 256 , and this just happens to be the kind of encryption Spring crypto does out of the box. Sweet! The big aber is that whatever JRE is running the application has to be patched with Oracle's JCE  in order to do 256 bits. It's a fascinating story , the short version being that U.S. companies are restricted from exporting various encryption algorithms to certain countries, and some countries are restricted from importing them. Once I had patched my JRE with the JCE, I found it fascinating how straight forward it was to encrypt and decrypt using the Spring Encryptors. So just for fun at the weekend, I threw together a little desktop app that will encrypt and decrypt stuff for the given password...
Post a Comment
Read more

The Git Users Mailing List

A year ago or so, I came across the Git-user mailing list (aka. "Git for human beings"). Over the year, I grew a little addicted to helping people out with their Git problems. When the new git-scm.com webpage launched , and the link to the mailing list had disappeared, I was quick to ask them to add it again . I think this mailing list fills an important hole in the Git community between: The Git developer mailing list git@vger.kernel.org  - which I find to be a bit too hard-core and scary for Git newbies. Besides, the Majordomo mailing list system is pretty archaic, and I personally can't stand browsing or searching in the Gmane archives. The IRC channel #git on Freenode, which is a bit out-of-reach for people who never experienced the glory days of IRC. Furthermore, when the channel is busy, it's a big pain to follow any discussion. StackOverflow questions tagged git , these come pretty close, but it's a bit hard to keep an overview of what questio...
Post a Comment
Read more

Git tools for keeping patches on top of moving upstreams

At work, we maintain patches for some pretty large open source repositories that regularly release new versions, forcing us to update our patches to match. So far, we've been using basic Git operations to transplant our modifications from one major version of the upstream to the next. Every time we make such a transplant, we simply squash together the modifications we made in the previous version, and land it as one big commit into the next version. Those who are used to very stringent keeping of Git history may wrinkle their nose at this, but it is a pragmatic choice. Maintaining modifications on top of the rapidly changing upstream is a lot of work, and so far we haven't had the opportunity to figure out a more clever way to do it. Nor have we really suffered any consequences of not having an easy to read history of our modifications - it's a relatively small amount of patches, after all. With a recent boost in team size, we may have that opportunity. Also the need for be...
1 comment
Read more

Managing dot-files with vcsh and myrepos

Say I want to get my dot-files out on a new computer. Here's what I do: # install vcsh & myrepos via apt/brew/etc vcsh clone https://github.com/tfnico/config-mr.git mr mr update Done! All dot-files are ready to use and in place. No deploy command, no linking up symlinks to the files . No checking/out in my entire home directory as a Git repository. Yet, all my dot-files are neatly kept in fine-grained repositories, and any changes I make are immediately ready to be committed: config-atom.git     -> ~/.atom/* config-mr.git     -> ~/.mrconfig     -> ~/.config/mr/* config-tmuxinator.git       -> ~/.tmuxinator/* config-vim.git     -> ~/.vimrc     -> ~/.vim/* config-bin.git        -> ~/bin/* config-git.git               -> ~/.gitconfig config-tmux.git       -> ~/.tmux.conf     config...
Post a Comment
Read more