Skip to main content

Git-SVN Mirror without the annoying update-ref

This post is part of a series on Git and Subversion. To see all the related posts, screencasts and other resources, please click here

So no sooner than I had done my git-svn presentation at JavaZone, I got word of a slightly different Git-SVN mirror setup that makes it a bit easier to work with:

In short, my old recipe includes an annoying git update-ref step to keep the git-svn remote reference up to date with the central bare git repo. This new recipe avoids this, so we can simply use git svn dcommit  directly.

So, longer version, with the details. My original recipe is laid out in five steps:

  1. Clone a fresh Git repo from Subversion. This will be our fetching repo.
  2. Set up a bare repo.
  3. Configure pushing from the fetching repo to bare repo
  4. In the shoes of a developer, clone the repo
  5. Set up an SVN remote in the developer's repo
In the new approach, we redefine those last two steps:

(See the original post for how to do the first three.)

4. Clone the bare repo

Note the -o parameter that specifies that we want to call the origin remote "mirror":

tfnico@flint:~/sources/git/>git clone -o mirror ~/git-repos/websites.git/

We can now track the remote branches, and pull the latest changes as they come in to the bare repo.

tfnico@flint:~/sources/git/websites/>git checkout -t mirror/yksi
tfnico@flint:~/sources/git/websites/>git pull --rebase

(Remember to use --rebase to avoid merge commits in case you have local commits.)

When the time comes to push commits back to Subversion, we have to complete the last step.

5. Set up an SVN remote in the developer's repo

We'll do an svn-init with the same parameters as we cloned in the beginning, adding a prefix that matches the name of the remote "mirror":

tfnico@flint:~/sources/git/websites/>git svn init --prefix=mirror/ -s file:///svn-repos/company-repo/websites/

The point of the mirror/ prefix is that whenever we do a pull, the git-svn reference will be updated along with the normal remote reference. So now you can do git svn dcommit without any further ado.

Comments

  1. Great, I came to a similar conclusions earlier - but haven't seen it written down and forgot it in the meantime as well (so kept using update-ref to keep the two pointers refs/remotes/trunk refs/svn/trunk refs/remotes/mirror/svn/trunk in sync).

    Have you run ran into any negative side-effects/gotchas with this --prefix setup since?

    ReplyDelete
  2. Hi Gin, thanks for your comment!

    Generally, no, I haven't seen any real problems with the prefix setup.

    There is one tiny detail: If you accidentally push directly to the bare bare repo (instead of doing a dcommit), and you need to revert that by doing a forced push to remote/svn, you don't have any local reference any more to which commit the SVN HEAD is at. However, this is not really a problem if you are (a) careful, or (b) take measures to make the bare repo non-writable for developers.

    ReplyDelete
  3. Good point, thanks Thomas. I'd probably want to ensure that trouble doesn't strike (most our devs are new to git and might do mistakes). However, I'd prefer not making the whole mirror repo read-only (some of as are pushing pure-git branches there as well..), so will take a look at the gitolite setup: my impression is that it may be able to protect specific branches.

    ReplyDelete
  4. Gin: Perhaps a more practical way to avoid accidents is to have each developer disable the push configuration in their local repositories:

    http://stackoverflow.com/questions/527833/how-to-configure-git-to-avoid-accidental-git-push

    ReplyDelete
  5. Here's a nice one-liner for the above setup:

    git config remote.mirror.pushurl WARNING_USE_SVN_DCOMMIT_INSTEAD_OF_PUSH

    This will cause developers to see an error message when trying to push to the bare repo (mirror).

    ReplyDelete
  6. Good idea - thanks for that too!
    I'd still prefer to do it once (on the server), rather than setting it up on the growing numbers of developer boxes. I guess there is not easy way to share repo configuration (apart from nasty tricks to commit them too) - I'd be happy to be wrong one this.

    Also, do you think it's easy and fast to migrate to this -synced-prefix setup? (the repo is big)
    I guess I could do a 'git remote rename oldmirror svn' , where 'svn' is default; but I'd prefer that one to 'mirror' instead.. can't see anything obvious on man git-svn. Any ideas? thanks again.

    ReplyDelete
  7. Hmm, I think that git-svn needs to be re-initialized (with the --prefix parameter), and to avoid any mess, I've done this in clean clones (so far).

    However, this all takes place on the developer's machines. There's no need to change anything with the fetching/bare repos. So the developers can clone their old repo locally (fastest), add the proper mirror remote, and do git svn init with the --prefix=mirror/.

    For me most of the work was rebuilding the IDE configuration files that were scattered around in the repository work tree.

    ReplyDelete
  8. It would be great if a simple 'git svn init' worked. I was just concerned about the metadata (.git/svn/{.metadata,refs}) and stuff that might take a lot of time to rebuild. In contrast, the pure git-counterpart 'git remote rename' feels fast and safe.

    If a simple 'git-svn init --prefix..' followed by a 'git svn rebase' will rebuild the metadata quickly and safely, that's great. Why shall I use "clean clones"? What kind of mess are referring to: e.g old stuff in .git/svn dir, or anything else?

    ReplyDelete
  9. @Gin: To be honest, I've never tried svn init twice in the same repository. It could be that it works fine, and the old configuration is properly overwritten. Maybe you have to remove the old svn-remote first, and/or the .git/svn folder. Try it out, and let me know. (but if you're worried about corrupting your working git-svn clone, make a copy of it first).

    ReplyDelete
  10. I tried this method a few times today and kept getting snagged when cloning the bare repo. Kept getting "warning: remote HEAD refers to nonexistent ref, unable to checkout."

    Inside the bare repo I found that HEAD pointed to refs/heads/master but the master branch did not exist. So I edited HEAD to use refs/heads/trunk and was able to proceed. Everything is looking fine so far.

    This was on MacOSX 10.6.8 with git 1.7.8.3.

    ReplyDelete
  11. After a little more reading and playing around it looks like I could just as well done the following to fix up the bare repo, to create the master branch and use trunk to set the start-point.

    $git branch master trunk

    ReplyDelete
  12. Hi James, thanks for commenting!

    Yes, after creating the bare repo, the "default branch" has to be set to be trunk like this:

    git symbolic-ref HEAD refs/heads/trunk

    I'm not really sure why I didn't include that step in this guide. I think I did do it in my JavaZone presentation, so I must have discovered that after I wrote the original guide. I'll see if I get a chance to work that in. Maybe write a completely overhauled guide some day :)

    ReplyDelete
  13. Anyone following along here may find this snippet useful:

    test $(git svn find-rev $(git log --all --oneline -n 1 |awk '{print $1}')) -ne $(svn info $(git config --local svn-remote.svn.url) |awk '/^Revision:/ {print $2}')

    I am using it with the Jenkins ScriptTrigger plugin to poll for when the svn Revision no longer matches the latest in the fetching repo, in order to automate the mirroring.

    Of course, if you have admin access to the svn repo you should probably just set up a commit hook there.

    FYI, the output looks like this:
    +++ git log --all --oneline -n 1
    +++ awk '{print $1}'
    ++ git svn find-rev 90fcd3c
    ++ awk '/^Revision:/ {print $2}'
    +++ git config --local svn-remote.svn.url
    ++ svn info file:///Users/james/src/svn-test
    + test 77 -ne 77

    ReplyDelete
  14. Tanks for the tip, James! If I ever get around to doing the big re-write of the big git-svn guide, I'll include this.

    ReplyDelete
  15. James Stansell21/1/12 16:15

    I'm glad you like it! Feel free to suggest improvements. In particular on MS/Win it would need cygwin, whereas rewriting in groovy could make it easier to use there.

    Thanks again for the very helpful git-svn guide!

    ReplyDelete
  16. James Stansell25/1/12 09:31

    Just an update about my jenkins job to fetch from svn and push to the bare repo. The snippet above was interesting but I think duplicated (poorly) what git svn fetch already does. So now I have moved the git svn fetch into the trigger script.

    cd /Users/james/src/git-svn/git-test-fetch
    fetchinfo="$(git svn fetch)"
    [[ $? -ne 0 || ${#fetchinfo} -eq 0 ]] && exit 1 #the fetch either failed or found no change
    read d <<< $(echo "$fetchinfo" |awk '/^r/ {print $1}') #description
    cat <<- _EOD
    < cause$z>< pre>
    $(git svn log --oneline --show-commit --remotes -${d// *r/:})

    $fetchinfo
    < /pre>< /cause>
    < description$z>$d< /description>
    _EOD

    ReplyDelete
  17. @tfnico, a few items you might want to consider for the big re-write of the big git-svn guide:

    1. Use a local svn repository if possible. Yes, svnsync is your friend. Splitting up the network transfer and the svn->git migration into two separate steps will greatly speed up the entire process, even for medium-size projects.

    2. Use an authors.txt file. Your git log will thank you!

    3. When you don't have admin access to the svn server, jenkins can be used to poll for changes, as I wrote above.

    Also, some related projects which are currently active:

    * reposurgeon: http://esr.ibiblio.org/?p=4071

    * svn-dump-fast-export: https://plus.google.com/u/0/106045714513828999745/posts/DAZjRkgJusw

    ReplyDelete

Post a Comment

Popular posts from this blog

Managing dot-files with vcsh and myrepos

Say I want to get my dot-files out on a new computer. Here's what I do:

# install vcsh & myrepos via apt/brew/etc
vcsh clone https://github.com/tfnico/config-mr.git mr
mr update

Done! All dot-files are ready to use and in place. No deploy command, no linking up symlinks to the files. No checking/out in my entire home directory as a Git repository. Yet, all my dot-files are neatly kept in fine-grained repositories, and any changes I make are immediately ready to be committed:

config-atom.git
    -> ~/.atom/*

config-mr.git
    -> ~/.mrconfig
    -> ~/.config/mr/*

config-tmuxinator.git  
    -> ~/.tmuxinator/*

config-vim.git
    -> ~/.vimrc
    -> ~/.vim/*

config-bin.git   
    -> ~/bin/*

config-git.git          
    -> ~/.gitconfig

config-tmux.git  
    -> ~/.tmux.conf    

config-zsh.git
    -> ~/.zshrc

How can this be? The key here is to use vcsh to keep track of your dot-files, and its partner myrepos/mr for operating on many repositories at the same time.

I discovere…

Working in Teams over Working as Individuals

I recentlypostedthis sketch on Twitter:

Thanks to a few mighty retweets, it gathered a lot of views (9000 impressions, whatever that means). While that's fun and all, I still felt a bit sad that such an awfully simple insight can garner much more popularity than a thorough blog post that I put some hours into.

So, rather than let Twitter get away with this, I'll steal my own content back into the blog :)

The thread went like this:

Pondering how to battle individualism in companies. For some, it is counter-intuitive that teams can be more responsive, faster and even more accountable than single individuals.

Having "teams" in place is no guarantee that team work is happening. Be wary of too large teams, "I/me/mine", personal contact details instead of team point of contact. Good team is sailing crew, not galley slaves.

Beware heroes, go-to persons, calling in favors and other shadow handling of work. Real teams make the work explicit, both requests/needs and re…

Encrypting and Decrypting with Spring

I was recently working with protecting some sensitive data in a typical Java application with a database underneath. We convert the data on its way out of the application using Spring Security Crypto Utilities. It "was decided" that we'd be doing AES with a key-length of 256, and this just happens to be the kind of encryption Spring crypto does out of the box. Sweet!

The big aber is that whatever JRE is running the application has to be patched with Oracle's JCE in order to do 256 bits. It's a fascinating story, the short version being that U.S. companies are restricted from exporting various encryption algorithms to certain countries, and some countries are restricted from importing them.

Once I had patched my JRE with the JCE, I found it fascinating how straight forward it was to encrypt and decrypt using the Spring Encryptors. So just for fun at the weekend, I threw together a little desktop app that will encrypt and decrypt stuff for the given password and sa…

The End of GitMinutes (my podcast)

I'm just about ship GitMinutes episode 46, which is going to be the final episode. I'll just paste the outro script here, as it sums up the sentimental thoughts pretty well:

I’m happy to have finally finished [publishing the last episodes from Git-Merge 2017], just in time before Git-Merge 2018 takes place in March. I won’t be going there myself, so I’m counting on someone else to pick up the mic there.

It’s sad to be shipping this one as it is probably the last GitMinutes episode ever. To go a bit down memory lane, 6 years ago, my daughter was born, and as I used a little of that paternity leave to set up my podcasting infrastructure and produce the first few episodes. Initially it was just going to be 10 episodes and call the experiment finished. Instead, I got to 46 episodes, the last dozen or so lazily tailing the last few Git-Merge conferences.

To every one of my guests, thank you so much again for coming on to share your passion in this little niche of computer science a…

Joining eyeo: A Year in Review

It's been well over a year since I joined eyeo. And 'tis the season for yearly reviews, so...

It's been pretty wild. So many times I thought "this stuff really deserves a bloggin", but then it was too inviting to grab onto the next thing and get that rolling.

Instead of taking a deep dive into some topic already, I want to scan through that year in review and think for myself, what were the big things, the important things, the things I achieved, and the things I learned. And then later on, if I ever get around to it, grab one of these topics and elaborate in a dedicated blog-post. Like a bucket-list of the blog posts that I should have written. Here goes:
How given no other structures, silos will grow by themselves This was my initial shock after joining the company. Only a few years after taking off as a startup, the hedges began growing, seemingly almost by themselves, and against the will of the founders. I've worked in silos, and in companies without the…