Skip to main content

Git-SVN Mirror for multiple branches

This post is part of a series on Git and Subversion. To see all the related posts, screencasts and other resources, please click here

This extends the posts where I explained how to set up a git-svn mirror for a single directory.

NOTE: If you just want to use Git against a SVN repo on your own, stop reading ,now, and stick to the git-svn basics. However, if you want a setup where you can share a Git repository with colleagues and friends while still interfacing with Subversion, keep reading.

I'll show how to set up a git-svn mirror for a standard Subversion project with trunk, branches and tags. It's a bit like the single directory mirror, but in order to keep all branches in sync, it's a bit more fiddling. The good part is that this setup enables us to cherry-pick commits from one branch to the other. This is slightly smoother than using svn merge.

First of all, let's repeat how our Subversion and Git-repositories look physically (roughly the same as in previous posts):
We'll set this up in the following order.

  1. Clone a fresh Git repo from Subversion. This will be our fetching repo.
  2. Set up a bare repo.
  3. Configure pushing from the fetching repo to bare repo
  4. In the shoes of a developer, clone the repo
  5. Set up an SVN remote in the developer's repo


You'll need to have a Subversion repository url that points to a project with trunk and branches like this (we'll skip tags for the purpose of this how-to):

file:///svn-repos/company-repo/websites/trunk
file:///svn-repos/company-repo/websites/branches/yksi
file:///svn-repos/company-repo/websites/branches/kaksi

We'll do our development in trunk, and every time we make a release we branch out (so far we've released yksi and kaksi).

1. Clone the Subversion repo

Do the following:

tfnico@flint:~/git-repos/>git svn clone -s file:///svn-repos/company-repo/websites

This will run for a while (depending on the size of your repo), and create a websites directory. This will be our fecthing repo. The -s flag means "standard layout", and tells git-svn to scan the trunk and branches structure.

We can poll for changes from the SVN repo like this:

tfnico@flint:~/git-repos/websites>git svn fetch

2. Set up a bare repo

Create the bare repo like this:

tfnico@flint:~/git-repos/>git init --bare websites.git

In the next step we'll configure the fetching repo to push changes into this one.

3. Configure fetching repo to push changes

Now there are a lot of different ways to do this, but this is the easiest workflow I've found so far.

First, go into the fetching repo and set up the newly created bare repo as a remote repo with the alias 'origin':

tfnico@flint:~/git-repos/websites/>git remote add origin ../websites.git/

Now, if you look inside the fetching repo's config in : git-repos/websites/.git/config
you'll see the following entry:


[remote "origin"]
    url = ../websites.git/
    fetch = +refs/heads/*:refs/remotes/origin/*

Now modify the above so it looks like the config below:

[remote "origin"]
    url = ../websites.git/
    fetch = +refs/remotes/*:refs/remotes/origin/*
    push = refs/remotes/*:refs/heads/*


I won't explain the details of this, but we've changed the fetch operations to update the remote branches. We've also added a push configuration, saying that all remote branches should be pushed.

We can now push the remote branches into the bare repo:


tfnico@flint:~/git-repos/websites>git push origin


We can update with changes from SVN and push to the repo in one go:


tfnico@flint:~/git-repos/websites>git svn fetch; git push origin


You'll want to automate these steps somewhat, so that they occur regularly in a cron-job, or as an svn commit hook.

Update (21.08.2012): Set trunk as the default branch to be cloned:

tfnico@flint:~/git-repos/websites>cd ../websites.git
tfnico@flint:~/git-repos/websites.git>git symbolic-ref HEAD refs/heads/trunk

We are now done with setting up the infrastructure. Next up: The developer's clone.

Update (25.09.2011): Instead of doing the last two steps as described below, I recommend you rather jump to my newer recipe that makes things a bit easier

4. Clone the bare repo

Pretty straight forward:

tfnico@flint:~/sources/git/>git clone ~/git-repos/websites.git/

We can now track the remote branches, and pull the latest changes as they come in to the bare repo.

tfnico@flint:~/sources/git/websites/>git checkout -t origin/yksi
tfnico@flint:~/sources/git/websites/>git pull --rebase

(Remember to use --rebase to avoid merge commits in case you have local commits.)

When the time comes to push commits back to Subversion, we have to complete the last step.

5. Set up an SVN remote in the developer's repo

We'll do an svn-init with the same parameters as we cloned in the beginning, like this:

tfnico@flint:~/sources/git/websites/>git svn init -s file:///svn-repos/company-repo/websites/

Before we can do this in the yksi branch, we need to update the pointer to git-svn's awareness of Subversion:

tfnico@flint:~/sources/git/websites/>git update-ref refs/remotes/yksi refs/remotes/origin/yksi


The above command is the biggest PITA with this Git-SVN mirror setup:


After each pull/rebase, before you can push/dcommit back to SVN, you have to update-ref on the respective branch/trunk.


If you don't, you'll get a message like this:

tfnico@flint:~/sources/git/websites/>git svn dcommit
Committing to file:///svn-repos/company-repo/websites/branches/yksi ...
Transaction is out of date: File '/websites/branches/yksi/README.txt' is out of date at /opt/local/libexec/git-core/git-svn line 573


So, to correct the above, repeat this:

tfnico@flint:~/sources/git/websites/>git update-ref refs/remotes/yksi refs/remotes/origin/yksi

It's a bit of hassle, but I'm sure you'll figure out a neat little script for doing this in no time.


Update (20/11): I've figured out a script for you! (or rather the friendly folks on #git did):

Put this in your .gitconfig aliases:


upci = !git update-ref refs/remotes/$(git branch | grep '^*' | awk '{print $2}') refs/remotes/origin/$(git branch|grep '^*'|awk '{print $2}') && git svn dcommit


And from now on use that alias for pushing to SVN.


That brings us through the basics. I would've loved to show this off in a screencast, but unfortunately my Camtasia trial has expired, and I'm not so keen on dishing out euros for this little hobby. And there is no good free screencasting software for Mac.

Comments

  1. Hello Thomas,

    I've found your git posts very helpful! Thanks a lot for putting them up.

    I'm working on setting up a bridge as you describe here. I've got a question about problems I have with the developer's repo.

    Once I run 'git update-ref refs/remotes/trunk refs/remotes/origin/trunk', I get messages about refname 'trunk' being ambiguous. The ref is required to push changes, but unless I delete it immediately afterwards it causes problems.

    Before deleting the extra ref, 'git branch -a' shows this:

    * trunk
    remotes/origin/trunk
    remotes/trunk


    I'm using git 1.7.5. Do you get the same results?

    Thanks, David

    ReplyDelete
  2. Hi David!

    Thanks for the feedback :)

    This ambiguous message is just a warning. I should have mentioned it in the post, but I've been living with this warning all the time now, and it hasn't caused any harm.

    ReplyDelete
  3. Thomas,

    Thanks for the clarification.

    Have you tried experimenting with changing the svn-remote stanza in .git/config?

    It is currently set to

    [svn-remote "svn"]
    url = http://svnserver/svn/main
    fetch = project/trunk:refs/remotes/trunk
    branches = project/branches/*:refs/remotes/*
    tags = project/tags/*:refs/remotes/tags/*

    It seems that if it was set to

    [svn-remote "svn"]
    url = http://svnserver/svn/main
    fetch = project/trunk:refs/remotes/origin/trunk
    branches = project/branches/*:refs/remotes/origin/*
    tags = project/tags/*:refs/remotes/origin/tags/*

    the additional refs (e.g. remotes/trunk) wouldn't be needed.

    - David

    ReplyDelete
  4. A quick update: I've been working for a while with the above change (on trunk only, so far) and it is working great!

    ReplyDelete
  5. Hi David, thanks for commenting again.

    Nope, I haven't experimented with changing the branch refspecs. I kind of like having the branches inside remotes, cause then I know they come from SVN (they're not just normal remote branches).

    It's an interesting tip though.. I'll have to try this out some time.

    ReplyDelete
  6. hi thomas, i got your workflow to work, but having an issue about svn branches that are deleted. my pure git repo and, subsequently, git dev repo, does not purge the deleted svn branches from the remotes list. is there any way to update the workflows/config in order to also know when svn branches are deleted? thanks!

    ReplyDelete
  7. Hi Joe,

    Sorry for taking so long answering.

    In short: No, git-svn doesn't detect branch deletion. However, I think if you do delete a branch in the repository, git-svn will not re-create it (at least not before there is a new commit in the branch and git-svn fetch is run).

    So I think you'll have to do branch deletion manually after deleting it in SVN.

    ReplyDelete
  8. Hi Thomas,

    Very nice setup. However it isn't clear for me how to prevent developers from pushing to trunk in bare-mirror repository directly? Of course it is possible to make whole repo read-only and allow only fetch-repo push there. But this also prevents developers from pushing into other branches in bare-mirror which is ok and definitely a nice feature to have.

    ReplyDelete
  9. Hi mrsmith,

    The only way I can think is to have the bare-mirror repository as two different remotes, for example "mirror", and "origin" (it's the same remote repository, it just has two different names).

    You can set your master/trunk branch (or any other branch which belongs to SVN) to use the remote "mirror", and then block pushes to this remote:

    git config remote.mirror.pushurl WARNING_USE_SVN_DCOMMIT_INSTEAD_OF_PUSH

    Other branches you can set to use the remote "origin", which have no restrictions for pushing.

    ReplyDelete
    Replies
    1. Since this is user-side changes, it is just another form of jentlemen agreement :)

      I thought about second mirror after "bare" for devs branches and no write acess to bare for everyone. So even second mirror is broken by accedental push "bare" will still ok.

      Also, have you ever run into trouble whith direct pushes?

      Delete
    2. Of course there's the occasional accidental push into the svn branches (even done some myself). It is quickly fixed by doing a push --force back to the last revision that came from svn, and notify the team so they can reset locally if they managed to pull in the mean time.

      Delete
  10. Hi Thomas,

    After I do a git svn dcommit, I have a merged file that is very difficult to track the diff. The following is an example. There is only 2 lines different but the diff file doesn't show it in an easy way to see the diff. Imagine I have a merged file with 1000 lines and only one line changed. Is this true? Is there any way to avoid it?

    -
    - This is sample text
    -
    +
    +
    + This is a new text
    + This is sample text
    + New text was added
    +

    desired behavior:
    -
    -
    + This is a new text
    - This is sample text
    + new text was added
    -

    ReplyDelete

Post a Comment

Popular posts from this blog

Open source CMS evaluations

I have now seen three more or less serious open source CMS reviews. First guy to hit the field was Matt Raible ( 1 2 3 4 ), ending up with Drupal , Joomla , Magnolia , OpenCms and MeshCMS being runner-ups. Then there is OpenAdvantage that tries out a handful ( Drupal , Exponent CMS , Lenya , Mambo , and Silva ), including Plone which they use for their own site (funny/annoying that the entire site has no RSS-feeds, nor is it possible to comment on the articles), following Matt's approach by exluding many CMS that seem not to fit the criteria. It is somewhat strange that OpenAdvantage cuts away Magnolia because it "Requires J2EE server; difficult to install and configure; more of a framework than CMS", and proceed to include Apache Lenya in the full evaluation. Magnolia does not require a J2EE server. It runs on Tomcat just like Lenya does (maybe it's an idea to bundle Magnolia with Jetty to make it seem more lightweight). I'm still sure that OpenAdvant

Encrypting and Decrypting with Spring

I was recently working with protecting some sensitive data in a typical Java application with a database underneath. We convert the data on its way out of the application using Spring Security Crypto Utilities . It "was decided" that we'd be doing AES with a key-length of 256 , and this just happens to be the kind of encryption Spring crypto does out of the box. Sweet! The big aber is that whatever JRE is running the application has to be patched with Oracle's JCE  in order to do 256 bits. It's a fascinating story , the short version being that U.S. companies are restricted from exporting various encryption algorithms to certain countries, and some countries are restricted from importing them. Once I had patched my JRE with the JCE, I found it fascinating how straight forward it was to encrypt and decrypt using the Spring Encryptors. So just for fun at the weekend, I threw together a little desktop app that will encrypt and decrypt stuff for the given password

What I've Learned After a Month of Podcasting

So, it's been about a month since I launched   GitMinutes , and wow, it's been a fun ride. I have gotten a lot of feedback, and a lot more downloads/listeners than I had expected! Judging the numbers is hard, but a generous estimate is that somewhere around 2000-3000 have listened to the podcast, and about 500-1000 regularly download. Considering that only a percentage of my target audience actively listen to podcasts, these are some pretty good numbers. I've heard that 10% of the general population in the western world regularly listen to podcasts (probably a bit higher percentage among Git users), so I like to think I've reached a big chunk of the Git pros out there. GitMinutes has gathered 110 followers on Twitter, and 63, erm.. circlers on Google+, and it has received 117 +'es! And it's been flattr'ed twice :) Here are some of the things I learned during this last month: Conceptually.. Starting my own sandbox podcast for trying out everythin

The academical approach

Oops, seems I to published this post prematurely by hitting some Blogger keyboard shortcut. I've been sitting for some minutes trying to figure out how to approach the JavaZone talk mentioned in my previous blog-post. Note that I have already submitted an abstract to the comittee, and that I won't publish the abstract here in the blog. Now of course the abstract is pretty detailed on what the talk is going to be about, but I've still got some elbow room on how to "implement" the talk. I will use this blog as a tool to get my aim right on how to present the talk, what examples to include, what the slides should look like, and how to make it most straightforward and understandable for the audience. Now in lack of having done any presentations at a larger conference before, I'm gonna dig into what I learned at the University, which wasn't very much, but they did teach me how to write a research paper, a skill which I will adapt into creating my talk: The one

Git Stash Blooper (Could not restore untracked files from stash)

The other day I accidentally did a git stash -a , which means it stashes *everything*, including ignored output files (target, build, classes, etc). Ooooops.. What I meant to do was git stash -u , meaning stash modifications plus untracked new files. Anyhows, I ended up with a big fat stash I couldn't get back out. Each time I tried, I got something like this: .../target/temp/dozer.jar already exists, no checkout .../target/temp/core.jar already exists, no checkout .../target/temp/joda-time.jar already exists, no checkout .../target/foo.war already exists, no checkout Could not restore untracked files from stash No matter how I tried checking out different revisions (like the one where I actually made the stash), or using --force, I got the same error. Now these were one of those "keep cool for a second, there's a git way to fix this"situation. I figured: A stash is basically a commit. If we look at my recent commits using   git log --graph --