svn:externals for DRY

This is quite interesting, I happened to get a chance to work with ‘svn:externals’ property and using it turned out to be a good learning exercise. BTW, I planned to use svn:externals to clear off some duplications in one of the repositories.

The story goes like this, I had a friend who manages a svn repository with couple of projects. For some reason, I happened to check-out the projects from his repository and I found some duplication on the first sight which I felt, it could be avoided.

The repo structure was some thing like this, (simpler version)

        - src
        - test
        - lib (set of third-party libraries)
        - resources
        - src
        - test
        - lib (set of third-party libraries as same in Project1/lib)
         - resources
         - # a similar structure, not to mention, the lib folder is seen again duplicated.

Ideally, I should have gone for using some awesome dependency management tool available (Apache Maven or Apache Ivy), but, since its going to take in lot of changes to implement the same, I decided to take smaller steps.

  • First, externalize the common library and avoid duplication.
  • Second, refactor the build files to start using a dependancy management tool.

First step first:
Here is a simple tutorial I framed in order to show how an svn:externals can be used to avoid this kind of duplication in simpler steps. You can try this out on your local machine by setting up a simple repository emulating the above structure and see how it works.

For those, who doesn’t know what is svn:externals, please do check out more details on the same in SVN red book. But I am damn sure, you would get some basic understanding by end of this blogpost.

Ok, let us start with creating a simple svn repository in our local sandbox.
To create a repository, follow these steps

/home/user/$: mkdir myrepository
/home/user/$: cd myrepository
/home/user/myrepository/$: svnadmin create project1 (P.S. I am assuming you have downloaded and installed svn in your localsandbox)

Now its time to import some code into the empty repo. So create a folder named ‘project1’ with the ‘src’, ‘lib’, ‘test’ and ‘resources’ subfolders. Add some couple of library files (jar files may be) into the lib folder.
Assuming your folder is at, /home/user/project, its time to check-in this structure into the newly created svn repo as shown below:

/home/user$: svn import ./project1/ file:///home/user/myrepository/project1 -m "message:pushing the project1 to the svn repo"

With the above command, you are telling svn that push all the files & folders under ‘project1’ into the svn repo located at “file:///home/user/myrepository/project1” with a message.
Congratulations! you have created a new svn repo with some files and folders in it.

Similarly, you can create another project with name ‘project2’, create an equivalent repository using ‘svnadmin’ as shown above. Make sure that, this project’s ‘lib’ folder to has got the same library files as you had in ‘project1’.
This is truly a sin, I have just asked you to duplicate the library files. I promise, will not do that anymore. Lets go ahead and see how can we fix this duplication.

The solution would be, to have the library files in one common place in the repository and have both the projects ‘project1 and project2’ use it. This is where svn:externals come into handy.
Before going ahead using the svn:externals, lets create a common repository in svn to store the library files.
Here are the steps you need to follow:

/home/user/$: cd myrepository
/home/user/myrepository/$: svnadmin create library

Now have the library files copied to a folder at a some location like, ‘/home/user/common-lib’. Now go ahead and import the library files into the newly created ‘library’ repo as follows:

/home/user$: svn import ./common-lib/ file:///home/user/myrepository/library -m "message:pushing the common lib files to the library repo"

Its time to remove the duplications, In order to do that, just checkout the ‘project1’ and ‘project2’ into a folder like ‘/home/user/localworkingcopy’. The following svn commands should do that for you.

/home/user/$: mkdir localworkingcopy
/home/user/$: cd localworkingcopy
/home/user/localworkingcopy/$: svn co file:///home/user/myrepository/project1
/home/user/localworkingcopy/$: svn co file:///home/user/myrepository/project2

Now, remove the duplication and re-commit the changes into svn repo.

/home/user/localworkingcopy/$: cd project1
/home/user/localworkingcopy/project1/$: svn delete lib/
/home/user/localworkingcopy/project1/$: svn status (this should show the local change as 'lib' folder is marked for deletion)
/home/user/localworkingcopy/project1/$: svn commit . -m "message: removed the lib folder"

Similarly follow the same above mentioned steps to delete the ‘lib’ folder from ‘/home/user/localworkingcopy/project2’ as well.
Now, both our projects are devoid of its required libraries, lets make them to fetch their libraries from the newly created ‘library’ repo using svn:externals. To do that, you need to set some value to the svn:externals property using the command as follows:

/home/user/localworkingcopy/project1/$: svn propedit svn:internals .

#notice the ‘.’ at the last, its required because, I am mentioning to the svn repo that, I am setting some value for svn:externals pertaining only for/within the folder, ‘project1’. The above command should openup an editor (in my case, it was vim).
Just put the following in the first line and save & close the editor.

lib file:///home/user/myrepository/library

The above line is actually a key-value pair where ‘lib’ is the key and the svn URL, ‘file:///home/user/myrepository/library’ is the value. All that it implies is, ‘svn update’ to check-out the content from the ‘library’ repo under ‘project1/lib’ folder. When we do an svn update, a ‘lib’ folder will be created and the respective contents will be checked-out. And on every consecutive svn updations, this ‘lib’ folder will also be updated based on any changes in the ‘library’ repository.
Note: You can set as many key-value pairs for svn:externals property as you want each key-value in an individual line within the vim editor. In our case we just need one.

This is one final, important step you need to follow. The step is to persist/commit the changes you have made for svn:externals property into the svn. So lets commit the changes by the following command.

/home/user/localworkingcopy/project1/$: svn commit . -m "set the svn:externals to look up the common library folder"

Follow the same steps of setting svn:externals property for ‘/home/user/localworkingcopy/project2’ as well.

Now, its time to taste the fruit of all our hard-work. Run svn update command to check-out the library files into the ‘project1’ and ‘project2’ as follows:

/home/user/localworkingcopy/project1/$: svn update
/home/user/localworkingcopy/project2/$: svn update

At this juncture you should be able to see the lib folder created with all its content in both the projects.
But there is a small catch here, even though the new ‘library’ repo is likely to see any quite often changes, whenever I run ‘svn update’ for ‘project1/project2’, I going to check the changes in ‘lib’ folder as well which might cause some delay during every updation.
If I am sure that, there would be less frequent changes to the common ‘library’ repo, I can use the following command to avoid the continous look-up for changes from within ‘project1’ and ‘project2’ using the following command.

/home/user/localworkingcopy/project1/$: svn update --ignore-externals
/home/user/localworkingcopy/project2/$: svn update --ignore-externals

That is all it is. Finally after all these changes, the repo would look like the following:

      # common library files are located here.
    -Project1 (with svn:externals property set to point the common 'library' repo)
      - src
      - test
      - resources
    -Project2 (with svn:externals property set to point the common 'library' repo)
      - src
      - test
      - resources
    -Project3 (with svn:externals property set to point the common 'library' repo)
       #with its contents and without the lib folder

I hope you got a basic idea about how svn:externals property works. 🙂


About this entry