Using Subversion's support for externals in the BDS repository

I realized I probably should have linked to the book on Subversion earlier in case you haven’t found it yet. Now, yesterday I blogged a bit about how the Delphi team’s SVN repository is setup, today I’ll discuss how SVN externals are used in the BDS repository. One thing externals allow you to do is link different repositories together.

The “TP” tree, the main BDS repository, is made up of a few different SVN repositories. One example of an external repository is the C++ compiler. The C++ compiler that’s used to build the Delphi compiler, among other things, is not built within the TP tree rather the binaries are provided via an external link. The main reasons being it provides the C++ compiler team the freedom to work at their own pace and not worry about affecting the larger BDS repository and helps reduce the build overhead for those of us building the BDS tree all the time which takes roughly 15 minutes on this machine. There are other such external links but pretty much fall under this same usage pattern.

If you’re a Delphi/C++Builder developer one use of externals could be for your third party components. Since you’re probably less/unlikely to modify that code you could create a separate repository which could feed your main repository with those binaries rather than constantly rebuilding static code. The same could be true of any utility packages, DLLs or .DCUs within your project that change infrequently. Be sure to refer to the book mentioned above for more of the specifics on SVN externals.

In the case of the IDE, all IDE, RTL and VCL packages are checked into the main repository and built during the normal build process. The IDE itself is very modular and many of the different features/packages can be build independently either from the command line with MSBuild or from the Project Manager. There is an IDE project group file checked in which contains a large number of the products packages making it easier to work on such a modular code base.

There’s lots more to talk about so I’ll continue chipping away if people are finding this of value.

Please let me know if you find this useful/interesting. Thanks!

12 thoughts on “Using Subversion's support for externals in the BDS repository

  1. It’s always useful to find out out the big/huge software projects are handled! Even thought my own projects are typically only a dozen files or so each, I’ve been converted to SVN and CC ever since CodeRage and after your posts… 🙂

  2. Externals are really useful.
    I’m still having one small issue with subversion. I have several projects that use the same ‘common’ directory. But when I want to make a TAG of the trunk it’s quite hard to link the current version of that common directory. What’s the best way of doing that?
    So far I started using externals to get that common directory in my project path (as a subdir).
    I can change the external property to a specific revision of the common directory, make a tag, and change it back to the head revision. I can also create a branch from the current project in my trunk. Adjust the external property and create a tag from the branch.
    Are there some kind of ‘standards’ to do this?
    Regards,
    Robin

  3. Robin,
    We also have multiple (well, dozens) apps that use our common library. I haven’t come across best practices for common code, but I’d imagine people treat it in a similar fashion as they do vendor code. i.e. App links to a tag of a vendor’s third-party library. I think the red-bean svn book describes this approach.
    Well, that is what we have started to do. Our common library is now treated as a seperate project. Bugs, features get added to our issue tracker for common. Applications then link to a release tag of common.
    Another book that people might be interested in is the Pragmatic Programmers SVN book: http://pragmaticprogrammer.com/titles/svn2/index.html

  4. Hey Nicolas,
    I just checked out the vendor method. Which I guess is probably the best way. We also have other third party libraries, these are currently installed on the developers computers, but not stored into subversion. Maybe that should be changed, so we can use the vendor method for 3rd party libraries too.
    I also checked out the release-branch method too. Now we create tags directly from the trunk. I think I will do some tests with that release-branch method. Maybe it can also solve our common directory issues.
    It’s another way of using subversion, but a much better way we are using subversion at this moment. Atleast we are using version control, a few years ago we used to be working on a shared drive on a server. That was what I call horrible. Just imagine working with 2 people or more on the same project (or using the same common directory). Files were overwritten all the time, changes were lost. Compile times were slow. Glad things have changed a bit.:)

  5. Keep up the articles on the build and version control process. I think a lot of people end up looking for real world examples and getting insights like this are great.
    I think anyone using svn should really give the book a quick read. Set aside an hour just to understand it and get a feeling for the things it can do besides just the basic update/commit cycle. There are lots of great ways to integrate it as well with other tools.
    A question: are you using any of the bugtraq properties for issue tracking? We started using them when we switched to using trac as our bug tracking/wiki solution and it’s been great. We actually use trac without binding it directly to subversion as we are a commercial project and as great as source code browsing is from trac it’s not really in our best interest 🙂

  6. I’ll also find the posts to be interesting and useful.
    One item that I have struggled with is how to correctly deal with third party visual components. For example my application uses the DevExpress grids and a number of other controls. Right now I don’t have anything for those components in my svn repository. If I did create a vendor branch how do I make sure that the correct versions of the packages are loaded into the IDE depending on the branch that I currently have checked out?
    I have also wondered if I did check the third party stuff in to svn if I would be able to do a checkout on a new machine and have a working IDE without running the installs for each third party component.
    Right now we only have two developers so we just make sure we always run the upgraded installs at the same time.

  7. Maybe when the BPL and DCP files are also included in subversion you can link those in your IDE. But ofcource that’s probably not the best solution. At work we use 2 different versions of Dev Express Quantum Grid (v3.2 and 4). They can be installed both at the same time. Another story is Asta components. We have several versions (2.60 / 3.00), some older projects use 2.60 and newer projects use 3.00. But for compatibilty reasons those 2.60 versions can’t be upgraded to 3.00. So for older projects 2.60 has to be installed (in designtime) and newer projects 3.00. And that’s not all. The internal company library relies on Asta. But uninstalling 3.00 and reinstalling 2.60 will break the company library. Switching can take half a day if something goes wrong. And most of the time it’s only for a few lines of code change. For most projects we try to do more creation in runtime instead of designtime so only path changes are necessary. But I think there must be a better way to do this.
    So far I started using cruisecontrol.net to build my projects in the background and publish the latest builds on the company server. That works great. No need to install design time packages. But ofcourse forms opened with the wrong component version can lead to corrupt DFMs like missing and unknown published properties.

  8. Mark,
    I’ve been mulling over that problem for a while now and even emailed Steve T about it. Currently there isn’t an ideal way of making sure you have the correct version (and only the correct version) of a third-party component installed when you check out a project from svn.
    One work around is to make sure all third-party libraries are installed on your system in a common location (i.e. same location on every dev computer). Make sure they are all installed in the IDE. For each application, add the libraries that are not required in the excludes section of the config file.
    We’ve started doing this at work, and it generally works fine. The only problem comes when a new library or new version of an existing library is added to the IDE. You have to make sure apps that don’t use these, include them in the excludes section.
    As far as checking in the third-party code into svn. Absolutely. We check in all of our third-party libraries into a vendor repository. Each library is tagged (REL-x.y.z) and each of our applications link to these tags. The main reason for doing so is that we can keep track of which apps use what libraries, and when we are doing automated builds, we can use these links to build against the third-party libaries. Also if you have runtime packages, it is very easy as part of your build process to include them with the app as their already available in the vendor folder of your project.

  9. I work with about 20 other developers. We use BDS 2006 with Delphi (Win32 target) today.
    I am converting about 50-60 applications to a modular package-based design (from monolithic executables). So far progress has been mostly smooth; there are nearly a million lines of code in about 40 BPLs.
    But there are difficulties:
    – until we completely evaluate the use of BPLs, build the install sets, and run through a QA cycle, we cannot deliver code linked in that state, so we have to keep the source compiling with packages and without packages. I wrote a wrapper around DCC32 to remove the -LU line from the project CFG files so I could turn off runtime packages from the command line.
    – creating a new package is very tedious. Every application’s BDSPROJ/cfg file has to be updated to include the new package in its runtime packages list. I wrote an application to do this for me. A centralized place to store a standard set of runtime packages would be ideal.
    – downloading and compiling different versions sometimes causes the IDE to unload design-time packages. This isn’t that big of a deal, however: we simply recompile and reinstall if we need to.
    – We frequently do full command-line builds on our developer machines before we complete a check-in to VSS. The packages are compiled during that build and must be output to a folder different than the one that the IDE uses because they are locked by the IDE.
    – Sometimes we cannot do a command line build while the IDE is open because it locks a few packages. Usually closing an active project fixes this.
    – Run | Run can be VERY slow because of QC 44629. We have developers waiting up to 4 minutes after selecting Run until our largest program starts.
    So any insight you can provide into how to manage such a large migration is definitely helpful and interesting to me, thanks a lot !!!

Comments are closed.