I've blogged a lot about how CodeGear uses Continuous Integration (CI) in it's development process for Delphi/BDS. What want to talk about here is how you can use these same tools for your development process.

First, is the issue of version control and for that we use Subversion otherwise known as SVN. SVN is open source and fortunately there is a convenient "1 click setup" described as follows:

Svn1ClickSetup takes a user through the steps necessary to install the Subversion command-line utilities and TortoiseSVN, as well as creating a repository and initial project.

This setup really lives up to it's name so if you're looking for a fast and easy way to get started with some great source control software this is what you're looking for. The install includes the TortioseSVN client which is an SVN client that's implemented as a Windows Shell Extension and is what the majority of developers at CodeGear use.

A Continuous Integration Server

That may sound a bit more daunting than perhaps it should but basically it's the software that pulls from your repository, builds your project and reports any errors. The Delphi development team uses CruiseControl.NET server and CCTray for client side notifications of build failures both of which are open source projects and even though it has ".NET" on the end of its name the server isn't restricted to building .NET only projects.

Once again the installation is very straight forward and if you have IIS installed you'll even be able to monitor your CI server via a browser including forcing a new build.

Ok, now that you've read all about it have a look at the video. Or can download it here.

Here is the ccnet.config file used in the video, and no, "steve" is not my normal password.

[UPDATED: April 23, 2007] Fixed video and .zip file download links.

Jud Cole, the DUnit Project Admin just announced that there is a new version of DUnit available for download.

Recent additions to DUnit 9.3.x

• FASTMM support, including optional memory leak checking on a per test basis, resulting in improved execution speed of many tests.
• New DUnitW32 project group for Delphi 2005+ with new projects DUnitW32,
  DUnitTestLibW32 and UnitTestsW32.
• New DUnit4Net project group for Delphi 8+ with existing project UnitTests4Net.
  Automatic MadExcept support for stack tracing in TestFramework.pas.
• Optional checking that each test case calls at least one CheckXXX method.
• Optional detection of each test case that overrides the global GUI test case settings.
• Optional halting of a repeated test on the first failure.
• Carried the GUITestRunner.pas changes to the main form creation over to NGUITestRunner.pas and QGUITestRunner.pas.
• Added a display refresh timer to GUITestRunner to ensure display update in long unit tests.
• A couple of minor bug fixes.

Great work and congrats to the DUnit crew!

Tomorrow, SD Times is hosting a free online seminar at 1pm PST called "Bridging the Gap: Successfully Managing Agile Projects in the Waterfall Enterprise" that I plan to watch. Up until last year the Delphi team had used Waterfall, or perhaps more accurately a Waterfall-like, development process which has seen both ends of the success spectrum. Over the past few months I've posted a number of times about using Agile and discussed some of the things the Delphi team has been doing related to Agile, continuous integration and automation. I recently remarked that I'm not entirely sold on Agile and perhaps this presentation will address some of the issues.

Anyway, I thought it was worth mentioning as it seems interesting and I'm always looking for fresh ideas to improve our development process.

Last October I blogged about GUI testing being hard and I meant that without the requirement of running as part of a Continuous Integration (CI)environment. Performing GUI automation within a CI environment imposes it's own unique challenges where tests have to be able to:

• Handle all cases where the GUI app fails to shutdown properly
• Handle unexpected UI from the GUI app (crash dialogs, Windows fatal error dialogs etc.)
• Terminate the application if it takes too long to respond
• Log output in the CI environments desired format (we had to add MSBuild output support to our test framework)

Additionally, the CI environment itself must allow GUI application execution. We're using the console version of CruiseControl.NET so we can monitor the status messages that appear. The console version presents a bit of a unique problem in that all output is captured so we're not able to see the test output during execution.

It's been 83 days since the aforementioned post and 790+ CruiseControl.NET builds (not all of which succeeded) and our GUI automation is humming along nicely. The test has caught numerous problems along the way and improved the overall quality of builds getting to QA. The QA smoke test, the first test QA runs on a build to determine if it's worth further inspection, has also been consistently passing at 100% for quite awhile now and it too is catching failures and saving valuable QA time and resources.

Of course there are many factors which have contributed to the overall success rate of our testing efforts for example:

• First and foremost we fixed 1,000's of bugs in BDS 2006 including the numerous patches we've released
• We committed R&D resources to improve and maintain our test framework
• The model generator I mentioned here which allows us to have up-to-date models with each build
• We've enabled R&D to develop their own GUI tests suites
• We're leveraging virtual PC's to quickly and consistently test a broad range of IDE/VCL features

One of the coolest Zombie tests I've ever seen is one written by Chris Hesik a few months ago. It tests both the managed and unmanaged debuggers by debugging the IDE in itself. The test:

1. Starts the BDS IDE
2. Loads a copy of it's own project group
3. Recompiles all of the packages from the entire IDE then the BDS.EXE
4. Sets breakpoints in the code
5. Launches BDS IDE under the debugger
6. Tests all sorts of debugger functionality

Now, that's a serious test! Above is just a brief summary of steps and hardly does justice to all of the work the test actually performs but if you've ever written GUI automation I'm sure you can appreciate what's involved in testing two copies of the same application where one instance is debugging the other not to mention running under CI. Chris and I spent quite a bit of time going over the finer details of how this was all going to work and I have to say he did a fabulous job of putting it all together. I'll ping him and see if perhaps he can write a blog post about it.

In an earlier post about Agile I mentioned that I'm on the fence about it, I said that largely because of the number of meetings I have each week. I'm feeling tied to the Outlook calendar and the constant chime of a meeting reminder. I've never used the full Outlook, though did use Outlook Express for a long time but when it fell victim to the Internet Explorer update schedule I gave up on it and moved to Mozilla's Thunderbird mail client which I've been using since well before the 1.0 release.

The main problem is I'm involved in three distinct areas ASP.NET, VCL and test automation each of which have different deliverables. At the beginning of the Highlander project after a few lengthy discussions we decided to move Zombie, our GUI automation framework, into the core BDS source tree and invest R&D resources into improving and maintaining the code base. Having worked on Zombie for six years while I was in QA I was in a good position to hit the ground running though it meant troubleshooting whatever problems QA ran into until things stablized. The troubleshooting phase wreaked havoc with other work I'd been assigned during the sprint because it was a higher priority to deal with automation issues.

Here are some things that would make me feel more Agile:

• Fewer emails (particularly those related to new meetings, changed meetings etc.)
• Fewer standups
• I miss being co-located with QA their now on the second floor (R&D is on the third)
• More visibility into areas of the product I don't work on
• Better task progress tracking software we're currently using a web client and I believe a thick client UI would be much more efficient.

One thing that would help is an Exchange calendar tray application that requires a tiny amount of system resource but reminds me of meetings. Does the MSN chat client have this? I Googled all over and couldn't find what I was looking for so I've settled on using the Outlook web client. I have a Delphi utility application I wrote that I consider to be my swiss army knife and I tweaked it so the IE window running the Outlook client isn't in the foreground it becomes transparent. Additionally, it refreshes the browser every few minutes cirrcumventing the auto logoff timer thus keeping me logged in all day.

Since the beginning of the Highlander (vNext BDS) development cycle the team has been using Agile and at this point, it's safe to say I'm still on the fence about it. The decision to use Agile wasn't quick nor was it made on the first day of the project, rather it came after numerous discussions and actual training sessions with an Agile instructor/consultant. Now, I'm not blaming the consultant and in fact, if anything the instruction was useful for us to improve our development process for reasons unrelated to learning Agile. I believe there are projects which would/could benefit from a strictly Agile approach however, given our team size, the disparate skill sets our staff has, the size and complexity of the IDE I think we're likely to have to reexamine and adapt our use of Agile which isn't a surprise and I'm sure at least some people in the Agile community would say that that's what Agile is all about.

Prior to this year I'd say our development process was a modified version of waterfall which has seen it's ups and downs. One point I'd make though is that when there are changes in the engineering staff, the technology, the market, the business and the management is it fair to blame the process when things go wrong? I'll be the first to admit our process can and needs further improvement and we're committed to and actively working on those improvements today. In fact, I've blogged a lot about our development process much of which has evolved even since last year and not just because of our use of Agile.

Like most engineers my team has lots of ideas about how things could be improved and in many cases those ideas are acted upon independently and our process incrementally improves. Perhaps one of the best things to come from all the focus on Agile and XP is simply the fact it has gotten engineers actively interested in thinking about the development process. Of course, it hasn't hurt that the discussion has been accompanied by lots of tools for things like unit testing and continuous integration which I believe really helps drive engineering interest.

It's interesting, I think most of the process improvements our team has enjoyed this past year occurred organically driven, not by management, but by individual engineers with a desire to utilize the maturing set of tools that are available today. I believe one critical turning point was our single minded focus on quality for the BDS 2006 release. I believe the release itself was high quality but since November of last year we've released at least seven updates more than triple what we normally ship. It's this same desire to ship quality code that has fueled much of the process changes we've put in place in the past 12 months.

I haven't really talked about why I'm on the fence about Agile and at some point I'll fill in the blanks but one thing is clear, IMO the BDS development team is actively moving in the right direction.

If you're new to reading my blog check out my Zombie posts regarding our automated testing efforts and have a look at the video I posted of our R&D smoke test for BDS.

I was talking to Dan Miser awhile back and he mentioned that it might be cool to blog about how the Delphi R&D team actually develops Delphi itself. I've been on the team a long time, well before D1 shipped and I've seen a lot of changes to our development process over the years, some gradual and others not. Right now, as we work on the next release I think we're in a very interesting phase of the evolution of our development process and I'll write about a few specific areas and mention some of the tools we're using. Perhaps a few other R&D folks will chime in and offer some additional "color commentary".

Building the IDE

You might wonder, does Borland actually use the tools they produce for their own development efforts? The answer is a resounding "YES!". The Delphi team in particular has always been a huge proponent of what Microsofties call "dog fooding" which is the act of using what you produce. Not only is the IDE itself written in Delphi, QA's patented automated testing framework is written in Delphi, our Bug Tracking system (RAID) both the SOAP client and server are written in Delphi (running against an Interbase server) and prior to StarTeam our version control client was an application written and maintained by the Delphi R&D team. In fact, there are numerous utilities used in our build process also written in Delphi as well as countless little personal utilities that nearly everyone in QA and R&D have developed for themselves over the years.

Anyway, the product is built using a typical "make" process from the command line which supports mutliple targets allowing us to build the multitude of SKU's. Once built day to day development work is performed using a "working" build of the IDE and a convienent "make work" target updates this local "working" build. Currently, the build takes about 20 minutes for a debug developer build on my 2.4GHz machine (with 1GB of RAM although that's not really critical when building from the command line) and is made up of over 25,000 files.

Delivery to Integration and QA

We have an automated build server which sends out email notifications to everyone who checked in since the last successful build in the event the build breaks. Since we now have developers around the globe working on the build it's incredibly important to keep it building successfully and any build breaks are usually tracked down very quickly. Once a the build is successful it is then promoted to the Integration team where they build the product into a complete installation for delivery to QA. Most of the time QA tests the product using the Architect SKU since it has all the "bells and whistles" and as we near certain milestones they'll run various SKU tests to ensure that all the different versions include all the necessary parts.

What's New and Interesting

To me the "new and interesting" parts of our development process which is still evolving is how we're leveraging the technology that Borland has acquired over the past few years, namely StarTeam and Caliber. Since the acquisitions we've been slowly working both products (change management and requirements management) and processes into our, let's call it, experience laden development process. We've made some great strides and I think we've come a long way since Anders Hejlsberg's Notepad based list of work items with dashes and "X's" and Gary Whizin's spiral notebook.

Using CaliberRM to develop and maintain our list of product requirements has given us a much better process of scoping product development and assigning work to individual engineers. I think it's measurably improved the transparency of our development process for our management team so that they can more effectively guide product development. Additionally, CaliberRM has provided a great place for us to capture requirements for features we don't have time for in the current release allowing us to maintain a list of good ideas from one version to the next. If only we ship a product with all the features we currently have scoped in Caliber. :)

Then there is StarTeam which not only allowed us to free up valuable Delphi R&D resources used maintaining an old tool but has also given us a bunch of features we'd all been wanting for years. It's nice to have a full development team working on a tool that your team uses everyday. Besides I don't think the StarTeam folks could ask for better "dog fooding" of their code so Borland get's double benefit from the Delphi team using it.

That's all for now. Let me know if you find this at all interesting and I'd be glad to share more about the inner workings of our team