Developing GUI test automation for a Continuous Integration environment

Last October I blogged about GUI testing being hard and I meant that without the requirement of running as part of a $g(Continuous Integration) (CI)environment. Performing GUI automation within a CI environment imposes it’s own unique challenges where tests have to be able to:

  • Handle all cases where the GUI app fails to shutdown properly
  • Handle unexpected UI from the GUI app (crash dialogs, Windows fatal error dialogs etc.)
  • Terminate the application if it takes too long to respond
  • Log output in the CI environments desired format (we had to add MSBuild output support to our test framework)

Additionally, the CI environment itself must allow GUI application execution. We’re using the console version of CruiseControl.NET so we can monitor the status messages that appear. The console version presents a bit of a unique problem in that all output is captured so we’re not able to see the test output during execution.

It’s been 83 days since the aforementioned post and 790+ CruiseControl.NET builds (not all of which succeeded) and our GUI automation is humming along nicely. The test has caught numerous problems along the way and improved the overall quality of builds getting to QA. The QA smoke test, the first test QA runs on a build to determine if it’s worth further inspection, has also been consistently passing at 100% for quite awhile now and it too is catching failures and saving valuable QA time and resources.

Of course there are many factors which have contributed to the overall success rate of our testing efforts for example:

  • First and foremost we fixed 1,000’s of bugs in BDS 2006 including the numerous patches we’ve released
  • We committed R&D resources to improve and maintain our test framework
  • The model generator I mentioned here which allows us to have up-to-date models with each build
  • We’ve enabled R&D to develop their own GUI tests suites
  • We’re leveraging virtual PC’s to quickly and consistently test a broad range of IDE/VCL features

One of the coolest Zombie tests I’ve ever seen is one written by Chris Hesik a few months ago. It tests both the managed and unmanaged debuggers by debugging the IDE in itself. The test:

  1. Starts the BDS IDE
  2. Loads a copy of it’s own project group
  3. Recompiles all of the packages from the entire IDE then the BDS.EXE
  4. Sets breakpoints in the code
  5. Launches BDS IDE under the debugger
  6. Tests all sorts of debugger functionality

Now, that’s a serious test! Above is just a brief summary of steps and hardly does justice to all of the work the test actually performs but if you’ve ever written GUI automation I’m sure you can appreciate what’s involved in testing two copies of the same application where one instance is debugging the other not to mention running under CI. Chris and I spent quite a bit of time going over the finer details of how this was all going to work and I have to say he did a fabulous job of putting it all together. I’ll ping him and see if perhaps he can write a blog post about it.

1 thought on “Developing GUI test automation for a Continuous Integration environment

  1. Hello, Stive!
    First of all I’d want to thank you for your Delphi related suite. Thanks a lot. You help people using Delphi. I use it since 1989. Of course, I used Turbo Pascal then.
    Now I have a question about namespaces in .NET, to which I seek answer for some time:
    Turbo Pascal and Delphi always had {$I filename} directive. In Pascal/Delphi a Unit was not always: "one unit"="one file". One unit could consist of several files. If they get compiled the result would be one namespace (one compiled unit). I think unit in Pascal/Delphi always was like namespace in .NET and it could consist of several files.
    If so what made people force Borland (now CodeGear) reintroduce namespaces? What did they want,"File-New-Namespace"?
    What was the problem?
    (Sorry for my broken english)

Comments are closed.