Last October I blogged about GUI testing being hard and I meant that without the requirement of running as part of a $g(Continuous Integration) (CI)environment. Performing GUI automation within a CI environment imposes it’s own unique challenges where tests have to be able to:
- Handle all cases where the GUI app fails to shutdown properly
- Handle unexpected UI from the GUI app (crash dialogs, Windows fatal error dialogs etc.)
- Terminate the application if it takes too long to respond
- Log output in the CI environments desired format (we had to add MSBuild output support to our test framework)
Additionally, the CI environment itself must allow GUI application execution. We’re using the console version of CruiseControl.NET so we can monitor the status messages that appear. The console version presents a bit of a unique problem in that all output is captured so we’re not able to see the test output during execution.
It’s been 83 days since the aforementioned post and 790+ CruiseControl.NET builds (not all of which succeeded) and our GUI automation is humming along nicely. The test has caught numerous problems along the way and improved the overall quality of builds getting to QA. The QA smoke test, the first test QA runs on a build to determine if it’s worth further inspection, has also been consistently passing at 100% for quite awhile now and it too is catching failures and saving valuable QA time and resources.
Of course there are many factors which have contributed to the overall success rate of our testing efforts for example:
- First and foremost we fixed 1,000’s of bugs in BDS 2006 including the numerous patches we’ve released
- We committed R&D resources to improve and maintain our test framework
- The model generator I mentioned here which allows us to have up-to-date models with each build
- We’ve enabled R&D to develop their own GUI tests suites
- We’re leveraging virtual PC’s to quickly and consistently test a broad range of IDE/VCL features
- Starts the BDS IDE
- Loads a copy of it’s own project group
- Recompiles all of the packages from the entire IDE then the BDS.EXE
- Sets breakpoints in the code
- Launches BDS IDE under the debugger
- Tests all sorts of debugger functionality
Now, that’s a serious test! Above is just a brief summary of steps and hardly does justice to all of the work the test actually performs but if you’ve ever written GUI automation I’m sure you can appreciate what’s involved in testing two copies of the same application where one instance is debugging the other not to mention running under CI. Chris and I spent quite a bit of time going over the finer details of how this was all going to work and I have to say he did a fabulous job of putting it all together. I’ll ping him and see if perhaps he can write a blog post about it.