Automated testing of windows applications is hard

I spent 5+ years working in Delphi QA where a wrote a large number of automated test suites using our internal automated testing framework known as Zombie. The past few months I’ve been spending a great deal of time working to improve Zombie and build an IDE smoke test that executes at the end of the build process whenever an engineer checks into our repository. To write the initial version of the test and get it integrated into our build process took one day but that was by far the easiest part.

For the past several weeks I’ve been diagnosing the failures and sending lots of department-wide email (yes, I’m sure people are sick of hearing from me) whenever a checkin causes smoke test to fail. I’ve been improving both the test and the underlying framework to ensure it has the best possible chance of completing without hanging no matter what happens in the IDE during test execution which includes IDE hangs, rapidly appearing nested error dialogs, Windows OS level error dialogs etc.

Recently I added a thread to the framework which monitors the IDE as it’s being tested for any critical error dialogs like AV’s, Asserts or Windows “Send Error Report” dialogs. The thread catches these errors subsequently blocks further test execution, logs a failure, dumps any useful content from the dialogs like call stack info to the log file and finally terminates the IDE in preparation for the next test. The test keeps going so as to provide as much information as possible as the quality of the overall product. I’ll see if Product Management will allow me to post a video of our smoke test running so people can get a better understanding of what it looks like.

So, if all the stars line up just right we get a clean build that’s been smoke tested and my CCTray app chimes in to let me know “Yet another successful build”.

3 thoughts on “Automated testing of windows applications is hard

  1. I find this really great !!! I wish you sucess in all this and get a even better Delphi. For me "testing" is completely new (I know about its existence but never applied it) so I wish also that the product management let you publish a video to see it all in images.. This last times my apps (that usually worked as a charm) are having on precise computers some AV’s I can’t reproduce.. so maybe testing would be the solution.. I will maybe try NUnit…

  2. Far from sick of hearing from you, I love coming in each day to my "smoketest" emails, gives me a sense of progression each and every day….way cool.
    Cheers Tim.

  3. Hi, i really need to know exactly what typies of testing can i run them on windows application by using atoumated testing tools and if that possible what is the tool???

Comments are closed.