I spent 5+ years working in Delphi QA where a wrote a large number of automated test suites using our internal automated testing framework known as Zombie. The past few months I've been spending a great deal of time working to improve Zombie and build an IDE smoke test that executes at the end of the build process whenever an engineer checks into our repository. To write the initial version of the test and get it integrated into our build process took one day but that was by far the easiest part.
For the past several weeks I've been diagnosing the failures and sending lots of department-wide email (yes, I'm sure people are sick of hearing from me) whenever a checkin causes smoke test to fail. I've been improving both the test and the underlying framework to ensure it has the best possible chance of completing without hanging no matter what happens in the IDE during test execution which includes IDE hangs, rapidly appearing nested error dialogs, Windows OS level error dialogs etc.
Recently I added a thread to the framework which monitors the IDE as it's being tested for any critical error dialogs like AV's, Asserts or Windows "Send Error Report" dialogs. The thread catches these errors subsequently blocks further test execution, logs a failure, dumps any useful content from the dialogs like call stack info to the log file and finally terminates the IDE in preparation for the next test. The test keeps going so as to provide as much information as possible as the quality of the overall product. I'll see if Product Management will allow me to post a video of our smoke test running so people can get a better understanding of what it looks like.
So, if all the stars line up just right we get a clean build that's been smoke tested and my CCTray app chimes in to let me know "Yet another successful build".