Steve Trefethen
Contact me
About Me View my LinkedIn profile

Powered by discountASP.NET
referal ID: sdtref
Why recommend discountASP.NET?
Need consulting?
Need Consulting?

Spread Thunderbird

Disclaimer

The posts on this weblog are provided AS IS with no warranties, and confer no rights. The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.

Using TestComplete Stores and Checkpoints

September 11 2009 9:52PM

image I’ve recorded a video for Falafel TV illustrating how to use TestComplete’s Stores and Checkpoints features including v7’s new Checkpoint Wizard dialog. In the video I demonstrate recording a Keyword test and leveraging various checkpoints to validate the test has performed as expected. The checkpoints illustrated in this video include:

  • Object
  • Table
  • File

If you have questions or have suggestions for future videos leave a comment. Also, if you’re looking for training or consulting on TestComplete feel free to contact me or call the Falafel office at (831) 462-0457.

One more note, Falafel is hosting a TestComplete Online Training Summit Oct 5-7th or November 9-11th which you can learn more about and sign up for here.

Other videos:
Distributed Testing Using TestComplete
HTTP Load Testing Using TestComplete

Digg It!DZone It!StumbleUponTechnoratiRedditDel.icio.usNewsVineFurlBlinkList

Tags: , , Automation | Videos

TestComplete and automating dynamic JavaScript menus

August 13 2009 8:41AM

TestComplete Online TrainingIf you read my blog regularly you know I conduct TestComplete training where we teach people how to use the product and get the most out of their automation investment. One of the issues that comes up regularly is how to deal with dynamic content on web pages such as popup menus implemented using JavaScript. I’ve blogged before about automation being hard and when it comes to web applications and dynamic display of HTML driven by JavaScript the challenge can be quite steep particularly for someone first learning GUI automation.

Automating Web Applications

GUI automation for web applications can be very different than testing standard Windows applications and there are some tools and techniques that can really help to resolve these particular challenges. Let’s take a look at an example.

Let’s there’s a dynamic JavaScript menu control on your application like this:

 image

The challenge for a recorder with this example can be mouse hover behavior of the menu. Hovering the mouse over the menu items activates the submenus automatically and some recording tools don’t capture that information. TestComplete records selection of the Orders item as follows in JScript:

function Test1()
{
  var  iexplore;
  var  page;
  TestedApps.iexplore.Run(1, true);
  iexplore = Aliases.iexplore;
  page = iexplore.pageHttpTrainingFalafelComDynami;
  page.ToURL("http://training.falafel.com/dynamiccontent/");
  page.formForm1.panelMenu1n3items.table.cell.tableMenu16Menu115.cell.linkOrder.Click();
  //Please wait until download completes: "http://training.falafel.com/dynamiccontent/NewItem.aspx"
  iexplore.pageHttpTrainingFalafelComDynami1.Wait();
}

Playing back this recording results in error indicating that TestComplete was trying to click outside the bounds of an object which in this case happens to be the Orders item but since it never appears on screen it’s impossible to click on it at all. Fortunately, there is an alternative which is to write a function that will handle menu selection for any item within the menu control using a familiar syntax to address the menu items.

However, there’s the minor issue of understanding exactly what it takes to develop such a routine and that means digging into the HTML of the page and coming up with a mechanism to uniquely identify the various menu items. To aid this process I leverage the Internet Explorer Developer Toolbar or the Developer Tools add-in which allows us to dig into the page right from within IE. Let’s take a look:

If you examine the href property of the Customer menu item you can see that it contains a string that’s unique, ‘Add\\New\\Customer’. In fact, the entire menu hierarchy has a similar structure which means our routine could search for items with that attribute, hover the mouse over each subitem and finally click the Orders items. Of course, if you examine the Orders item its href will be an actual URL an not a JavaScript call so we’ll need to find that item using it’s innerText property. Ok, now let’s look at the code, which doesn’t include any error checking:

function AspNetMenuSelect(page, menuitem)
{
  var items = menuitem.split("|");
  var itemstr = "";
  var count = 0;
  while(count < items.length - 1)
  {
    itemstr = itemstr + items[count];
    item = page.NativeWebObject.Find("href", "*" + itemstr + "*", "a");
    itemstr = itemstr + "\\\\";   // Add\\New\\Customer
    item.HoverMouse();
    count++;
  }
  item = page.NativeWebObject.Find("innerText", items[count], "a");
  item.Click();
  page.Wait();
}

The function is called as follows:

AspNetMenuSelect(page, “Add|New|Customer|Order”);

Where “page” represents Sys.Process(“iexplore”).Page(“*”). If you’re not familiar with TestComplete it exposes a “Page” object off of Internet Explorer (and Firefox) allowing a test to access elements of a web page.

The second parameter is a “|” separated string of menu items to select. The code breaks the string up into an array, searches the page looking for an anchor tag that has an href with the specified value and hovers the mouse over it. The process then repeats, building up the href the value we identified above until the last parent menu item is reached. The final piece is to locate the last menu item using the innerText property, click on it then wait for a page load to complete.

If you have a test automation tool, any tool, go ahead and record this test case where you click on the item highlighted above using a non-journaling mode and see how well it plays back. Then, post a comment below and let me know the outcome I’d be really interested in seeing some of the other tools compare. I will note that Selenium had no problem recording this example though it uses an entirely different technique.

Admittedly, it’s not terribly satisfying when you record a test and your tool of choice fails to replay it properly, a situation that can understandably leave people frustrated. It’s important to grasp the limitations of your recording software as I’ve yet to find one that works flawlessly in all cases. However, in the case of TestComplete, through scripting, we can dig in at a lower level and solve problem.

imageNow, if you made it this far which I hope you did because you can watch a video of this whole process on Falafel Software’s new tv.falafel.com website here.

Ok, so that was a bit of tease… :-)

Digg It!DZone It!StumbleUponTechnoratiRedditDel.icio.usNewsVineFurlBlinkList

Tags: , ,

GUI Test Automation with Ranorex Studio

March 25 2009 6:19AM

Today I got a demo of a tool called Ranorex Studio, a test automation product built using the SharpDevelop IDE. Aside from the IDE I last play with in the very early days of the project for me, the real interesting aspect was the C# framework it ships with that provides the bases of modeling UI elements in a similar manner to Zombie’s GEM framework. Tests can be written in C#, VB.NET or Iron Python and are written using this framework to build a test. To give you a taste of this kind of binding here is fragment from a typed class that models Windows Calc:

/// <summary>
/// The FormCalculatorAppFolder folder.
/// </summary>
public class FormCalculatorAppFolder : RepoGenBaseFolder
{

    /// <summary>
    /// Creates a new FormCalculator  folder.
    /// </summary>
    public FormCalculatorAppFolder(RepoGenBaseFolder appFolder) : 
            base("FormCalculator", "/form[@title='Calculator' or @title='Rechner']", appFolder, 3000, true)
    {
    }

    /// <summary>
    /// The ButtonAsterisk item.
    /// </summary>
    public virtual Ranorex.Button ButtonAsterisk
    {
        get 
        { 
            return CreateAdapterForPath<Ranorex.Button>("ButtonAsterisk", "button[@text='*']", 5000, null);
        }
    }
    /// <summary>
    /// The Button3 item.
    /// </summary>
    public virtual Ranorex.Button Button3
    {
        get 
        { 
            return CreateAdapterForPath<Ranorex.Button>("Button3", "button[@text='3']", 5000, null);
        }
    }
    /// <summary>
    /// The ButtonEqual item.
    /// </summary>
    public virtual Ranorex.Button ButtonEqual
    {
        get 
        { 
            return CreateAdapterForPath<Ranorex.Button>("ButtonEqual", "button[@text='=']", 5000, null);
        }
    }
    /// <summary>
    /// The OutputText item.
    /// </summary>
    public virtual Ranorex.Text OutputText
    {
        get 
        { 
            return CreateAdapterForPath<Ranorex.Text>("OutputText", "text[@controlid='403']", 5000, null);
        }
    }
    /// <summary>
    /// The ButtonClose item.
    /// </summary>
    public virtual Ranorex.Button ButtonClose
    {
        get 
        { 
            return CreateAdapterForPath<Ranorex.Button>("ButtonClose", "titlebar/button[@accessiblename='Close' or @accessiblename='Schließen']", 5000, null);
        }
    }
    /// <summary>
    /// The Button7 item.
    /// </summary>
    public virtual Ranorex.Button Button7
    {
        get 
        { 
            return CreateAdapterForPath<Ranorex.Button>("Button7", "button[@text='7']", 5000, null);
        }
    }
    /// <summary>
    /// The ButtonMinus item.
    /// </summary>
    public virtual Ranorex.Button ButtonMinus
    {
        get 
        { 
            return CreateAdapterForPath<Ranorex.Button>("ButtonMinus", "button[@text='-']", 5000, null);
        }
    }
    /// <summary>
    /// The Self item.
    /// </summary>
    public virtual Ranorex.Form Self
    {
        get 
        { 
            return CreateAdapterForPath<Ranorex.Form>("Self", "", 3000, null);
        }
    }

    
}

Notice in this example methods of class FormCalculatorAppFolder use XPATH regular expressions to “bind” their respective UI element which I think this is an interesting approach with a lot of flexibility. Also notice the idea here is similar to that of strongly typed dataset for data access. The tool provides a set of “adapters” which are used to drive various types of UI at varying degrees of accessibility (WinAPI, .NET, Web, MSAA).

I haven’t had enough time using the tool or framework to draw a conclusion as to the validity of the approach or quality of implementation but I have seen a few unexpected error dialogs which are things that will need to be fixed right away.

Are you familiar with or have you used this tool? If so, what do you think?

If not, what tools are you using for GUI automation?

Digg It!DZone It!StumbleUponTechnoratiRedditDel.icio.usNewsVineFurlBlinkList

Tags: , ,

Handling unexpected Windows in GUI test automation

January 19 2009 7:07PM

Since I conduct training for AutomatedQA’s TestComplete I like to follow the support forum to see the types of questions people are asking. Recently, Michael Groves posted the following question:

We do testing on our applications on production boxes that are setup as our production builds.  This being the case, we get applications that try to update i.e. virus, Adobe etc... and mess up test runs.  Does anyone know of a way have the test script ignore these items as they come up and continue with testing our app?

Michael’s talking about a class of unexpected windows which aren’t part of the application being tested. This includes things like Microsoft’s own Windows Update dialogs that popup, sometimes persistently, asking to reboot the machine. Unfortunately, there isn’t a simple answer to the question which is perhaps best illustrated by the lack of response he got (aside from me). Clearly, turning off such update prompts would eliminate the problem but they come from all kinds of different sources and may not be easy to disable particularly in environments where software configuration is strictly controlled.

While at Borland working on Delphi’s automation framework I solved, or perhaps a better way to put it would be mitigated, the problem at the framework level meaning test developers could write tests without ever thinking this particular issue. The solution used a secondary thread that constantly probed the active window and under certain conditions would dismiss the window allowing the test to proceed unimpeded. Additionally, the thread handled various exceptions dialogs raised by the application under test (AUT) and aborted the current test allowing subsequent tests to run. In most cases the AUT was terminated and the framework simply re-launched the AUT for the next test. The approach afforded test writer’s freedom to focus on their tests and write much less “defensive” code.

Those pesky #32770 windows!

If you’ve ever been knee deep in Windows GUI automation you’re likely to be familiar with #32770 windows. #32770 refers to a commonly used window class that you’ll find in all sorts of places throughout Windows. For example, the Windows Update dialog, mentioned above, can be particularly problematic not only because it keeps popping up but because it too is a #32770 window. The #32770 class is also used by the .NET Frameworks Assertion Failed dialog as well as other standard error dialogs not to mention Windows Task Manager and Windows common dialogs, ok have I made my point yet?

Clearly, identifying windows by window class alone is insufficient. To that end, the solution I developed cached a list of #32770 window handles that existed prior to test execution to help distinguish them from instances created by the AUT.

WinSpy UtilityAs Michael points out, there may be numerous other applications that display UI during test execution that the test may need to dismiss to prevent false failures. Years ago I wrote a tool (pictured right) to help figure out the specifics of such windows available here. It’s a simple utility application displaying a hierarchy of windows on your system listing Window Handle, {Classname}, Caption, thread ID, and dimensions. I found this tool to be very useful in developing solutions to deal with errant popup windows.

TestComplete’s Approach to Unexpected Windows

If you’re using TestComplete (TC) you can take advantage of it’s OnUnexpectedWindow event, fired when a keyboard or mouse command is unable to be directed to the intended window.

TestComplete onunexpectedwindow event

The event calls a function you write in script giving you the opportunity to manually handle the unexpected window which is passed in as a parameter. TC has a built-in mechanism to handle to try and dismiss the window but in certain cases that can lead to unintended consequences, an issue we explore in the TC training class that Falafel offers. Even if you don’t use this event TestComplete uses a fairly elaborate strategy to deal with unexpected windows and it’s important to clearly understand what it’s doing so you can understand what happens when things fail.

What’s the answer?

I don’t have the perfect answer as it’s often dependent on the environment where the tests are run but developing a flexible system to deal with unexpected windows the rule. In this particular case I suggested to Michael that it might be a good idea to write a console application that handles these unwanted dialogs and simply execute that app from TC and leave it running throughout the test. Using a separate application would help avoid TC’s unexpected window logic which can take time to trigger.

What’s you’re approach to dealing with unexpected windows?

Digg It!DZone It!StumbleUponTechnoratiRedditDel.icio.usNewsVineFurlBlinkList

Tags:

GUI models allow for compile time error detection of changes that break test automation

February 12 2008 12:54PM
When I was working at CodeGear one of the tools I wrote generated models of .DFM’s (Delphi’s form file format) for use with GUI automation. A model is a class that mirrors the control hierarchy found on a form with classes that can perform automation tasks (clicking, typing, determining location etc.) against the controls. For more information on models refer to this post.

Modeling a Form, an Example

Below is an example of a simple Delphi form and it’s corresponding model class:

Delphi Form Class Corresponding Model
type
TMyForm = class(TForm)
MessageText: TLabel;
Yes: TButton;
No: TButton;
Cancel: TButton;
...

end;
type
TMyFormModel = class(TBaseDlg)
MessageText: TLabelGem;
Yes: TButtonGem;
No: TButtonGem;
Cancel: TButtonGem;
...

end;

The model is generated off of the .DFM which looks like this:

object Form1: TForm1
  Left = 217
  Top = 88
  Width = 1082
  Height = 749
  Caption = 'Form1'
  object MessageText: TLabel
    Left = 19
    Top = 13
    Width = 32
    Height = 13
    Caption = 'Label1'
  end
  object Yes: TButton
    Left = 15
    Top = 43
    Width = 75
    Height = 25
    Caption = 'Button1'
    TabOrder = 0
  end
  object No: TButton
    Left = 101
    Top = 43
    Width = 75
    Height = 25
    Caption = 'Button2'
    TabOrder = 1
  end
end

Delphi’s VCL component library supports Visual Form Inheritance and since there isn’t enough contextual information to reconstruct the form’s class hierarchy without compiling the code the model generator is provided a file that contains these details so the generated models exactly mirrors the application’s form inheritance hierarchy not just the form itself.

So Why is a Model Generator Important?

At first, you might think sure, the model generator is going to save you lots of time coding which is entirely true. But, that’s not the only nor the largest benefit of using generated models. An additional benefit is that the models are compiled using a statically typed language providing for compile time error detection. For example let’s say a developer renames the "Yes" button to "YesBtn" or deletes the MessageText label?

If the model had been hand written or statically generated, which is the situation CodeGear was in, the error would only surface once the test suite had executed. At that point, log file analysis would have to be performed to distinguish between a real bug and a automation error, not a good situation to have QA people in day after day. With generated models these kinds of errors can be detected at compile time allowing for R&D to assess the impact of the change on QA’s automation and not vise versa. Btw, this also underscores the fact that the model generator can provide alias functionality thus preventing simple name changes from impacting test automation.

The faster developers can find out they’ve broken existing test suites the more likely the problem can be corrected even in the event that not all test suites are executed every build. Yet another benefit of continuous integration.

There are several other benefits like:

  • Give R&D the ability to determine the impact a given change will have on the existing test automation
  • Provide insight into the depth of testing through static code analysis and evaluating which models as well as which parts of models are being exercised
  • Allow R&D/QA to quickly automate new UI

Conclusion

Backing your GUI testing with compile time error checking allows you to leverage GUI test automation in several ways. In addition, a development team will have much more visibility into the impact a given change will have on the existing test automation as well as provide insight into areas that need additional testing.

It’s a shame the test framework developed at Borland starting back in 1994 has never made it into the hands of developers outside the company. Perhaps I should look to start an Open Source project for model driven testing using C# based on  IAccessible. Btw, I just Googled on IAccessible and the post I linked to is in the top ten, meaning this advice really does work.

See also: Automation

Digg It!DZone It!StumbleUponTechnoratiRedditDel.icio.usNewsVineFurlBlinkList

Tags: ,

Microsoft's ASP.NET MVC framework will be great for Automated UI testing

December 31 2007 6:12AM

I’ve written automated UI tests using Selenuim running against a large ASP.NET Web 2.0 ERP application with great success. However, IMO the single biggest productivity killer for writing such tests is ASP.NET WebForms ID name mangling where ID’s are changed from something like "lblSysMText" at design time to "ctl00_cphContents_dlMessages_ctl01_lblSysMText" at runtime. Fortunately, Selenuim supports the use of XPATH expressions so searching for the above tag can be done using the following:

//span[contains(@id, "lblSysMText")]

Name mangling is a huge problem and thwarts all sorts of automated test tools whether you’re using AutomatedQA's TestComplete or a something like Selenuim. Whenever you have a situation where a simple containership change like moving a control into a DIV can break an automated test you’ve got a problem. These sorts of changes occur all the time on WebForms which are actively being developed meaning unless your tests are written using a partial match logic, like what I’ve illustrated above, you’ll be faced with lots of bogus failures which not only kill productivity but undermine your test automation efforts as the tests will be viewed as fragile and a waste of time.

MVC to the Rescue

Scott Guthrie wrote:

This model view controller (MVC) framework for ASP.NET provides a structured model that enables a clear separation of concerns within web applications, and makes it easier to unit test your code and support a TDD workflow. It also helps provide more control over the URLs you publish in your applications, and more control over the HTML that is emitted from them.

With the MVC framework name mangling will be a thing of the past. I believe not only will the MVC framework make unit testing easier it will be a major win for automated UI testing for ASP.NET apps. You’ll no longer have to jump through hoops to figure out the ID of a given tag on the page.

Last but not least, with MVC style development you’ll finally be able to use ID’s like #lblSysMText in your CSS again.

Digg It!DZone It!StumbleUponTechnoratiRedditDel.icio.usNewsVineFurlBlinkList

Tags: , ,

Organizing your build process

November 19 2007 6:42AM

In this, my second post in a series on automated testing, I’m going to talk about a few steps you’ll need to take after your team has committed to automation. Diving into the deep end and immediately writing a bunch of tests isn’t the place to start. Organizing your project, preparing your code base and planning for automation are the first priorities. Of course, the assumption here is that you’re adding automation after the fact.

Organizing your build

If your project isn’t easy to build you’ve identified the first thing that needs fixing. Having a repeatable automated build is key to a successful test automation strategy. Essentially, getting your build organized means the following two things:

  • Version control
  • Continuous Integration

Version Control

Subversion
The first step to organizing your build and preparing it for continuous integration is to make sure it’s under version control. There are lots of ways to implement version control but it’s the first step to repeatability which is what test automation is all about. Personally, I really like SubVersion otherwise known as SVN, and would highly recommend it particularly if you’re just starting out. There’s been plenty written about the benefits of version control so I won’t go into that here just make your choice and get your code checked in.

Build Automation

The next step is automating your build process. Jeff Atwood wrote The F5 Key Is Not a Build Process discussing the benefits of moving your build process beyond your IDE of choice and I couldn’t agree more. Build automation is really going to be the key to successful test automation. When changes are committed to your repository a build gets kicked off and subsequently launches your test automation. With this setup you’re automation is guaranteed to run against every change to your repository immediately notifying you when a change has "broken" the build.

Note: Make sure your team understands that a break in test automation that’s kicked off as part your continuous integration process is as bad as checking in a syntax error. Yeah, read that again. Even if the code builds, if the smoke test fails as a result of the check-in it should be treated as though a syntax error were checked in.
cruisecontrol.net
For continuous integration I’m a fan of CruiseControl.NET but as with source control you have a lot of choices. CruiseControl.NET is open source and includes a web dashboard that’s easy to modify and supports writing plugins making it easy to extend the build system. Its rather light on documentation so if you don’t want get your hands a little dirty I’d recommend something like Automated Build Studio from AutomatedQA.

 

Putting it all Together

If you find this to be a bit daunting, have no fear I’ve put together a 10 minute video that demonstrates this entire process from beginning to end for a simple project. Of course, your project will be more complex but you’ll get a feel for how easy it is to get going. Note, I made this video in April '07 while still employed at Borland which is no longer the case nonetheless the video is still relevant.

Previous entries in this series:

Other related posts I’ve written:

Digg It!DZone It!StumbleUponTechnoratiRedditDel.icio.usNewsVineFurlBlinkList

Tags: , , , , ,