Category Archives: Testing

Publishing Python unit test results in Jenkins

When I switched to developing on an OS stack one of the first things I look for was a $g(Continuous Integration server) and settled on Hudson which, after some tumult with surrounding Oracle’s acquisition of Sun, was forked into Jenkins. Getting jenkins setup couldn’t be easier and the web UI is comprehensive and full of options.

My day-to-day development is in Python and I’ve written a bunch of tests based on the core unittest module though it doesn’t natively produce results that can be consumed by Jenkins. To that end, I searched around and found the necessary pieces which I wanted to capture.

First, you need to install the unittest-xml-reporting package which is described as:

PyUnit-based test runner with JUnit like XML reporting.

sudo easy_install unittest-xml-reporting

Once installed you need to add the following to your unittests so they will produce the necessary XML result output:

import xmlrunner
if __name__ == '__main__':

Next, in Jenkins click the configure link for your project and check the Publish JUnit test result report and set the path to the output location for the unit tests. In my case the full path to the XML output is /.hudson/jobs/publishing/workspace/trunk/test/test-reports. In Jenkins the path to use for publishing is **/trunk/test/test-reports/*.xml

Jenkins JUnit publisher settings

This will also add a chart to the project page in Jenkins:


TestComplete Keyword Testing Book

TestComplete Made EasierOver the past 6 months or so I’ve been working on writing a courseware book for TestComplete focused on Keyword Testing, a new feature as of v7. The book provides a detailed introduction to the TestComplete IDE with step-by-step examples of using the Keyword Testing scenarios covering a wide range of topics including Data Driven testing, Web GUI testing, Stores & Checkpoints and many others.

Selenium Training

Update: Oct 2010 FYI, I’ve since changed jobs and I’m no longer conducting training. 

Selenium Logo

If you are or would be interested in training on the Open Source web testing tool called Selenium please leave a comment or contact me and let me know some specifics about what type of training you’re looking for. I’m focused on .NET development and would be looking at covering Selenium from a .NET perspective.

What should UI developers know about TestComplete

TestComplete Online TrainingThis question came up in the AutomatedQA newsgroup and this was my response:

A short list off the top of my head…

For Windows Apps 

  • Avoid use of painted controls or use controls that support UI Automation or MSAA particularly in the event they are painted
  • Avoid non-standard custom UI skinning while it can look good it often will seriously thwart automation.
  • Provide UI cues when the state of the application is changing or when there is a long running operation (title/status bar messages)
  • Use keyboard shortcuts wherever possible "&" in menus, labels etc.
  • Be sure to name controls dynamically created at run time

For Web Apps

  • Avoid using frames
  • Try to adhere to the Accessibility Standards (Section 508 here in the US)

I’d also mention that if the code they (the developer’s) are writing is hard to read and they dread debugging it, automation it is likely to be a nightmare. No matter how many hints about automation one can give there really is no substitute for good design.

Any other suggestions you would add?

      Distributed Testing Using TestComplete

      imageI’ve recorded a video for Falafel TV illustrating how to use TestComplete’s distributed testing features. In the video I demonstrate recording a Keyword test and adding the necessary support to allow it to be executed in a distributed fashion. If you’re just getting started this is a good introduction and should get you headed in the right direction. I’ve got some notes for an advanced video on the subject but if you have suggestions let me know.

      HTTP Load Testing Using TestComplete

      image I just posted a new video to Falafel TV  illustrating how to create an HTTP Load Test using TestComplete v7.1. The video covers the basics of getting an HTTP Load Test recorded and running against a website including scaling the test out to four virtual users running on two different machines.

      In the interest of keeping the video to a reasonable time length other features related to HTTP Load Testing will be covered in upcoming videos.

      HTTP Load Testing with AutomatedQA's TestComplete 6 and Remote Agent

      Recently, I was teaching a class where the client was interested in HTTP load testing for which Test Complete has great support. Test Complete ships with a tool called Remote Agent that can be installed separately and allows for HTTP load testing using virtual users on multiple machines all controlled by a single instance of TestComplete. In fact, you can load test with up to 250 virtual users allowing a single tester to easily leverage an entire network of computers.

      The following outlines the steps I took to demonstrate how easy it is to setup and execute HTTP load tests.

      Setting up an HTTP Load Test ProjectTestComplete Project Manager

      The first step is to create a new HTTP Load Test project.

      1. Select File | New Project
      2. Click the HTTP Load Testing Template
      3. Click Ok
      4. The Project Wizard appears and you can simply click Finish which will give you a project that looks like the image to the right.

      As you can see the project has several nodes by default.

      • Stations: represents the machines that will be conducting the tests. Master is my current machine, the one I’m running TestComplete on.
      • Tasks: is the HTTP traffic you’re interested in testing.
      • Tests: allows you to assign tasks to various stations for test execution.
      • Scripts: is for manually writing test automation.

      Setting up Remote Agent

      For this example, I’m using a virtual machine running Windows XP. I’ve installed TestComplete 6.0 and I’m running Remote Agent which runs in a console window on the desktop. If you’re going to try and duplicate my setup be sure that your VPC network adapter is not set for Shared networking (NAT).

      image The next step is to setup the machines you’ll use for load testing under the Stations node of your project.

      1. Right click the Stations node and select Add New… (you should see the dialog to the right)
      2. Enter the IP address of your Remote Agent machine, in my case that’s
      3. Click Ok

      This will give you a new node under Stations, “VPC” which is your VM, or the machine running Remote Agent.

      Record an HTTP Task

      The next step is to create the HTTP traffic you want to load test your server and the easiest way is using TestComplete’s recording facilities. For HTTP load testing this requires setting TestComplete to be a proxy for your web browser.

      1. Configure Internet explorer to use TestComplete as a proxy allowing it to monitor and record HTTP traffic. For IE you can do that from the Connections tab of the Internet Options dialog setting the proxy to localhost on port 9999. For Firefox open the Tools|Options dialog and look on the Advanced page, under the Network tab.
      2. Click the record button on the TestComplete toolbar and once recording begins click the “Record and HTTP Task” button (circled in red below):
      3. This will bring up the Select Load Testing dialog where you can decide to record a new HTTP traffic or append it to an existing task.
      4. Once you click OK you can begin recording HTTP traffic by simply using your browser. TestComplete will record the HTTP traffic and add it to the task you specified.

      Once recorded you can modify the web requests to fit your needs. Here is an example of what the recorded HTTP traffic looks like:


      Running HTTP Load Testing

      The next step is to configure individual Stations to perform the Tasks you’ve created. To execute your tests, TestComplete sends the recorded HTTP traffic to the web server bypassing the browser which means clients hosting Remote Agent will have no UI appear on screen other than the agent itself.



      As you can see I’ve configured one virtual user to sent HTTP traffic starting half a second after the Master with a User-Agent of IE6.

      Running HTTP Load Testing

      The final step is to run your HTTP load tests and examine the output. To do that you can simply right click your test under the Tests node in the Project Explorer. The results at first will appear overwhelming but undoubtedly provide you with a detailed report of how you server performed.


      Wrap up

      As you can see TestComplete makes it quick and easy to create and run HTTP Load Tests so if this is something you’re looking for you might want to give it a try.

      [UPDATED: Jan, 17 2008] Fix a few typos, and clarify adding a new Stations node.