Automated Results Comparison Isn't Always Easy

Recife Summer School (Recife, Brazil)

2009 Recife Summer School (Brazil)  (1/2 Day Class)

Automating tests means both exercising the software and determining whether the resultant  behavior is expected. Capturing and comparing results can be tricky and difficult.  The talk presents some of the issues and ways to deal with them.

Five factors about automating results comparison need to be factored into any test automation effort.  The talk describes the five factors (listed below), explains why each is an issue, and goes into some of  the implications and possible actions to deal with them.

The five factors are:

  • Which [potential] results to compare
  • What results to expect
  • Not all expected differences indicate errors
  • "Fuzzy" comparisons
  • When to generate and compare results

Presented at 2000 Recife Summer School 2009 (Brazil)

Class Slides