A Taxonomy of Test Oracles

1998 Quality Week

Extended Abstract:
Automation of testing is often a difficult and complex process.  The most familiar aspects of test automation are organizing and running of test cases and capturing  and verifying test results. Generating the expected results is often done using a mechanism called a test oracle. This paper describes several classes of oracles created to provide various types of verification and validation.  Several relevant characteristics of oracles are described and the advantages and disadvantages for each type of oracle are described.

In software testing, the mechanism used to generate the expected results is called an oracle. (In this paper, the first letter will be capitalized when referring to the Oracle for a specific test.) Many different approaches can be used to create, capture, and compare test results. The author, for example, has used the following methods for generating expected results:

  • Manual verification of results (human oracle)
  • Separate program implementing the same algorithm
  • Simulator of the software system to produce parallel results
  • Debugged hardware simulator to emulate hardware and software operations
  • Earlier version of the software
  • Same version of software on a different hardware platform
  • Check of specific values for known responses
  • Verification of consistency of generated values and end points
  • Sampling of values against independently generated expected results

Software tests themselves can be classified in many different ways. Automated tests that include evaluation of results need some kind of oracle regardless of the type or purpose of the tests. Yet, the mechanism for evaluation  of results ranges from none (the program or system didn’t crash) to exact (all values, displays, files,  etc., are verified). Various levels of effort and exactness are appropriate under different circumstances. The nature and complexity of an oracle is also dependent upon those circumstances.

Presented at 1998 Quality Week (QW), and October, 1998 Los Altos Workshop on Software Testing (LAWST 5)

Taxonomy Slides

Taxonomy Paper