UTPP - Another Unit Test Framework #

There are countless articles explaining the benefits of testing but still there are people who find it too boring, too time-consuming or too difficult to implement. Let me get over this subject briefly before delving into the description of this particular framework.

Writing small tests should be the bread and butter of your workday as a software developer. While you design your system top-down looking first at the big picture and decomposing it into smaller tasks, you implement your system from bottom up, making or choosing small parts that you assemble into bigger and bigger assemblies. In this phase, tests are the scaffolding that holds together your yet unfinished building. You want to have the confidence that you made something durable before moving up to the next higher level. I find myself writing tests immediately after finishing a piece of code to verify that it does what I expected it to do and check different corner cases that would be hard to verify with the complete system. I also write tests latter on in the integration phase or even after the system has been shipped in response to a bug report. In these cases, the test serves first to “illuminate” the bug and then to show that it has been solved by the code change. These tests serve as “guard rails” latter on when an upgrade fails regression testing. Tests added in this phase also serve to show how incomplete my original design was and how many requirements I forgot to take into account¹.

Please note that I’m not advocating here for Test-driven development. I find the syntagm a bit silly and sounds like a construction engineer advocating for “Scaffold-driven construction”. Scaffolds are important tools that have to be used when needed but they don’t drive a construction any more than a crane or an excavator do.

Why a New Framework? #

There are many test frameworks you can choose from but I found myself particularly attracted to UnitTest++, a framework originally created by Noel LLopis and Charles Nicholson. Before making UnitTest++ framework, in another article, Noel LLopis spelled out some of his requirements for a test framework:

• Minimal amount of work needed to add new tests.
• Easy to modify and port.
• Supports setup/tear-down steps (fixtures).
• Handles exceptions and crashes well.
• Good assert functionality.
• Supports different outputs.
• Supports suites.

UnitTest++ was based on these requirements and fulfills most of them. I found however a problem: the implementation is not very tight with WAY too many objects and unfinished methods for my taste. Instead of choosing another framework I decided to re-implement UnitTest++ and that’s how UTPP (Unit Test Plus Plus) came into existence. It borrows the API from UnitTest++ but the implementation is all new.

Using the framework #

Here is a short example of how to use the test framework:

#include <utpp/utpp.h>

bool earth_is_round ();
double earth_radius_km ();

TEST (EarthShape)
{
  CHECK (earth_is_round ());
}

TEST (HowBigIsEarth)
{
  CHECK_CLOSE (6371., earth_radius_km(), 1.);
}

int main (int argc, char** argv)
{
  return UnitTest::RunAllTests ();
}

The program contains two tests: one that checks if the earth_is_round function returns true and another one that checks if the earth_radius_km function is close enough to the expected value. The main program runs all the tests and, if all goes well, returns 0. Tests are introduced by the TEST macro followed by a block of code. Throughout the test you can check different conditions using one of the CHECK_... macros. The example above showed two of these macros: CHECK verifies that a condition is true, while CHECK_CLOSE verifies that two values are closer than a specified limit.

There are many macros to verify different conditions during a test. You can see the Reference section for a complete list.

Each macro has a ..._EX variant that takes additional parameters that are used to generate the failure message. The first additional parameter is interpreted as a printf format string and the rest are passed to a printf-like function. For instance the following code:

int actual = 2, expected = 3;
CHECK_EQUAL_EX (expected, actual, "Values %d and %d are different", actual, expected);

Will produce the message:

... Values 2 and 3 are different

There is nothing special to be done when adding new tests: you just write them and they will get executed. Tests can be in the same source file or in different ones. They will still be picked up automatically and executed. However, there is no guarantee about the execution order.

Here is another example using CHECK_EQUAL macro:

const char *planet_name () {
  return "Earth";
}

TEST (PlanetName)
{
  CHECK_EQUAL ("Earth", planet_name ());
}  

This macro can compare numbers, strings or in general any any values for which an equality operator is defined.

You can also test if an exception is thrown using CHECK_THROW macro:

class flat_earth_exception : public std::exception {
public:
  const char *what () { return "Earth is not flat!"; }
};

void go_to_end_of_earth ()
{
  throw flat_earth_exception();
}
TEST (EndOfTheEarth)
{
  CHECK_THROW (flat_earth_exception, go_to_end_of_earth ());
}

Exceptions thrown outside of a CHECK_THROW macro are considered failures and are caught by try... catch blocks that wrap each test.

Fixtures #

When performing a test you need certain objects and values to be in a known state before the beginning of the test. This is called a fixture. In UTPP any object with a default constructor can be used as a fixture. Your tests will be derived from that object and the state of the object is defined by the fixture constructor.

Example:

void exchange_to_eur (double& usd, double& eur);

struct Account_fixture {
  Account_fixture () : amount_usd (100), amount_eur (0), amount_chf (0) {}
  ~Account_fixture () {}
  double amount_usd;
  double amount_eur;
  double amount_chf;
};
TEST_FIXTURE (Account_fixture, TestExchangeEur)
{
  exchange_to_eur (amount_usd, amount_eur);
  CHECK_EQUAL (0, amount_usd);
  CHECK (amount_eur > 0);
}

A test that uses a fixture is defined using a TEST_FIXTURE macro that takes as arguments the name of the fixture and the name of the test. The fixture constructor is invoked right before the beginning of the test and it insures that amount_usd is set to 100. Because the test object is derived from the fixture object, any public or protected members of the fixture are directly available in the test body. When the test finishes, the fixture destructor gets called to release any resources allocated by the constructor.

More than one test can use the same fixture and it will be setup the same way at the beginning of each test:

void exchange_to_chf (double& usd, double& chf);
  
TEST_FIXTURE (Account_fixture, TestExchangeChf)
{
  exchange_to_chf (amount_usd, amount_chf);
  CHECK_EQUAL (0, amount_usd);
  CHECK (amount_chf > 0);
}

In this case both tests, TestExchangeEur and TestExchangeChf start with the same configuration.

Floating point results can be compared using CHECK_CLOSE_... macros with an optional parameter that is the acceptable error limit. If the tolerance is not specified, the default value UnitTest::default_tolerance is used. Note that, in this case, the default tolerance must be set to a non-zero value².

Results Handling - Reporters #

All output from the different CHECK_... macros together with other general messages are sent an object called a reporter. This object is responsible for generating the actual output. The default reporter sends all results to stdout. There is another reporter for generating an XML file in a format similar with NUnit. Yet another reporter can send all output using the OutputDebugString function.

To change the reporter used you create a reporter object and pass it to RunAllTests function:

  std::ofstream os ("test.xml");
  UnitTest::ReporterXml xml (os);
  UnitTest::RunAllTests (xml);

Test Grouping #

Tests can be grouped in suites:

SUITE (many_tests)
{
  TEST (test1) { }
  TEST (test2) { }
}

All tests from one suite are going to be executed before the next suite begins. If the main program invokes the tests by calling UnitTest::RunAllTests() function, there are no guarantees as to the order of execution of each suite or for the order of tests within the suite. There is however a function:

int UnitTest:RunSuite (const std::string& suite_name);

that runs only one suite.There is also a function that prevents a suite from running:

void UnitTest::DisableSuite (const std::string& suite_name);

Internally, a suite is implemented as a namespace and that helps preventing clashes between test names: you have to keep test names unique only within a suite.

Timing #

Each test can have a limit set for its running time. You define these local time limits using the UNITTEST_TIME_CONSTRAINT(ms). This macro creates an object of type UnitTest::TimeConstraint in the scope where it was invoked. When this object gets out of scope, if the preset time limit has been exceeded, it generates a message that is logged by the reporter.

In addition to these local time limits, the UnitTest::RunAllTests() function takes an additional parameter that is the default time limit for every test. If a test fails this global time limit, the reporter generates a message a failure message. If using the global time limit, a test can be exempted from this check by invoking the UNITTEST_TIME_CONSTRAINT_EXEMPT macro.

Reference #

UTPP macro definitions can be divided in the following groups:

• test lifetime control macros
• assertion verification macros
• exception verification macros
• Google Test compatibility macros

In addition, there are a few functions for execution control.

Test lifetime control macros #

• SUITE (name) groups multiple tests that should be run together. Tests that form a suite are enclosed in braces. Suites don’t need to be defined in one source file.

• TEST (test_name) - defines the beginning of a test. Inside a suite, test names must be unique. This macro must be followed by a code block that becomes the body of the test function.

• TEST_FIXTURE (fixture_name, test_name) - defines a test that requires a fixture. The fixture can be any structure or class that has a default constructor. Fixture constructor is executed prior to test execution.

• ABORT (expr) - if expr is false (0) the current test is terminated. If a fixture was setup, it is destructed now.

• ABORT_EX (expr, ...) - same as ABORT macro, terminates current test if expr is false. First additional argument is interpreted as a printf format string and additional arguments are passed to a printf-like function.

Assertion verification macros #

As explained before, each macro in this group has a ..._EX variant that takes additional arguments to create the failure message. Those aren’t going to be repeated here.

• CHECK(expr) verifies that expr is true.

• CHECK_EQUAL (expected, actual) - verifies that the value of actual expression is equal to the value of expected expression. The two values can be of any type that has a equality operator.

• CHECK_CLOSE (expected, actual [,tolerance]) - verifies that actual value is within tolerance from expected value. If tolerance argument is missing, the default value is UnitTest::default_tolerance. Specially useful for floating point values.

• CHECK_ARRAY_EQUAL (expected, actual, count) - verifies that each element of actual array is equal to corresponding element of expected array. Arrays have count elements. This macro is for C-style array; C++ containers that know their size can use CHECK_EQUAL macro.

• CHECK_ARRAY_CLOSE (expected, actual, count [,tolerance]) - verifies that each element of actual array is within tolerance from corresponding element in expected array. Arrays have count elements. As in the case of the scalar version CHECK_CLOSE, if tolerance expression is missing, the UnitTest::default_tolerence is used.

• CHECK_ARRAY2D_EQUAL (expected, actual, rows, columns) - verifies that each element of actual two-dimensional array is equal to expected array.

• CHECK_ARRAY2D_CLOSE (expected, actual, rows, columns [,tolerance]) - verifies that each element of actual two-dimensional array is within tolerance from expected array.

• CHECK_FILE_EQUAL (expected_fname, actual_fname) - verifies that content of actual_fname file is the same as that of expected_fname.

Exception verification macros³ #

Macros in this group, also have …_EX counterparts that take additional parameters to create the failure message.

• CHEK_THROW(expr, except) - verifies that evaluation of expr throws an exception of type except.

• CHECK_THROW_EQUAL (expr, value, except) - verifies that evaluation of expr throws an exception of type except with given value. except type must have an equality operator.

Google Test compatibility macros #

The following table shows Google Test macros that can be used with UTPP and their equivalent UTPP definition

Keep in mind that, as opposed to Google Test, these macros do not return a stream so they cannot be used with the insertion operator <<. However, UTPP macros, in their ..._EX form accept printf-like arguments that are appended to the failure message.

Timing macros #

• UNITTEST_TIME_CONSTRAINT(ms) - initializes a time constraint for a block of code. If block execution takes more than the specified number of milliseconds, a failure occurs.
• UTPP_TIME_CONSTRAINT_EXEMPT - flags a test as exempt from global time constraint.

Functions #

• DisableSuite (const std::string& suite_name) - disables a particular suite.
• EnableSuite (const std::string& suite_name) - re-enables a previously disabled suite.
• RunAllTests (Reporter& rpt, int max_time_ms) - run all tests from all enabled suites. Returns number of failed tests.

Architecture #

In its simplest form, a test is defined using the TEST macro using the following syntax:

TEST (MyFirstTest)
{
  // test code goes here
}

A number of things happen behind the scenes when TEST macro is invoked:

1. It defines a class called TestMyFirstTest derived from Test class. The new class has a method called RunImpl and the block of code following the TEST macro becomes the body of the RunImpl method.

2. It creates a small factory function (called MyFirstTest_maker) with the following body:

   Test* MyFirstTest_maker ()
   {
    return new MyFirstTest;
   }
   

   We are going to call this function the maker function.

3. A pointer to the maker together with the name of the current test suite and some additional information is used to create a TestSuite::Inserter object (with the name MyFirstTest_inserter). The current test suite has to be established using a macro like in the following example:

   SUITE (LotsOfTests)
   {
     // tests definitions go here
   }
   

   If no suite has been declared, tests are by default appended to the default suite.

4. The TestSuite::Inserter constructor appends the newly created object to current test suite.

5. There is a global SuitesList object that is returned by GetSuitesList() function. This object maintains a container with all currently defined suites.

The main program contains a call to RunAllTests() that triggers the following sequence of events:

1. One of the parameters to the RunAllTests() function is a TestReporter object either one explicitly created or the default reporter that sends all results to stdout.

2. The RunAllTests() function calls SuitesList::RunAll() function.

3. SuitesList::RunAll() iterates through the list test suites mentioned before and, for each suite calls the TestSuite::RunTests() function.

4. TestSuite::RunTests() iterates through the list of tests and for each test does the following:

   • Calls maker function to instantiate a new Test-derived object (like TestMyFirstTest).
   • Calls the Test::Run method which in turn calls the TestMyFirstTest::RunImpl. This is actually the test code that was placed after the TEST macro.
   • When the test has finished, the Test-derived object is deleted.

Throughout this process, different methods of the reporter are called at appropriate moments (beginning of test suite, beginning of test, end of test, end of suite, end of run).

CHECK... macros evaluate the condition and, if false, call the ReportFailure function, which in turn calls Reporter::ReportFailure function to record all failure information (file name, line number, message, etc.). To determine if a condition is true, the CHECK_EQUAL and CHECK_THROW_EQUAL macros invoke a template function:

template <typename Expected, typename Actual>
bool CheckEqual (const Expected& expected, const Actual& actual, std::string& msg)
{
  if (!(expected == actual))
  {
    std::stringstream stream;
    stream << "Expected " << expected << " but was " << actual;
    msg = stream.str ();
    return false;
  }
  return true;
}

The template function can be instantiated for any objects that support the equality operator. There are however explicit functions for comparison of character strings.

Conclusion #

To finalize, let’s review the requirements listed at the beginning and see how UTPP fares against them:

1. Minimal amount of work needed to add tests. You just write the test using TEST or TEST_FIXTURE macros. Test registration is automatic and tests can be in different source files.

2. Easy to modify and port. The code is very clean and well documented.

3. Supports setup/tear-down steps (fixtures). Any object with a default constructor can become a fixture. Fixtures are integrated into a test using TEST_FIXTURE macro. Object destructor takes care of tear-down.

4. Handles exceptions and crashes well. Tests can check for exceptions using CHECK_THROW and CHECK_THROW_EX macros. All other exceptions are caught and logged.

5. Good assert functionality. There are a variety of CHECK_... macros. As they translate internally into function templates, they can take arbitrary parameters.

6. Supports different outputs. The use of reporter objects makes it easy to redirect output to different venues. As is the library can direct output to stdout, debug output or an XML file but users can create their own reporters.

7. Supports suites. Yes, it does.

Although there is no shortage of unit test frameworks, if you spend a bit of time with UTPP, you might begin to like it.

History #

• 21-Apr-2020 Initial version.
• 01-Feb-2022 Header-only library
• 12-Feb-2024 Documented Google Test macros; default tolerance