Qt Unit Testing with Catch and Trompeloeil

Qt and catch logo 8 minutes read

In this blog post, you will learn how to do Qt unit testing with Catch and Trompeloeil.

I'll explain how to this with the qmake build system if you prefer CMake you can follow this guide. However, some of the details I describe here such as how to use Catch for testing event-driven applications do apply for CMake as well.

Hint: If you are more into diving straight into the code you can skip to the seed project on GitHub

Why bother to write Unit Tests?

Especially if you are relatively new to software engineering or you program software as a hobby you might wonder why you should write unit tests anyway.

I myself have started to do proper unit testing only very recently, mostly because unit testing always seemed to be unnecessarily hard to do with not a lot of benefits.

However, I have recently read "The Clean Coder" by Uncle Bob and "Developer Testing" which made change my opinion.

In fact, I discovered that unit testing is to business logic implementation, what live coding is to UI development. In my opinion, the shorter feedback loops drastically improve productivity while coding and also the maintainability of software in the long term.

If you have not tried Test Driven Development (TDD) I encourage you to do so. It looks like an unnecessary overhead in the beginning, but I promise you it pays off very quickly.

Why should I use Catch when there is Qt Test?

There are many reasons why I prefer using Catch over Qt Test.

Catch is a modern unit testing framework based on C++11 and up. Many of its features outpace Qt Test by far. Most important for me is that Catch tests are structured in a way that reduces redundancy between test cases.

Moreover, one of the key features of Catch is Behavior-driven development (BDD) testing meaning tests are structured in a way you would naturally describe a product feature.

Unfortunately, Catch does not support the Gherkin language yet such the Python BDD testing framework behave. Nevertheless, the BDD testing feature is great and by far the best implementation I have seen so for C++.

In fact, Catch is so easy to use that you will start to write tests more frequently.

What about Trompeloeil?

Once we start unit testing our application we soon require test doubles such as stubs, fakes, and mocks.

Since writing test doubles by hand is a cumbersome task, I recommend using a mock framework.

Trompeloeil stands out of the crowd because it uses modern C++14 features and it integrates nicely with Catch. Moreover, it also a header only library such as Catch.

Again, the idea is to make unit testing as easy possible so people will not shy back from doing it.

Setting up Qt for Unit Testing

First, let's start by creating a "Qt Subdir Project".

Select Subdirs Project
Select Subdirs Project

Qt Creator will ask right away for creating a new subproject. We choose a "Qt Console Application" here. I named it app.

Select Qt Console Application
Select Qt Console Application

Next, we create another subproject for our tests called tests of type "Auto Test Project".

Select Auto Test Project
Select Auto Test Project

In the same wizard, Qt Creator asks for creating the first test. In this window, we select "Qt Test" and "Requires QApplication".

Project and Test Information Wizard Settings
Project and Test Information Wizard Settings

Now we are ready for unit testing out application Qt Test.

Converting the Qt Test into a Catch Test

Now it's time to download the Catch and Trompeloeil header only libraries.

We will add both libraries to the 3rdparty directory of our project. This way, any developer can check out the project with a single command. If you are uneasy with the idea of adding 3rdparty libraries to your source tree I recommend you to take a look at the C++ package manager Conan.

Download the Header Files

First, create a new folder in your project root called 3rdparty/catch.

Then download the single header library from GitHub. Make sure to select a tag before heading to the library download, else you might end up with a development version of Catch.

A sane download link looks as follows: https://raw.githubusercontent.com/philsquared/Catch/v1.10.0/single_include/catch.hpp

Repeat this step for Trompeloil:

Download the single header library to 3rdparty/trompeloeil.

Download link: https://raw.githubusercontent.com/rollbear/trompeloeil/v28/include/trompeloeil.hpp

Add .pri Files

Okay, almost there. Now we need to add a .pri file for both, Catch and Trompeloeil.

Create a file called catch.pri in the 3rdparty/catch subdirectory and a file called trompeloeil.pri in the 3rdparty/trompeloeil directory with the following content.

INCLUDEPATH += $$PWD

Integrate Trompeloil with Catch

Trompeloeil supports custom error reporters which can be used to integrate it into different unit testing frameworks.

We use the example error reporter provided on GitHub and save in our 3rdparty/trompeloeil directory as trompeloeil_catch.hpp.

#ifndef TROMPELOEIL_CATCH_HPP
#define TROMPELOEIL_CATCH_HPP
#include "trompeloeil.hpp"
#include "catch.hpp"

namespace trompeloeil
{
  template <>
  void reporter<specialized>::send(
    severity s,
    const char* file,
    unsigned long line,
    const char* msg)
  {
    std::ostringstream os;
    if (line) os << file << ':' << line << '\n';
    os << msg;
    auto failure = os.str();
    if (s == severity::fatal)
    {
      FAIL(failure);
    }
    else
    {
      CAPTURE(failure);
      CHECK(failure.empty());
    }
  }
}

#endif // TROMPELOEIL_CATCH_HPP

Integrate Catch into the Test Project

Next, we need to modify our tests.pro file of the tests subproject.

  • Include our .pri files.
  • Add C++14 support for trompeloeil.
  • Include our app directory.
  • Add a main.cpp for Catch.

The file looks as follows after the modifications:

QT += testlib
QT -= gui

CONFIG += qt console warn_on depend_includepath testcase
CONFIG -= app_bundle
CONFIG += c++14

TEMPLATE = app

include(../3rdparty/catch/catch.pri)
include(../3rdparty/trompeloeil/trompeloeil.pri)

PROJECT_DIR = $$PWD/../app

INCLUDEPATH += $$PROJECT_DIR

SOURCES +=  \
    main.cpp \
    tst_apptests.cpp

Add a main.cpp

Catch can be configured to use a test runner. This type of setup is necessary to integrate Catch with the Qt event loop. It also allows us to utilize features of Qt Test within our Catch unit tests.

main.cpp

#define CATCH_CONFIG_RUNNER
#include <catch.hpp>
#include <trompeloeil_catch.hpp>

#include <QCoreApplication>
#include <QtTest>

int main( int argc, char* argv[] )
{
    QCoreApplication a(argc, argv);

    QTEST_SET_MAIN_SOURCE_PATH
    int result = Catch::Session().run( argc, argv );

    return ( result < 0xff ? result : 0xff );
}

Add a Hello World Test Case

Last but not least we need to add our first "Hello World" Catch Test Case.

For this purpose, we rewrite the tst_apptest.cpp file created by Qt Creator. Additional test cases go into additional source files.

tst_apptest.cpp

#include <catch.hpp>
#include <trompeloeil.hpp>

#include <iostream>

#include <QSignalSpy>

extern template struct trompeloeil::reporter<trompeloeil::specialized>;


TEST_CASE("Application Tests", "[app]")
{

    SECTION("Hello World") {
        REQUIRE(true);
    }
}

That's it. Now when we hit run in Qt Creator we should get following output by Catch:

Starting /home/alexander/projects/build-qt-qmake-catch-and-trompeloeil-seed-Desktop_Qt_5_10_0_GCC_64bit2-Debug/tests/tests...
===============================================================================
All tests passed (1 assertion in 1 test case)

/home/alexander/projects/build-qt-qmake-catch-and-trompeloeil-seed-Desktop_Qt_5_10_0_GCC_64bit2-Debug/tests/tests exited with code 0

Writing Qt Unit Tests with Catch

Now that we have everything ready and set up it is time to write our first unit test with Catch that involves classes from our application.

I will not copy the code of the classes under test over here. If you want to study the code please check out the GitHub repository.

Adding the Application Sources to the Test project

Let's start by including the application source code into our test. The optimum for doing this would be to create a static library containing the application code. That's extremely straightforward with CMake, but not so much with qmake and, therefore, I will take the alternative approach in this example.

For this purpose, we add a few additional lines to the tests.pro file.

PROJECT_SOURCES = \
    $$PROJECT_DIR/order.cpp \
    $$PROJECT_DIR/warehouse.cpp \
    $$PROJECT_DIR/viennawarehouse.cpp \
    $$PROJECT_DIR/autoorder.cpp

PROJECT_HEADERS = \
    $$PROJECT_DIR/order.h \
    $$PROJECT_DIR/warehouse.h \
    $$PROJECT_DIR/viennawarehouse.h \
    $$PROJECT_DIR/autoorder.h

and

SOURCES +=  \

    main.cpp \

    tst_apptests.cpp \

    $$PROJECT_SOURCES

HEADERS += \
    $$PROJECT_HEADERS

Testing Non-Event-Driven Parts of the Code

First of all, I will start with tests that do not require Qt's event loop to be running. This should be straightforward for you if you are already familiar with unit testing C++ code.

Okay, so lets by testing if our ViennaWarehouse has enough Tafelspitz and Schnitzel. I will use the classical section based approach for this purpose.

First of all, we want to make sure our ViennaWarehouse can provide less than 15, exactly 15 and not more 15 Schnitzels after initialization. (Let's assume our requirements state that the warehouse always starts with 15 Schnitzel from yesterday).

TEST_CASE("ViennaWarehouse starts with exactly 15 Schnitzel", "[app]")
{
    ViennaWarehouse warehouse;

    SECTION("Checking for less than 15 Schnitzel works") {
        REQUIRE(warehouse.hasInventory("Schnitzel", 10));
    }

    SECTION("Checking for exactly 15 Schnitzel works") {
        REQUIRE(warehouse.hasInventory("Schnitzel", 15));
    }

    SECTION("Checking for more than 15 Schnitzel fails") {
        REQUIRE(warehouse.hasInventory("Schnitzel", 200) == false);
    }
}

As you can see, we already save a lot of time, since we need to instantiate the ViennaWarehouse only once for all our test sections. However, the instantiation of ViennaWarehouse is run once for each section thanks to the macro and C++11 magic in Catch.

Next, let's use the mocking power of trompeloeil to the test the Order class.

First, we create a new mock class that inherits from our abstract Warehouse class. The macros and C++14 magic behind trompeloeil make this very convient and readable.

class WarehouseMock: public Warehouse
{
public:
    MAKE_CONST_MOCK2(hasInventory, bool(const QString&, int));
    MAKE_MOCK2(remove, bool(const QString&, int));
};

Using this newly created mock class we can now easily test our Order class.

TEST_CASE("Filling Order from Warehouse works", "[app]")
{
    Order order("Schnitzel", 50u);
    WarehouseMock warehouse;

    SECTION("filling removes inventory if in stock") {
        REQUIRE_CALL(warehouse, hasInventory("Schnitzel", 50))
                .RETURN(true);
        REQUIRE_CALL(warehouse, remove("Schnitzel", 50))
                .RETURN(true);

        order.fill(warehouse);

        REQUIRE(order.isFilled());
    }

    SECTION("filling does not remove inventory if not in stock") {
        REQUIRE_CALL(warehouse, hasInventory("Schnitzel", 50))
                .RETURN(false);

        order.fill(warehouse);

        REQUIRE_FALSE(order.isFilled());
    }
}

We see that writing the tests is now down to a few lines of test code. The big bonus here is that the tests are very readable.

Also, when we hit an error we get very clear and concise error messages.

  failure := "../../qt-qmake-catch-and-trompeloeil-seed/tests/tst_apptests.
  cpp:44
  Unfulfilled expectation:
  Expected warehouse.remove("Schnitzel", 50) to be called once, actually never
  called
    param  _1 == Schnitzel
    param  _2 == 50
  "

Testing Event-Driven Parts of the Code

When writing Qt code you very likely make use of Qt's fantastic event system. Event-driven programming is my opinion a very good strategy to keep the complexity of a system low.

However, event-driven systems may not be straightforward to test without Qt's test system. The good news is we can combine Qt Test features with Catch unit tests.

It is important to understand that Qt's signals and slots require the Qt event loop to be running. In fact, in Qt every thread has it's own event loop. We can start our own event loop by using QEventLoop.

However, it's more convenient to use the Qt event loop created by our QCoreApplication in the main.cpp of our test application. Now we can use the Qt event loop as we would for writing our Qt tests.

Let's take a look at how we could test the AutoOrder class:

TEST_CASE("Automatically ordering wares with AutoOrder works", "[app")
{
    WarehouseMock warehouse;

    GIVEN("We have an auto order of 1 Tafelspitz") {
        Order order("Tafelspitz", 1);
        AutoOrder autoOrder(&warehouse, &order);
        QSignalSpy waresOrderedSpy(&autoOrder, &AutoOrder::waresOrdered);
        QSignalSpy outOfWaresSpy(&autoOrder, &AutoOrder::outOfWares);

        WHEN("We start the auto order") {
            autoOrder.startOrdering(1);

        AND_WHEN("we have enough Tafelspitz in stock") {
            REQUIRE_CALL(warehouse, hasInventory("Tafelspitz", 1))
                    .RETURN(true);
            REQUIRE_CALL(warehouse, remove("Tafelspitz", 1))
                    .RETURN(true);

            THEN("we are notified about the order") {
                waresOrderedSpy.wait(10);

                REQUIRE(waresOrderedSpy.count() == 1);
            }
        }

        AND_WHEN("we don't have enough Tafelspitz in stock") {
            REQUIRE_CALL(warehouse, hasInventory("Tafelspitz", 1))
                    .RETURN(false);

            THEN("we are notified that we have just run out of stock") {
                outOfWaresSpy.wait(10);

                REQUIRE(outOfWaresSpy.count() == 1);
            }
        }
        }
    }
}

As you can see, the test code is now structured with Catchs BDD testing style syntax.

In my opinion, BDD style tests are easier to understand and read than classical unit tests.

Now when we take a look at a possible error message produced by Catch:

-------------------------------------------------------------------------------
Automatically ordering wares with AutoOrder works
     Given: We have an auto order of 1 Tafelspitz
      When: We start the auto order
  And when: we don't have enough Tafelspitz in stock
      Then: we are notified that we have just run out of stock
-------------------------------------------------------------------------------
../../qt-qmake-catch-and-trompeloeil-seed/tests/tst_apptests.cpp:92
...............................................................................

../../qt-qmake-catch-and-trompeloeil-seed/tests/tst_apptests.cpp:95: FAILED:
  REQUIRE( outOfWaresSpy.count() == 0 )
with expansion:
  1 == 0

We see that the BDD testing not only shows us where the error occurred but also what lead to the error in a natural language.

Conclusion

In this article, you have learned how to set up Qt with qmake for unit testing with Catch and Trompeloeil.

Moreover, you have learned how to write tests for event-driven as well non-event-driven parts of the code.

For reference, you can download the complete code created to write this tutorial from GitHub.

I hope you have enjoyed this tutorial and I thankful for any feedback, suggestions or your opinions on BDD-style unit testing.

Your
Machine Koder

Spread the love

5 thoughts on “Qt Unit Testing with Catch and Trompeloeil”

  1. Thanks for this. After reading I struggled to understand what is the need for Trompeloeil, how does this play into the unit testing? Thanks!

    1. Trompeloeil is a mocking framework. It helps you creating test doubles of your classes or classes from external libraries. Using this test doubles you can for example check if a function was called with specific arguments or set the return value for a function from an external library.

  2. Wel... consider yourself added to my blogroll. I have like six other blogs I read on a weekly basis, guess that number just increased to seven! Keep writing!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.