Testing ROS powered Robots with pytest

In this article, you will learn how to use Pytest to successfully test ROS powered robots. In particular I explain how to employ pytest for testing ROS nodes.

Introduction

ROS, the Robot Operating System, which is actually no operating system but a middleware plus a huge number of tools to build a distributed system, comes with excellent Python bindings.

ROS uses its own package format and build tool called catkin which streamlines processes such as builds, documentation, and testing.

However, one thing that really dissatisfied me was that it only supports Python unittest and nosetests per default.

In my opinion, pytest is the far better unit testing framework for Python, as it drastically simplifies the process of writing tests which in the end leads to more tested code and better software.

Luckily, rostest does not really care which tool is used to run the unit tests, as long as it outputs JUnit XML compatible reports.

Since documentation on testing ROS nodes is sparse in general, I decided to write this short little blog post in case me or someone else needs a starting point in the future.

rostest and Python

As already mentioned, I found the official documentation on how to successfully use rostest with Python very sparse.

Here is what I found:
* Official rostest documentation
* Test examples
* How to run tests with catkin

Based on the documentation, I managed to build an example project with working unit and integration tests.

Pytest integration

ROS calls its unit tests with gtest-style parameters, meaning that it passed the output file with the --gtest_output parameter to the unit test.

pytest has a slightly different syntax, instead of the --gtest_output parameter, it expects the --junitxml argument.

Therefore, I decided to add a small pytest_runner.py script to the project:

#!/usr/bin/env python
from __future__ import print_function

import os
import sys
import rospy
import pytest


def get_output_file():
    for arg in sys.argv:
        if arg.startswith('--gtest_output'):
            return arg.split('=xml:')[1]

    raise RuntimeError('No output file has been passed')


if __name__ == '__main__':
    output_file = get_output_file()
    test_module = rospy.get_param('test_module')
    runner_path = os.path.dirname(os.path.realpath(__file__))
    module_path = os.path.join(runner_path, test_module)

    sys.exit(
        pytest.main([module_path, '--junitxml={}'.format(output_file)])
    )

The runner script converts the output file parameter to the pytest supported format. Additionally, it passes the test directory pytest.

The test directory is constructed from the runner script path and a test_module name passed via ROS param.

The accompanying ROS test launcher file listener_test.launch looks as follows:

<launch>
  <node pkg="my_pkg" type="publisher" name="publisher" />
  <param name="test_module" value="listener"/>
  <test test-name="test_listener" pkg="my_pkg" type="pytest_runner.py" />
</launch>

rostest starts the publisher node in the my_pkg package and then runs the pytest runner. Therefore, all pytests will be run with the same publisher instance. If you want to restart the publisher node during tests, you need to separate them into different test_modules.

We can start the tests with catkin run_tests --this inside the ROS package directory.

As output, we get something along the lines of:

[Testcase: testtest_listener] ... ok

[ROSTEST]-----------------------------------------------------------------------

[my_pkg.rosunit-test_listener/test_listener_receives_something][passed]
[my_pkg.rosunit-test_listener/test_listener_receives_hello_mesage][passed]

SUMMARY
 * RESULT: SUCCESS
 * TESTS: 2
 * ERRORS: 0
 * FAILURES: 0

rostest log file is in /home/alexander/.ros/log/rostest-pc-26218.log
-- run_tests.py: verify result "/home/alexander/projects/ros/build/my_pkg/test_results/my_pkg/rostest-tests_listener_test.xml"

Unit tests vs ROS node tests

The really great thing about pytest is that it really makes unit testing easy and quick. To shorten you feedback loops I recommend you to clearly distinguish between unit tests and ROS node or integration tests.

A well-developed software module should be easily testable with unit test, if it is not you should probably reconsider your software design. With pytest you can place your unit tests physically close to your source code files, I prefer to place them in a tests folder inside the module.

Integration tests, on the other hand, test the integration between different components and take usually a long time to execute. In the case of ROS node tests, this includes spinning up multiple ROS nodes for example.

Since we as developers don't like to wait for tests to execute, we will very likely run the ROS node tests less frequently. Therefore, it makes sense to separate execution of unit tests from the execution of integration and ROS node tests during development.

For this purpose, I run pytest directly in my Python module inside the source directory, skipping rostest.

Nevertheless, we should not skip the unit tests in when running the rest of the tests:

<launch>
  <param name="test_module" value="../src"/>
  <test test-name="test_lib" pkg="my_pkg" type="pytest_runner.py" />
</launch>

Conclusion

In this blog post, you learned about using pytest for testing ROS nodes.

I uploaded the example project to
GitHub for
you to reproduce the steps described in this article.

I hope this blog post was interesting for you and I would like to hear your feedback. Don't forget to subscribe to get updated with new articles.

Your
Machine Koder