In a previous article, I explained how Machinekit and ROS could be combined to build fully autonomous robots. In this article, I focus on my project of retro-fitting an industrial robot arm with Machinekit and using it with ROS.
The combination of ROS and Machinekit leverages the low-level real-time motion control capabilities of Machinekit and the high-level robotic tools that ROS provides.
DISCLAIMER: The robot in the pictures and videos is not a Tormach robot, I just put a Tormach sticker on it.
The manipulator I used for my retrofitting project is a Borunte BRTIRUS0805A six degrees of freedom (6-DOF) robot arm. It can lift approximately 5kg, so it's great for projects which don't require moving around a high payload.
It features a closed-loop motor-control system with Innovance I620P AC servo drives and 20-bit serial absolute encoders.
Initially, it came with a proprietary robot controller. Looking at the main board of the control computer, I could find several Lattice FPGAs, which are probably used for step generation and encoder feedback.
Moreover, it came with a control pendant, with a standard industrial robot control software, so not very intuitive and easy to use.
Since my goal was to make this robotic arm work with Machinekit and ROS, I threw out the original controller in favor of some Mesa-Electronics FPGA Anything I/O Ethernet cards. In particular, I used one 7i80 as a control interface, one 7i52 to interface with the differential-pair step-generation inputs and encoder feedback outputs, and two 7i37 Isolated Anything-IO adapters to connect the I/Os of the robot.
The Project Setup
My software setup is divided into two parts:
The Machinekit HAL configuration works wholly stand-alone and does not require ROS to work. This has the advantage that further applications and projects, such as controlling the robot via Autodesk Mimic or Blender, do not depend on ROS.
The ROS configuration plugs into the HAL configuration via the
hal_ros_control implements a generic
ros_controller providing the standard ROS interface on one the ROS side and joint position, velocity or torque outputs on the HAL side. In our case, we use position command feedback, since we use position control mode on the AC servos.
The Machinekit HAL Configuration
The HAL configuration consists of the following parts:
- position control interface via hostmot2 to the MESA electronics anything I/O card
- lots of I/Os to control the breaks and AC servo signals
- read-out of the absolute encoder position via Modbus
- a safety and software emergency stop chain
- a lamp controller for the robots signal lamp
- a HAL component to control the RobotIQ HAND-E via Modbus
The HAL configuration can be used with the real hardware or in simulation mode to test with any robot attached.
The ROS Configuration
The ROS configuration consists of the MoveIt! configuration for the Borunte robot and a support package.
The support package contains the robot URDF description as well as configuration files for the
ros_controller. Moreover, it defines the connections of the robot I/O pins and
hal_hw_interface to the HAL configuration.
Host Computer Setup
Both, the ROS configuration and the HAL configuration, run on one x86 4-core computer.
As OS we use Debian Stretch Linux with a PREEMPT_RT kernel.
Two cores of the computer are isolated for use with RT processes; the other two cores run userland processes. This ensured by passing the parameter
isolcpus=2,3 kernel to the modeline.
Why do we use two cores for RT and not only one? On many consumer-grade CPUs, multiple cores share the same L1 cache. Using only L1 cache for multiple processes means, that processes on one core can affect the latency of processes on another core by using up the L1 cache. Luckily, some consumer-grade and most server CPUs have non-uniform memory access (NUMA). This means that L1 cache is not shared between all CPUs, but only a CPU-set. For our computer, these CPU sets are 0-1 and 2-3.
Furthermore, we use the Linux cgroups to partition the executed processes into RT and non-RT groups and to bind to them to a pre-defined CPU set. This ensures that only RT processes are executed on CPUs 2 and 3.
Using this technique, we get RT jitter consistently below 3us, which is an excellent value.
Since the Machinekit and ROS setup is somewhat complicated to install, we decided to use Docker to run our application.
In short, the complete ROS and Machinekit installation moved into a ROS image. This has the considerable advantage that one can develop on a performant development computer and still have the same installation as on the target platform.
Moreover, building ROS and Machinekit can take some time on a low-performance CPU. Using a Docker-based setup, building the Docker image can be done on a high-performance workstation, cutting the build time in 1/10th.
An additional advantage of the Docker-based setup is that a specific project version can be matched with a Docker image. This enables testing of new installations or rolling back to older releases without any problems.
The key points to make this work is PREEMPT_RT and Open GL GUI support in Docker. Both are possible, but I will explain this in another blog post.
This project proves that it is possible to build fully open source industrial robots with Machnekit and ROS.
The source code for the project is available on GitHub. Instructions on how to set up a system for ROS and Machinekit will follow.
- borunte_hal - The Machinekit HAL configuration.
- borunte_robot - The ROS configuration.
- hal_ros_control - ROS controller for Machinekit HAL.
Moreover, using Docker not only to deploy web applications but also to deploy complete ROS and Machinekit installations proves useful.
If you have enjoyed reading this article, please leave a comment.
“The key points to make this work is PREEMPT_RT and Open GL GUI support in Docker. Both are possible, but I will explain this in another blog post.”
Is it really possible to run Machinekit in Docker? As far as I know, Docker is not a realtime environment. I once had tried and failed to run LinuxCNC in Docker, because it cannot create shared memory in the kernel space.
Can't wait for the coming post about Docker and PREEMPT_RT.
Yes, it is possible since Docker itself does not interfere with the Linux RT kernel.
The quintessence is to use the `--privileged` mode, and in case you choose to use `partrt` for CPU isolation, you also need to modify Dockers `cgroups` configuration.
The blog post about this topic is on my TODO shortlist.
Pingback: Combining Machinekit and ROS to create Open Source Robotic Systems
Very impressive what you have done, I have many old industrial robots I am going to retrofit using a similar method,
What are you using for a teach pendant?
For this particular project, we use a touchscreen monitor with a Qt-based UI.
Unfortunately, there is no out of the box solution that works similar to the dialog based UIs or standard teach pendants of industrial robots in ROS. I'm working on a custom solution for a customer.
Just wondering, would you as a retrofitter be interested in such a solution, if available?
Depending on price and the functionality of said custom interface yes I would be interested,
Do you have some examples of what you have developed?
Nothing online. I will contact you privately.
What closed loop frequency were you able to achieve with your controller running in ROS with the RT Patch?
We run at 1kHz in ROS-Control / HAL. On the MESA cards themselves, the closed loop runs at a much higher frequency to run the position or velocity control with the servos/encoders.
Thats amazing thanks! and which of these are you using: https://www.gigabyte.com/us/Mini-PcBarebone
Oops ignore - you're using the mesa card!
Quick question, how did you manage to run a GUI application in docker? Did you use X11 forwarding to display the GUI?
Yes, exactly. This script here should give you an idea how it works: https://github.com/machinekoder/borunte-docker/blob/master/docker-dev.sh
An alternative method is to use VNC to run the application.
I've been looking for better control software for the open-hardware AnninRobotics AR3, as what's provided is quite basic... adding MachineKit + ROS would make this a fully open-source project all the way up and down the stack.
There doesn't appear to have been much of an attempt to do this (best I could find was https://www.anninrobotics.com/forum/general-discussion/beagle-bone-black-machine-kit-ar2) so I might try my hand at it. One complication is that my setup is primarily ROS2, and ros2_control was just recently anounced for ROS2 Foxy (https://discourse.ros.org/t/announcing-ros2-control-for-foxy/18274). I expect your hal_ros_control would need a refactor to use the new release.
Do you have any suggestions steps to adapt your code to support the AR3 using ROS2? Thanks!
"Do you have any suggestions steps to adapt your code to support the AR3 using ROS2? Thanks!"
I was also interested in this, so asked on Github here: https://github.com/zultron/hal_ros_control/issues/4
This is interesting! I have been looking for an example on how to use Blender simulation with ROS. I am new with Blender and I have a project that will use Blender and integrate it with a manipulator via ROS/ROS2/micro-ros. Would you mind to share your robot's blender model and gazebo model so I can reproduce and try it myself? Thank you!
Asked here: https://github.com/zultron/hal_ros_control/issues/4