Machinetalk explained Part 3: Technologies

This blog post is a continuation of my Machinetalk explained series describing the concepts and ideas behind the Machinetalk middleware.

This part is about the technologies and building blocks of Machinetalk ZeroMQ, Protobuf and DNS-SD over mDNS.

The previous parts are:

Message Transport – ZeroMQ

Machinetalk uses ZeroMQ as a transport library which is an asynchronous low-latency messaging library developed by iMatix.

iMatix authorizes the ZeroMQ library under the terms of the GNU LGPLv3 open-source license.

ZeroMQ is not only a networking library but also a concurrency framework. The messaging middleware abstracts the underlying networking protocols and provides socket communication over different transports, including in-process, inter-process, Transmission Control Protocol (TCP) and several multicast protocols.

Compared to other messaging middleware, ZeroMQ provides a lower level of abstraction. This leads to an overall smaller footprint and code as well as binary size compared to similar middleware solutions

. Basically, the ZeroMQ API is similar to Unix domain sockets.

The messaging library provides a small set of transport dependent Quality of Service (QoS) parameters. However, it is important to know that ZeroMQ does not provide application level QoS features. The reason stated in the ZeroMQ documentation is the following: “Implementing QoS on middleware level (as done in most messaging systems) proves inadequate once physical network issues are taken into account.”

. This design decision is a fundamental difference to middleware alternatives such as DDS.

As pointed out by Woodall analyzing the messaging library as middleware for ROS2, ZeroMQ is not a complete middleware solution to be used out of the box. It is required to combine it with assisting serialization and service discovery libraries to use it to create a complete middleware for distributed systems.

Similar to other messaging libraries, ZeroMQ provides different N-to-N networking patterns to connect sockets. The two networking patterns used in Machinetalk are the 1-to-1 ROUTER-DEALER pattern and the N-to-N PUB-SUB pattern (see the diagram below).

zeromq_patterns_overview-svg

The ZeroMQ ROUTER-DEALER pattern is a particular form of the request-reply messaging pattern that enables completely asynchronous communication between service provider and service consumer. Each dealer socket has a unique identity that the router can use to return messages to the corresponding destination nodes. Moreover, using the internal addressing mechanism it is easily possible to chain several router sockets to create a message broker.

zeromq_brokers-svg

The ZeroMQ PUB-SUB pattern uses message topics. A topic is a string describing the context of a message. A subscriber can subscribe to one, many or all topics at the same time. Depending on the transport either subscriber or publisher nodes filter the message topics.

ZeroMQ uses prefix matching and does not support wildcard or regular expression matching. Multicast transports provide no methods for the subscriber to communicate with the publisher. Therefore, topic filtering happens on the subscriber side. However, Source-Specific Multicast (SSM) transports provide better scalability for multiple subscribers compared to unicast transports since the distribution of
packages is managed by network interfaces and routers

.

In case an unicast transport such as TCP is used, ZeroMQ provides ways of informing publishers about subscribers subscribing and unsubscribing from topics. Machinetalk uses this feature is used for syncing purposes.

The popularity of web applications has increased in the last few years. Therefore, it seems only logical to provide a way for web applications to use Machinetalk. ZeroMQ is not available natively for web browsers. Thus, Machinetalk uses a message broker that bridges ZeroMQ connections to WebSockets, which is a standardized web technology.

Message Serialization

For data serialization, Machinetalk uses Protocol Buffers (Protobuf). “Protocol buffers are a flexible, efficient, automated mechanism for serializing structured data”. Google has developed Protobuf for in-house use in communication protocols and persistent data storage. Moreover, the specification and tools have been open sourced with version two of the Protobuf library.

In contrast to the XML and JSON Protobuf is a binary serialization format. Therefore, serialized Protobuf messages are not human readable. However, compared to most other binary formats such as NML, Protobuf provides full introspection support. Protobuf messages are in general not self-explanatory but can be decoded with the correct message description (see the diagram below).

Advantages of Protobuf compared to JSON and XML are faster parsing speeds and smaller message size

. Furthermore, Protobuf message fields are numbered and allow backward compatibility. The Protobuf message parser ignores unknown fields and initializes missing fields with default values. This behavior enables backward compatibility inside application code without explicit version checking.

protobuf_communication-svg

Protobuf uses an IDL for defining the message structure. The .proto files containing the IDL definitions are used as input for a language-specific code generator named protoc. The code generator creates source files in the target programming language which can be used to encode, decode and store data in memory.

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;
}

Protobuf per default supports the programming languages Python, C++, and Java. Community implementations provide further language support such as for JavaScript.

However, Protobuf also has disadvantages compared to self-descriptive serialization formats. Protobuf requires that message sender as well as message receiver possess a valid message descriptor. This limitation supplements additional complexity, especially in prototyping environments. Furthermore, generating the language-specific classes from the IDL definition files requires an extra step in the build chain.

protobuf_package-svg

The Protobuf library per default does not support self-descriptive messages. Therefore, the receiver of a Protobuf encoded message needs to know which type of message has been received to decode the message payload. The Machinetalk approach to solving this problem is to use a single top-level message to store the actual Machinetalk message type inside the Protobuf message as depicted in the picture above.

Furthermore, the Machinetalk message format has been designed with memory and computation power constrained environments in mind. Instead of using a Protobuf sub-message per Machinetalk message, the Machinetalk message format encourages the reuse of Protobuf fields between different Machinetalk messages. As a result, the correlation between Machinetalk messages and Protobuf messages is N to 1.

Service Discovery

For service discovery, Machinetalk uses the Zeroconf service discovery technology DNS-SD over mDNS. This technology enables service discovery on link-local networks without dedicated DNS servers. mDNS uses the DNS protocol over multicast UDP on IPv4 address 224.0.0.251 or IPv6 address FF02::FB and port 5353. Responses to mDNS queries are cached and pruned to reduce network traffic.

DNS-SD makes use of the following DNS records reserved for service discovery:

  • PTR records map service names to service instances. A PTR DNS query returns a list of
    service instances.
  • SRV records usually contain link-local hostnames which can be resolved by an A or AAAA mDNS query. Sophisticated clients can resolve link-local as well as wide-area DNS hostnames.
  • A and AAAA records are used to resolve IPv4 and IPv6 addresses with a hostname.
  • TXT records contain service specific properties stored as key-value pairs.

Machinetalk uses the DNS TXT records to announce additional information about services:

  • dsn: The dsn property stores the ZeroMQ address of a service. ZeroMQ does support multiple means of transport. However, ZeroMQ cannot automatically negotiate the protocol type upon service connection.
  • uuid: The uuid property contains the Machinekit instance Universally Unique Identifier (UUID). This Machinekit UUID is used to collate services with Machinekit instances.
  • instance: This property stores the process instance UUID of services. The value is used to detect service restarts and service announcements on multiple interfaces.
  • version: The version property carries the service version id. The version id is be used to negotiate service versions before connecting.

The illustration shows the typical flow of an mDNS/DNS-SD service discovery look-up.

service_discovery-svg

  1. First, the client creates a query for the service specification PTR record in the form of _<service>._sub._machinekit._tcp._local..
  2. The server returns a response with a list of DNS records in the following form: <service instance name>._<service>._sub._machinekit._tcp._local.
  3. The next step for the client is to query the SRV and TXT records for the received responses.
  4. Some server implementations return SRV as well as A or AAAA records as part of the reply to the PTR query to improve network efficiency. However, other implementations require querying for these records manually.

Services, Components and Protocols

ZeroMQ, Protobuf and mDNS/DNS-SD are the building blocks of Machinetalk. To glue this building blocks together, Machinetalk organizes itself into service and components.

machinetalk_terms2-svg

Machinetalk services consist of service consumer and service provider. Service providers announce themselves over the decentralized service discovery where a service browser can find them. Matching services connect to each other using a channel, which is a network connection using a defined Machinetalk protocol.

Machinetalk components work with one or multiple services. Services usually provide functionality that is useful for more than one particular component. For example, the halrcmd service provides methods to access HAL Remote components and also functions to manipulate other elements of an HAL configuration.

The next table illustrates the correlation between services and components. As can be seen, not every component uses multiple services.

Component Services Function
HAL Remote Component halrcomp, halrcmd HAL Remote component interface.
Application Launcher launcher, launchercmd Launching and monitoring Machinekit instances.
Application Config config Deploying user interfaces and configurations.
Application Command command Executing machine commands.
Application Status status Monitoring the machine status.
Application Error error Providing machine error messages.
Application Log log Providing Machinekit log information.
Preview preview, previewcmd Remote preview interface.

Presented in the following table is an overview of the Machinetalk services currently implemented as part of the Machinekit project. A service name uniquely identifies each service. Furthermore, the underlying ZeroMQ pattern is clearly defined by the channel protocol. Listed in the table is also the application inside the Machinekit project that implements the service.

Service Name ZeroMQ Pattern Implemented by Function
halrcmd ROUTER-DEALER Haltalk HAL remote API
halrcomp XPUB-SUB Haltalk HAL Remote component updates.
log PUB-SUB msgd Publishing Machinekit log messages.
launcher XPUB-SUB mklauncher Machinekit launcher configuration.
launchercmd ROUTER-DEALER mklauncher Starting and stopping Machinekit instances.
config ROUTER-DEALER configserver Deploying user interfaces and configurations.
status XPUB-SUB mkwrapper Machine status updates.
command ROUTER-DEALER mkwrapper Executing machine commands.
error PUB-SUB mkwrapper Publishing machine error messages.
file FTP mkwrapper Remote access to GCode program files.
previewcmd ROTUER-DEALER previewmodule Starting and stopping the remote preview interpreter.
preview PUB-SUB previewmodule Remote preview interpreter results.

Coming Up

In this blog, post we learned about the building blocks of Machinetalk: ZeroMQ, Protobuf, and mDNS/DNS-SD.

Furthermore, we examined the combination of these technologies to build the Machinetalk middleware.

In the next blog post of this series, I’m going to present the Machinetalk communication in more detail using HAL remote as an example.

I’m happy about all your feedback and comments. Particularly on this topic, it would be great if you post your concerns or any design problems you detected.

Your,
Machine Koder

2 thoughts on “Machinetalk explained Part 3: Technologies”

Leave a Reply

Your email address will not be published. Required fields are marked *