Skip to content

Cross-host communication

Learning Objectives

This tutorial extends an iceoryx2 system across host boundaries using a tunnel:

  1. Enable cross-host communication with the iox2 link tunnel CLI, without touching application code
  2. Embed the link in your own application for full control over its execution
  3. Drive the link in polling and reactive modes
  4. Switch the communication mechanism by instantiating a different carrier

So far Larry has been a self-contained system: his sensors, his control logic, and his actuators all reside on one host and talk over iceoryx2's shared memory. But shared memory does not extend beyond the boundary of the host.

Imagine now that the user wants to remotely track where Larry is and how much battery he has left. Some kind of dashboard hosted on another device on the same network.

The required data is already flowing through iceoryx2's shared memory. A link can hook into that flow and propagate it to other hosts. Since our dashboard will also run iceoryx2, we will use a link with a tunneling backend which propagates the raw shared memory samples over the network.

Tip

A tunnel can run over any communication mechanism, not just network communication. All that is required is an implementation of the Carrier traits available in iceoryx2-link-carrier.

For example, links have been used to extend communication between processors on the same board that can only communicate over inter-processor mechanisms.

Application setup

As in previous articles in this series, the data flowing through Larry are defined as ZeroCopySend records:

type
  [Iox2TypeName('BatteryState')]
  TBatteryState = record
  public
    ChargePercent: Single;
  end;

type
  TPosition
  public
    X: Single;
    Y: Single;
  end;

Note

The example above uses an [Iox2TypeName] attribute to set the payload type name for cross-language compatibility. Without this attribute, the name of the record would be used, without any 'T' prefix. In this case, that would be 'BatteryState' as well, so this attribute is not really needed.

For demonstration purposes, we use the event-driven communication execution pattern, which is the same approach used by many popular network communication libraries. However, with iceoryx2, you are free to choose the best approach that suits your application. The link can be driven in different ways. See execution control patterns for an overview.

Note

The link currently only supports the publish-subscribe and event messaging patterns. Support for request-response and blackboard is planned.

We set Larry up to send a notification on a corresponding event service along with every sample that is published:

var CycleTime := TDuration.FromSecs(1);

TIox2Log.SetLogLevelFromEnvOr(TLogLevel.Info);

var Node := TNodeBuilder.New
  .Name(TNodeName.Create('Larry').Value)
  .Create(TServiceType.Ipc)
  .Value;

var BatteryPublisher := Node
  .ServiceBuilder(TServiceName.Create('larry/battery').Value)
  .PublishSubscribe<TBatteryState>
  .OpenOrCreate
  .Value
  .PublisherBuilder
  .Create
  .Value;

var BatteryNotifier := Node
  .ServiceBuilder(TServiceName.Create('larry/battery').Value)
  .Event
  .OpenOrCreate
  .Value
  .NotifierBuilder
  .Create
  .Value;

var PositionPublisher := Node
  .ServiceBuilder(TServiceName.Create('larry/position').Value)
  .PublishSubscribe<TPosition>
  .OpenOrCreate
  .Value
  .PublisherBuilder
  .Create
  .Value;

var PositionNotifier := Node
  .ServiceBuilder(TServiceName.Create('larry/position').Value)
  .Event
  .OpenOrCreate
  .Value
  .NotifierBuilder
  .Create
  .Value;

while (Node.Wait(CycleTime).HasValue) do
begin
  var Battery := BatteryPublisher.LoanUninit.Value;

  var BatteryState: TBatteryState;
  BatteryState.ChargePercent := 87;
  var InitializedBattery := Battery.WritePayload(BatteryState);
  InitializedBattery.Send.Value;
  BatteryNotifier.Notify.Value;

  var Position := PositionPublisher.LoanUninit.Value;

  var Pos: TPosition;
  Pos.X := 1;
  Pos.Y := 2;
  InitializedPosition := Position.WritePayload(Pos);
  InitializedPosition.Send.Value;
  PositionNotifier.Notify.Value;
end;

In the dashboard application running on another host, we set up listeners to wake the thread when new data arrives, and subscribers to receive the data. A WaitSet is used to react to events from both listeners in a single thread.

TIox2Log.SetLogLevelFromEnvOr(TLogLevel.Info);

var Node := TNodeBuilder.New
  .Name(TNodeName.Create('Dashboard').Value)
  .Create(TServiceType.Ipc)
  .Value;

var BatterySubscriber := Node
  .ServiceBuilder(TServiceName.Create('larry/battery').Value)
  .PublishSubscribe<TBatteryState>
  .OpenOrCreate
  .Value
  .SubscriberBuilder
  .Create
  .Value;

var BatteryListener := Node
  .ServiceBuilder(TServiceName.Create('larry/battery').Value)
  .Event
  .OpenOrCreate
  .Value
  .ListenerBuilder
  .Create
  .Value;

var PositionSubscriber := Node
  .ServiceBuilder(TServiceName.Create('larry/position').Value)
  .PublishSubscribe<TPosition>
  .OpenOrCreate
  .Value
  .SubscriberBuilder
  .Create
  .Value;

var PositionListener := Node
  .ServiceBuilder(TServiceName.Create('larry/position').Value)
  .Event
  .OpenOrCreate
  .Value
  .ListenerBuilder
  .Create
  .Value;

var WaitSet := TWaitSetBuilder.New
  .Create(TServiceType.Ipc)
  .Value;

var BatteryGuard := WaitSet.AttachNotification(BatteryListener).Value;
var PositionGuard := WaitSet.AttachNotification(PositionListener).Value;

var OnEvent := function(const AAttachmentId: TWaitSetAttachmentId): TCallbackProgression
  begin
    if (AAttachmentId.HasEventFrom(BatteryGuard)) then
    begin
      // drain every pending notification, otherwise the 
      // TWaitSet wakes again immediately and spins
      BatteryListener.TryWait(
        procedure(const AEvent: TEventActivation)
        begin 
        end).Value;

      var Battery := BatterySubscriber.Receive.Value;
      while (Battery <> nil) do
      begin
        WriteLn('battery: ', Battery.Payload.ChargePercent, '%');
        Battery := BatterySubscriber.Receive.Value;
      end;
    end
    else if (AAttachmentId.HasEventFrom(PositionGuard)) then
    begin
      // drain every pending notification, otherwise the 
      // TWaitSet wakes again immediately and spins
      PositionListener.TryWait(
        procedure(const AEvent: TEventActivation)
        begin 
        end).Value;

      var Position := PositionSubscriber.Receive.Value;
      while (Position <> nil) do
      begin
        var Pos := Position.Payload;
        WriteLn('position: (', Pos.X, ', ', Pos.Y, ')');
        Position := PositionSubscriber.Receive.Value;
      end;
    end;

    Result := TCallbackProgression.Continue;
  end;

WaitSet.WaitAndProcess(OnEvent).Value;

// The guards *must* be released *before* the other objects!
BatteryGuard.Release;
PositionGuard.Release;

Neither application is aware that the data will be propagated over the network. They are both set up as regular iceoryx2 applications.

Extending over the network

With the applications set up, we now need to spin up a tunnel to extend the communication over the network. The tunnel can either be run in a separate process or embedded into the application. First, let's use the iceoryx2-cli to run the tunnel in a separate process.

Installing the CLI

Start by installing the iceoryx2-cli which is the entry point for all CLI commands:

cargo install iceoryx2-cli

Next, the CLI for a tunnel over a specific carrier must be installed. For this article, we shall use a tunnel over a Zenoh carrier:

cargo install iceoryx2-integrations-zenoh-link-tunnel-cli

The command iox2 link tunnel --list can be used to verify that the installation was successful. This command lists all installed tunnels discovered by the CLI.

The tunnel needs to be installed on both hosts.

Running a tunnel process

The CLI can now be used to start a tunnel process on both hosts:

iox2 link tunnel zenoh

Each tunnel discovers iceoryx2 services, both on its own host and on the peers reached through its carrier. Larry's tunnel sees larry/battery and larry/position, offers them to its peers and forwards all samples published to them over the wire. The dashboard's tunnel creates matching services on the dashboard host, receives the samples from the wire, and injects them into the iceoryx2 shared memory to be received by their subscribers. The two applications communicate over services, as if they were running on the same host.

By default the tunnel polls for new services and pending samples every 100 ms, however since the event-based communication pattern is used, it is possible to configure the tunnel to execute reactively.

The tunnel running on Larry only has work to do when new samples are published on the services that need to be propagated:

iox2 link tunnel zenoh --listener "larry/battery" --listener "larry/position"

The tunnel running on the dashboard host only has work to do when data arrives through the carrier:

iox2 link tunnel zenoh --reactive

Nothing here is application code. Both applications were written, compiled, and deployed with no knowledge that a tunnel exists. Communicating across the hosts is purely a deployment decision.

Note

The embeddable link is currently only available as a Rust library. C++, C, Python and Delphi applications must use the tunnel from a separate process with the iox2 link tunnel CLI shown above; the remainder of this section applies to Rust.

Running the tunnel as its own process is convenient, but it's not the only option. The link is a library, and it spawns no threads of its own so you have full control over when and where to run it. It can be executed within a dedicated time slot of a time-triggered architecture, or on a separate thread that you control, pinned to a specific core so as not to interfere with application execution.

First add the link, the tunnel backend, and your chosen carrier as dependencies:

iceoryx2-link = { version = "X.Y.Z" }
iceoryx2-link-tunnel = { version = "X.Y.Z" }
iceoryx2-integrations-zenoh-link-carrier = { version = "X.Y.Z" }
zenoh = { version = "1.9" }

The link can then be instantiated in your application from a node and a backend. Here the backend is a tunnel over the ZenohCarrier, but any other carrier can be dropped in, even your own. Once created, the link can be manually driven:

use core::time::Duration;
use iceoryx2::prelude::*;
use iceoryx2_integrations_zenoh_link_carrier::ZenohCarrier;
use iceoryx2_link::Link;
use iceoryx2_link_tunnel::Tunnel;

const POLL_INTERVAL: Duration = Duration::from_millis(100);

let config = iceoryx2::config::Config::global_config();
let node = NodeBuilder::new().config(config).create::<ipc::Service>()?;
let carrier = ZenohCarrier::create(zenoh::Config::default())?;
let mut link = Link::new(node, Tunnel::new(carrier, config));

while link.node().wait(POLL_INTERVAL).is_ok() {
    link.discover()?;
    link.propagate()?;
}

Execution is driven via two API calls. The discover() call reconciles which services exist locally and over the wire. The propagate() call moves the data in both directions: from the wire into iceoryx2 shared memory and vice versa. Calling the two on the link in a timed loop replicates the default behavior of the CLI.

Now let's take a look at how to replicate the reactive configurations provided by the CLI. Recapping the details, both links must be set up to react to different things:

  1. Larry's link reacts to the services notified when a sample is published
  2. The dashboard's link reacts to data arriving over the network

Since Larry's link does not react to network communication, it needs nothing from the carrier. Listeners are created for the services notified with each publish, and a WaitSet is set up to react to notifications from both of them in a single thread:

use iceoryx2::prelude::*;
use iceoryx2_integrations_zenoh_link_carrier::ZenohCarrier;
use iceoryx2_link::Link;
use iceoryx2_link_tunnel::Tunnel;

let config = iceoryx2::config::Config::global_config();
let node = NodeBuilder::new().config(config).create::<ipc::Service>()?;
let carrier = ZenohCarrier::create(zenoh::Config::default())?;
let mut link = Link::new(node, Tunnel::new(carrier, config));

// wake whenever Larry publishes locally, so freshly produced samples are
// pushed out promptly
let battery_listener = link
    .node()
    .service_builder(&"larry/battery".try_into()?)
    .event()
    .open_or_create()?
    .listener_builder()
    .create()?;
let position_listener = link
    .node()
    .service_builder(&"larry/position".try_into()?)
    .event()
    .open_or_create()?
    .listener_builder()
    .create()?;

let waitset = WaitSetBuilder::new().create::<ipc::Service>()?;
let _battery_guard = waitset.attach_notification(&battery_listener)?;
let _position_guard = waitset.attach_notification(&position_listener)?;

waitset.wait_and_process(|_| {
    let _ = battery_listener.try_wait(|_| {});
    let _ = position_listener.try_wait(|_| {});
    let _ = link.discover();
    let _ = link.propagate();

    CallbackProgression::Continue
})?;

The dashboard's link only cares about data arriving over the network, so it asks the link for a listener. This listener is notified when the carrier has new data or the services offered by the peers change. No WaitSet is needed here, since there is only one listener to wait on:

use iceoryx2::prelude::*;
use iceoryx2_integrations_zenoh_link_carrier::ZenohCarrier;
use iceoryx2_link::Link;
use iceoryx2_link_tunnel::Tunnel;

let config = iceoryx2::config::Config::global_config();
let node = NodeBuilder::new().config(config).create::<ipc::Service>()?;
let carrier = ZenohCarrier::create(zenoh::Config::default())?;
let mut link = Link::new(node, Tunnel::new(carrier, config));

// the listener wakes when the carrier has delivered something to ingest
// from the wire
let listener = link.listener()?;

while listener.blocking_wait(|_| {}).is_ok() {
    link.discover()?;
    link.propagate()?;
}

Both approaches run the same discover() and propagate() calls; they differ only in what decides when those calls happen. Once again, the link has no background threads, so you are free to choose where and when these operations run, allowing you to plan for and minimize unpredictability.

Switching the mechanism

Let's say that later in development you decide a different mechanism should carry the cross-host communication. Perhaps you want to reach the dashboard over the internet by integrating with existing MQTT infrastructure, or you need to connect processors on the same board that can only communicate over inter-processor mechanisms.

All that is required, is to switch out the carrier when creating the tunnel. In our examples above, we have been using Zenoh, so the links were of the type:

Link<ipc::Service, Tunnel<ZenohCarrier>>

The first parameter is the local iceoryx2 service type; the second is the backend, a tunnel over the Zenoh carrier. The link and the tunnel know nothing about Zenoh. They are written against the carrier traits, and discover() and propagate() are the same calls regardless of what carries the bytes.

Switching over to a different mechanism is therefore only a change of the carrier given to the tunnel on creation:

Link<ipc::Service, Tunnel<SomeOtherCarrier>>

The publishers, the subscribers, the embed loops, and the choice of polled or reactive execution are all untouched.

Over time, additional carrier implementations will be provided upstream alongside iceoryx2, however if there isn't an existing implementation for the mechanism you would like to use, it is possible to implement your own using the traits provided in iceoryx2-link-carrier.

Further Reading