Cross-host communication
Learning Objectives
This tutorial extends an iceoryx2 system across host boundaries using a tunnel:
- Enable cross-host communication with the
iox2 link tunnelCLI, without touching application code - Embed the link in your own application for full control over its execution
- Drive the link in polling and reactive modes
- Switch the communication mechanism by instantiating a different carrier
So far Larry has been a self-contained system: his sensors, his control logic, and his actuators all reside on one host and talk over iceoryx2's shared memory. But shared memory does not extend beyond the boundary of the host.
Imagine now that the user wants to remotely track where Larry is and how much battery he has left. Some kind of dashboard hosted on another device on the same network.
The required data is already flowing through iceoryx2's shared memory. A link can hook into that flow and propagate it to other hosts. Since our dashboard will also run iceoryx2, we will use a link with a tunneling backend which propagates the raw shared memory samples over the network.
Tip
A tunnel can run over any communication mechanism, not just network communication. All that is required is an implementation of the Carrier traits available in iceoryx2-link-carrier.
For example, links have been used to extend communication between processors on the same board that can only communicate over inter-processor mechanisms.
Application setup
As in previous articles in this series, the data flowing through Larry are defined as ZeroCopySend records:
type
[Iox2TypeName('BatteryState')]
TBatteryState = record
public
ChargePercent: Single;
end;
type
TPosition
public
X: Single;
Y: Single;
end;
Note
The example above uses an [Iox2TypeName] attribute to set the payload type name for cross-language compatibility. Without this attribute, the name of the record would be used, without any 'T' prefix. In this case, that would be 'BatteryState' as well, so this attribute is not really needed.
For demonstration purposes, we use the event-driven communication execution pattern, which is the same approach used by many popular network communication libraries. However, with iceoryx2, you are free to choose the best approach that suits your application. The link can be driven in different ways. See execution control patterns for an overview.
Note
The link currently only supports the publish-subscribe and event messaging patterns. Support for request-response and blackboard is planned.
We set Larry up to send a notification on a corresponding event service along with every sample that is published:
var CycleTime := TDuration.FromSecs(1);
TIox2Log.SetLogLevelFromEnvOr(TLogLevel.Info);
var Node := TNodeBuilder.New
.Name(TNodeName.Create('Larry').Value)
.Create(TServiceType.Ipc)
.Value;
var BatteryPublisher := Node
.ServiceBuilder(TServiceName.Create('larry/battery').Value)
.PublishSubscribe<TBatteryState>
.OpenOrCreate
.Value
.PublisherBuilder
.Create
.Value;
var BatteryNotifier := Node
.ServiceBuilder(TServiceName.Create('larry/battery').Value)
.Event
.OpenOrCreate
.Value
.NotifierBuilder
.Create
.Value;
var PositionPublisher := Node
.ServiceBuilder(TServiceName.Create('larry/position').Value)
.PublishSubscribe<TPosition>
.OpenOrCreate
.Value
.PublisherBuilder
.Create
.Value;
var PositionNotifier := Node
.ServiceBuilder(TServiceName.Create('larry/position').Value)
.Event
.OpenOrCreate
.Value
.NotifierBuilder
.Create
.Value;
while (Node.Wait(CycleTime).HasValue) do
begin
var Battery := BatteryPublisher.LoanUninit.Value;
var BatteryState: TBatteryState;
BatteryState.ChargePercent := 87;
var InitializedBattery := Battery.WritePayload(BatteryState);
InitializedBattery.Send.Value;
BatteryNotifier.Notify.Value;
var Position := PositionPublisher.LoanUninit.Value;
var Pos: TPosition;
Pos.X := 1;
Pos.Y := 2;
InitializedPosition := Position.WritePayload(Pos);
InitializedPosition.Send.Value;
PositionNotifier.Notify.Value;
end;
In the dashboard application running on another host, we set up listeners to wake the thread when new data arrives, and subscribers to receive the data. A WaitSet is used to react to events from both listeners in a single thread.
TIox2Log.SetLogLevelFromEnvOr(TLogLevel.Info);
var Node := TNodeBuilder.New
.Name(TNodeName.Create('Dashboard').Value)
.Create(TServiceType.Ipc)
.Value;
var BatterySubscriber := Node
.ServiceBuilder(TServiceName.Create('larry/battery').Value)
.PublishSubscribe<TBatteryState>
.OpenOrCreate
.Value
.SubscriberBuilder
.Create
.Value;
var BatteryListener := Node
.ServiceBuilder(TServiceName.Create('larry/battery').Value)
.Event
.OpenOrCreate
.Value
.ListenerBuilder
.Create
.Value;
var PositionSubscriber := Node
.ServiceBuilder(TServiceName.Create('larry/position').Value)
.PublishSubscribe<TPosition>
.OpenOrCreate
.Value
.SubscriberBuilder
.Create
.Value;
var PositionListener := Node
.ServiceBuilder(TServiceName.Create('larry/position').Value)
.Event
.OpenOrCreate
.Value
.ListenerBuilder
.Create
.Value;
var WaitSet := TWaitSetBuilder.New
.Create(TServiceType.Ipc)
.Value;
var BatteryGuard := WaitSet.AttachNotification(BatteryListener).Value;
var PositionGuard := WaitSet.AttachNotification(PositionListener).Value;
var OnEvent := function(const AAttachmentId: TWaitSetAttachmentId): TCallbackProgression
begin
if (AAttachmentId.HasEventFrom(BatteryGuard)) then
begin
// drain every pending notification, otherwise the
// TWaitSet wakes again immediately and spins
BatteryListener.TryWait(
procedure(const AEvent: TEventActivation)
begin
end).Value;
var Battery := BatterySubscriber.Receive.Value;
while (Battery <> nil) do
begin
WriteLn('battery: ', Battery.Payload.ChargePercent, '%');
Battery := BatterySubscriber.Receive.Value;
end;
end
else if (AAttachmentId.HasEventFrom(PositionGuard)) then
begin
// drain every pending notification, otherwise the
// TWaitSet wakes again immediately and spins
PositionListener.TryWait(
procedure(const AEvent: TEventActivation)
begin
end).Value;
var Position := PositionSubscriber.Receive.Value;
while (Position <> nil) do
begin
var Pos := Position.Payload;
WriteLn('position: (', Pos.X, ', ', Pos.Y, ')');
Position := PositionSubscriber.Receive.Value;
end;
end;
Result := TCallbackProgression.Continue;
end;
WaitSet.WaitAndProcess(OnEvent).Value;
// The guards *must* be released *before* the other objects!
BatteryGuard.Release;
PositionGuard.Release;
Neither application is aware that the data will be propagated over the network. They are both set up as regular iceoryx2 applications.
Extending over the network
With the applications set up, we now need to spin up a tunnel to extend the communication over the network. The tunnel can either be run in a separate process or embedded into the application. First, let's use the iceoryx2-cli to run the tunnel in a separate process.
Installing the CLI
Start by installing the iceoryx2-cli which is the entry point for all CLI commands:
Next, the CLI for a tunnel over a specific carrier must be installed. For this article, we shall use a tunnel over a Zenoh carrier:
The command iox2 link tunnel --list can be used to verify that the installation was successful. This command lists all installed tunnels discovered by the CLI.
The tunnel needs to be installed on both hosts.
Running a tunnel process
The CLI can now be used to start a tunnel process on both hosts:
Each tunnel discovers iceoryx2 services, both on its own host and on the peers reached through its carrier. Larry's tunnel sees larry/battery and larry/position, offers them to its peers and forwards all samples published to them over the wire. The dashboard's tunnel creates matching services on the dashboard host, receives the samples from the wire, and injects them into the iceoryx2 shared memory to be received by their subscribers. The two applications communicate over services, as if they were running on the same host.
By default the tunnel polls for new services and pending samples every 100 ms, however since the event-based communication pattern is used, it is possible to configure the tunnel to execute reactively.
The tunnel running on Larry only has work to do when new samples are published on the services that need to be propagated:
The tunnel running on the dashboard host only has work to do when data arrives through the carrier:
Nothing here is application code. Both applications were written, compiled, and deployed with no knowledge that a tunnel exists. Communicating across the hosts is purely a deployment decision.
Embedding the link
Note
The embeddable link is currently only available as a Rust library. C++, C, Python and Delphi applications must use the tunnel from a separate process with the iox2 link tunnel CLI shown above; the remainder of this section applies to Rust.
Running the tunnel as its own process is convenient, but it's not the only option. The link is a library, and it spawns no threads of its own so you have full control over when and where to run it. It can be executed within a dedicated time slot of a time-triggered architecture, or on a separate thread that you control, pinned to a specific core so as not to interfere with application execution.
First add the link, the tunnel backend, and your chosen carrier as dependencies:
iceoryx2-link = { version = "X.Y.Z" }
iceoryx2-link-tunnel = { version = "X.Y.Z" }
iceoryx2-integrations-zenoh-link-carrier = { version = "X.Y.Z" }
zenoh = { version = "1.9" }
The link can then be instantiated in your application from a node and a backend. Here the backend is a tunnel over the ZenohCarrier, but any other carrier can be dropped in, even your own. Once created, the link can be manually driven:
use core::time::Duration;
use iceoryx2::prelude::*;
use iceoryx2_integrations_zenoh_link_carrier::ZenohCarrier;
use iceoryx2_link::Link;
use iceoryx2_link_tunnel::Tunnel;
const POLL_INTERVAL: Duration = Duration::from_millis(100);
let config = iceoryx2::config::Config::global_config();
let node = NodeBuilder::new().config(config).create::<ipc::Service>()?;
let carrier = ZenohCarrier::create(zenoh::Config::default())?;
let mut link = Link::new(node, Tunnel::new(carrier, config));
while link.node().wait(POLL_INTERVAL).is_ok() {
link.discover()?;
link.propagate()?;
}
Execution is driven via two API calls. The discover() call reconciles which services exist locally and over the wire. The propagate() call moves the data in both directions: from the wire into iceoryx2 shared memory and vice versa. Calling the two on the link in a timed loop replicates the default behavior of the CLI.
Now let's take a look at how to replicate the reactive configurations provided by the CLI. Recapping the details, both links must be set up to react to different things:
- Larry's link reacts to the services notified when a sample is published
- The dashboard's link reacts to data arriving over the network
Since Larry's link does not react to network communication, it needs nothing from the carrier. Listeners are created for the services notified with each publish, and a WaitSet is set up to react to notifications from both of them in a single thread:
use iceoryx2::prelude::*;
use iceoryx2_integrations_zenoh_link_carrier::ZenohCarrier;
use iceoryx2_link::Link;
use iceoryx2_link_tunnel::Tunnel;
let config = iceoryx2::config::Config::global_config();
let node = NodeBuilder::new().config(config).create::<ipc::Service>()?;
let carrier = ZenohCarrier::create(zenoh::Config::default())?;
let mut link = Link::new(node, Tunnel::new(carrier, config));
// wake whenever Larry publishes locally, so freshly produced samples are
// pushed out promptly
let battery_listener = link
.node()
.service_builder(&"larry/battery".try_into()?)
.event()
.open_or_create()?
.listener_builder()
.create()?;
let position_listener = link
.node()
.service_builder(&"larry/position".try_into()?)
.event()
.open_or_create()?
.listener_builder()
.create()?;
let waitset = WaitSetBuilder::new().create::<ipc::Service>()?;
let _battery_guard = waitset.attach_notification(&battery_listener)?;
let _position_guard = waitset.attach_notification(&position_listener)?;
waitset.wait_and_process(|_| {
let _ = battery_listener.try_wait(|_| {});
let _ = position_listener.try_wait(|_| {});
let _ = link.discover();
let _ = link.propagate();
CallbackProgression::Continue
})?;
The dashboard's link only cares about data arriving over the network, so it asks the link for a listener. This listener is notified when the carrier has new data or the services offered by the peers change. No WaitSet is needed here, since there is only one listener to wait on:
use iceoryx2::prelude::*;
use iceoryx2_integrations_zenoh_link_carrier::ZenohCarrier;
use iceoryx2_link::Link;
use iceoryx2_link_tunnel::Tunnel;
let config = iceoryx2::config::Config::global_config();
let node = NodeBuilder::new().config(config).create::<ipc::Service>()?;
let carrier = ZenohCarrier::create(zenoh::Config::default())?;
let mut link = Link::new(node, Tunnel::new(carrier, config));
// the listener wakes when the carrier has delivered something to ingest
// from the wire
let listener = link.listener()?;
while listener.blocking_wait(|_| {}).is_ok() {
link.discover()?;
link.propagate()?;
}
Both approaches run the same discover() and propagate() calls; they differ only in what decides when those calls happen. Once again, the link has no background threads, so you are free to choose where and when these operations run, allowing you to plan for and minimize unpredictability.
Switching the mechanism
Let's say that later in development you decide a different mechanism should carry the cross-host communication. Perhaps you want to reach the dashboard over the internet by integrating with existing MQTT infrastructure, or you need to connect processors on the same board that can only communicate over inter-processor mechanisms.
All that is required, is to switch out the carrier when creating the tunnel. In our examples above, we have been using Zenoh, so the links were of the type:
The first parameter is the local iceoryx2 service type; the second is the backend, a tunnel over the Zenoh carrier. The link and the tunnel know nothing about Zenoh. They are written against the carrier traits, and discover() and propagate() are the same calls regardless of what carries the bytes.
Switching over to a different mechanism is therefore only a change of the carrier given to the tunnel on creation:
The publishers, the subscribers, the embed loops, and the choice of polled or reactive execution are all untouched.
Over time, additional carrier implementations will be provided upstream alongside iceoryx2, however if there isn't an existing implementation for the mechanism you would like to use, it is possible to implement your own using the traits provided in iceoryx2-link-carrier.
Further Reading
-
The control-flow pattern the dashboard and tunnel employ.
-
Overview of execution patterns possible with
iceoryx2. -
The feature that this tutorial puts to work.