Cross system Communication

The connections between subsystems tends to be one of the most complicated and bug prone parts of a project. Thoughtful design of the API or definition of exactly what is communicated is the most important aspect, followed by the mechanism of communication.

Some important considerations when thinking about communication between subsystems are:

  • Cycle time differences: Is one running faster than the other?
  • State differences: Do the systems have states that can get out of phase and lock?
  • Error correlation: Does an error on one system cause reactions on the other one?

Managing external connections

There are several tools with which a task can interface to the outside world:

Direct IO manipulation is one way. Tasks can and should interface directly with the IO that they are responsible for.

Piper Is designed to tack on to the top of a task. Think of piper as a plugin system. If you have been working on your subsystem by entering commands into the watch window, a Piper Submodule interface is meant to take the place of entering commands directly, and instead does it on state changes based on PackML.

ATN Is designed to make loose connections between subsystems that don’t or shouldn’t know the details of the systems they are interfacing with.

UI Many operations of a subsystem come directly from the user.

Recipes Many subsystems can benefit from direct recipe integrations. Recipes can save parameters for operations that are then later called from other places. It is ok for recipes to directly set recipe values on variables local to many different tasks.

CNC or Sequencers The CNC kernal is very powerful as a process controller. It has the benefit that process changes are saved in text based, interpreted files. That makes modifying process logic easier. The downside is that the files are text based, and not compiled.. meaning that users can change them, and they can become difficult to track/maintain across many machines.

What if my task really does need to interface to another task/subsystem?

Be thoughtful and expose only what is required. Doing so will help keep side effects to a minimum. No matter what approach you take, this is the best advice we have.

Mechanisms for communication

Old School Approach

The tried and true method has always been global variables.

Create specific g[MySubystems]Api variables that include just the inputs and outputs of the subsystem. Do not include internal variables in this stucture. Keeping the interface small makes it more clear how to use it (and makes loading it into the watch window much easier).

The downside here is that the systems are tightly coupled. Changing MySubsystem WILL affect code in other areas of the system. Hopefully you were at least thoughtful enough to have a minimally feasible API.

New Hotness

Registration systems.

AlarmX Reactions

An example of this would be AlarmX Reactions. With AlarmX reactions, one system can raise an alarm event, other systems can monitor Alarms and impliment Reactions. AlarmX Reactions have the benefit of decoupling systems that don’t need to know the details about other systems. A well designed reaction system can simplify subsystem interfaces and effectifely decouple systems. A poorly designed reaction system is a complicated global variable system with all the usually downsides, plus the added headaches of obfuscation.

Downside of alarm reactions:

  1. String based Reactions are prone to mistypes
  2. Reactions don’t have handshakes? Can alarms be handled if they are raised and lowered before the reaction block is called? I actually don’t know…
  3. Reactions are one-way communication
  4. Linking non-alarm systems through the alarm system is unintuitive

Alternatives to registration

For context, lets take a look at some of the alternatives. If you want to jump right in, here is a helpful link.

This type of registration system greatly simplifies system in which many parallel or sub systems require operations from common devices. Let’s consider other implementations.

Centralize Checks

With centralized checks, all the checks for a given operation are done in the task that does the operation. Let’s look at an example:


void CoolingCyclic(){

    MotorPower = gMyMotor1.isPowered || gMyMotor2.isPowered || gMyMotor3.isPowered;

    TempControlActive = tempZone1.isActive || tempZone1.isActive || tempZone1.isActive;

    TempControlHot =    tempZone1.temp > tempZone1.tooHotTemp || 
                        tempZone1.temp > tempZone1.tooHotTemp || 
                        tempZone1.temp > tempZone1.tooHotTemp;

    enableCooling = MotorPower || TempControlActive || TempControlHot;

    if( enableCooling ){

        //...Do cooling

    }

}

Benefits:

All the cooling code is in one place, and you know exactly why your cooler is running.

But there are at least three problems with this approach:

  1. If you add or remove ANY devices that need cooling, you have to modify this code
  2. If the function of any of these devices change, you have to modify this code
  3. The functioning of the remote systems is now distributed. The code that determines if cooling is required for a specific subsystem is not within that subsystem

Let’s look at another approach:

Distributed Checks with consolidation

void CoolingCyclic(){

    CoolingRequired = 0;
    for( motor=0; motor< max_motor_index ){

        CoolingRequired  = CoolingRequired || gCooling.motorRequiresCooling[motor];        

    }

    for( tempController=0; tempController< max_tempController_index ){

        CoolingRequired =   CoolingRequired || gCooling.tempControllerCoolingRequired[tempController];        

    }

    enableCooling = CoolingRequired;

    if( enableCooling ){

        //...Do cooling

    }

}

void MotorSubSystem(){

    gCooling.motorRequiresCooling[ THIS_MOTORS_INDEX ] = thismotor.isPowered 
                || thismotor.temp > configuration.overtemp;

}

Benefits:

  1. The code that decides if each subsystem requires cooling can just be moved to that subsystem
  2. It is clear when cooling is required

Problems:

  1. We have to manage what index each device owns
  2. Adding new logic for when cooling turns on/off requires a lot of infrastructure changes
  3. This code may still change when new devices are added or removed

What if we try a similar approach, but without consolidation?

Distributed Checks without consolidation

void CoolingCyclic(){

    enableCoolingTof.Enable = gCooling.executeCoolingOn;
    enableCoolingTof.PT = 2000;

    if( enableCoolingTof.Q ){

        //...Do cooling

    }

    gCooling.executeCoolingOn = false;

}

void MotorSubSystem(){

    if( thismotor.isPowered || thismotor.temp > configuration.overtemp ){

        gCooling.executeCoolingOn = true;

    }
}

Benefits:

  1. The code that decides if each subsystem requires cool can be moved to THAT subsystem

Problems:

  1. The cooling code can not be very intellegent
  2. Instead, time based check-ins or complicated logic decides when to disable cooling

Now let’s take a look at Loupe’s solution

AllTogetherNow

ATN takes some of the concepts of reactions and applies it to more than just alarming. ATN implements a registration system where systems can register statuses for other systems to read, and subscribe to commands that other systems can write to. A key features of ATN is a “1 to many” and “many to 1” API.

Warning
The following code snippets are Pseudocode. Function names and parameters may be changed/truncated for clarity. Please see the Library docs for reference.

Statuses

Subsystems can register statuses to topics. Examples:

  • CoolingRequired
  • InhibitMotion
  • MyParentSystem.subsystemsReady

Many subsystems can register to each of the statuses. Registration should happen in the init function of a task, while status updates happen in the cyclic function.

void INIT(){

   //Register a state to be checked

   //Cooling is required if the system is powered on OR has a high temp
   registerState( gStates.CoolingRequired, gMysubsystem.status.powerOn );
   registerState( gStates.CoolingRequired, gMysubsystem.status.highTemp );

}

void CYCLIC(){

   //Set the status
   gMysubsystem.status.highTemp = gMysubsystem.status.temp > Configuration.highTemp;

   // gMysubsystem.status.powerOn is inherent to the subsystem, so no additional work is required
}

These “States” can be checked using

    //If anything has a registered "Cooling required bit" that is true, enable cooling
    // If nothing is registered to CoolingRequired, fallback to false.
   enableCooling = stateAnyTrue( gStates.CoolingRequired, false );
   ...
   stateAllTrue( gStates.PoweredOn, fallback  );
   stateAnyFalse( gStates.MotionOk, fallback  );
   stateAllFalse( gStates.SystemFault, fallback  );

Commands

Many subsystems can subscribe to group commands. Examples:

  • PowerOn
  • AbortAll
  • LoadRecipe
void INIT(){

    subscribeCommandBool( gCommands.PowerOn, mySub.command.powerOn );

}

void CYCLIC(){
    
    if( mySub.command.powerOn ){

        //Do power on
        IO.doEnablePower = true;

        //No need to respond.        
    }

    mySub.command.powerOn = false;

}

These commands can be triggered throughout the code using:

ExecuteCommand( gCommands.PowerOn )

Combining Commands and Statuses with handshaking

ATN implements a PLCOpen interface to simplify setting commands and getting status back from “0 to many” subsystems.

From the perspective of the caller, a PLCOpen function block can trigger actions on subscribers.

...

STATE_POWER_ALL:
   // States with function blocks
   // With Error, Busy, Done bits
   IF systemPowerOn.error OR systemPowerOn.aborted THEN
       // Handle error
   ELSIF systemPowerOn.done THEN
       // Do next thing
   ELSIF NOT systemPowerOn.busy THEN
       // Execute FUB
       systemPowerOn.Execute = true;
   END_IF

...

systemPowerOn.Command = gCommands.PowerOn;
systemPowerOn();

From the perspective of the subsystem, the commands are handled as normal. The additional code is registering for the command, and responding with a status.

void INIT(){

    subscribePLCOpen( gCommands.PowerOn, mySub.command.powerOn, mySub.internal.plcopen );

}

void CYCLIC(){
    
    if( mySub.command.powerOn ){

        //Do power on

        //Respond to PLCOpen calls
        mySub.internal.plcopen.status = ERR_OK;

    }

    mySub.command.powerOn = false;

}