Group Management Reference

Group management provides consistent list of active DGI processes as well as identifies a primary process for initiating distributed algorithms.

Using Group Management

GMAgent::ProcessPeerList is available by including gm/GroupManagement.hpp

Group management will send all modules in the system a PeerList message when the list of active processes in the system changes. This message will also contain the identity of the leader process. Group management provides a static method for to aid processing this message.

PeerSet ProcessPeerList(const PeerListMessage & msg)

Handles Processing a PeerList.

GMAgent::ProcessPeerList

Description:
Provides a utility function for correctly handling incoming peer lists.
Return
A PeerSet with all nodes in the group.
Parameters
  • msg -

    The message to parse

This method returns a PeerList of the processes in the group. The sender of the message will always be the group leader.

An example of processing the PeerList message is in Receiving Messages

Physical Topology

Group management can be configured to respect the topology of the physical network: it will not group two DGI unless a physical path exists between the two processes (as opposed to a cyber path).

Directions on configuring physical topology can be found in Physical Topology

Embedding Group State In Simulation

The group state is by manipulating a “Logger” device on the FREEDM system. The value of device is updated each time the check or timeout procedure is called, that is, once at the beginning of the Group Management phase.

Data is stored in the device as a bit field. Because of the way data is passed to and from the RTDS and PSCAD the data is transported as a float, although the individual bits are unaffected by this process, it may be necessary to convert the float to an unsigned integer to be able to preform the bit fiddling needed to access the information.

This setup assumes that all DGI are aware of all other DGI in the system and that all the Group Management status tables are the same. That is, there is not a DGI some DGI is aware of that some other DGI isn’t aware of. This is reasonable because of the experiments we are currently running, and the fact that the container for the other DGIs in the system is a map so when iterated the items are returned in alpha numeric order.

UUIDS ARE ENTERED IN THE BITFIELD IN ASCENDING ALPHANUMERIC ORDER.

## Bit Field Structure

Note that 2^0 is set based on the endianess of an x86_64 machine.

  1. (2^0) - Set to 1 if the DGI editing the device is the coordinator.
  1. (2^1) - Set to 1 if the DGI with the first UUID is in the same group as the DGI editing the bitfield
  1. (2^2) - Set to 1 if the DGI with the second UUID is in the same group as the DGI editing the bitfield.
  1. (2^n) - Set to 1 if the DGI with the nth UUID is in the same group as the DGI editing the bitfield.

Bits of non-members will be set to 0. The SGI will always set the bit relating to its own UUID to 1.

In order to use this feature, you need to define a Logger device with a “groupStatus” field. The DGI will write its group state to the first Logger device on the system.

Congestion Notifications

The group management module supports congestion notifications. The group management module responds to two different type of congestion notifications: hard and soft. A soft notification is generated when a network device suspects network congestions are in a state where the DGI may not meet its real-time communication deadlines.

When the DGI receives a soft notification, the group management module will switch into a slower “maintenance” mode. In this mode, elections are disabled for any DGI that receives the notification. Additionally, the time window for the delivery for the AYC/AYT messages and their responses are extended to compensate for the expected congestion.

When a hard notification is received by a leader, the leader will perform a group division. The leader generates two new groups from its current group and distributes the new group information the processes in its group.

The DGI currently implements a random group split: the leader process will randomly assign processes to one of two groups, attempting to create two equally sized groups. This has been shown to reduce queue usage for large groups. An area for possible improvement it to target group division around the placement of resources and network congestion.

Hard notifications also enable the behavior of soft notifications.

Implementation Details

Algorithm

Group Management is an implementation of the Garcia-Molina Invitation Election Algorithm found in “Elections in a Distributed System” published 1982 in IEEE Transactions on Computers.

GMAgent Reference

class freedm::broker::gm::GMAgent

Declaration of Garcia-Molina Invitation Leader Election algorithm.

Public Types

enum [anonymous]

Module states.

Values:

NORMAL
DOWN
RECOVERY
REORGANIZATION
ELECTION

Public Functions

GMAgent()

Constructor for using this object as a module.

GMAgent::GMAgent

Description:
Constructor for the group management module.
Limitations:
None
Precondition:
None
Postcondition:
Object initialized and ready to enter run state.

~GMAgent()

Module destructor.

GMAgent::~GMAgent

Description:
Class desctructor
Precondition:
None
Postcondition:
The object is ready to be destroyed.

int Run()

Called to start the system.

GMAgent::Run

Description:
Main function which initiates the algorithm
Precondition:
connections to peers should be instantiated
Postcondition:
execution of group management algorithm

Public Static Functions

CPeerNode ProcessConnectedPeer(const ConnectedPeerMessage & cpm)

Converts a connected peer field to a CPeerNode.

GMAgent::ProcessConnectedPeer

Description:
Processes a connected peer and returns the CPeerNode object for that peer.
Precondition:
None
Postcondition:
If this process didn’t know about this peer, its information will be added to the connection manager
Parameters
  • cpm -

    The Message with information about the peer

PeerSet ProcessPeerList(const PeerListMessage & msg)

Handles Processing a PeerList.

GMAgent::ProcessPeerList

Description:
Provides a utility function for correctly handling incoming peer lists.
Return
A PeerSet with all nodes in the group.
Parameters
  • msg -

    The message to parse

Protected Functions

CPeerNode GetMe()

Gets a CPeerNode representing this process.

GetMe

Description:
Gets a CPeerNode that refers to this process.
Return
A CPeerNode referring to this process.

std::string GetUUID() const

Gets the UUID of this process.

GetUUID

Description:
Gets this process’s UUID.
Return
This process’s UUID

Private Functions

ModuleMessage Accept()

Creates an Accept Message.

GMAgent::Accept

Description:
Creates a new accept message from this node
Precondition:
This node is in a group.
Postcondition:
No change.
Return
A GroupManagementMessage with the contents of an Accept message

CPeerNode AddPeer(std::string uuid)

Adds a peer to the peer set from UUID.

GMAgent::AddPeer

Description:
Adds a peer to allpeers by uuid.
Precondition:
the UUID is registered in the connection manager
Postcondition:
A new peer object is created and inserted in CGlobalPeerList::instance().
Return
A pointer to the new peer
Parameters
  • uuid -

    of the peer to add.

CPeerNode AddPeer(CPeerNode peer)

Adds a peer from a pointer to a peer node object.

GMAgent::AddPeer

Description:
Adds a peer to all peers by pointer.
Precondition:
the pointer is valid
Postcondition:
The peer object is inserted in CGlobalPeerList::instance().
Return
A pointer that is the same as the input pointer
Parameters
  • peer -

    a pointer to a peer.

ModuleMessage AreYouCoordinator()

Creates AYC Message.

GMAgent::AreYouCoordinator

Description:
Creates a new Are You Coordinator message, from this object
Precondition:
The UUID is set
Postcondition:
No change
Return
A GroupManagementMessage with the contents of an Are You Coordinator Message.
Limitations:
: Can only author messages from this node.

ModuleMessage AreYouCoordinatorResponse(std::string payload, int seq)

Creates A Response message.

GMAgent::AreYouCoordinatorResponse

Description:
Creates a response message (Yes/No) message from this node
Precondition:
This node has a UUID.
Postcondition:
No change.
Return
A GroupManagementMessage with the contents of a Response message
Parameters
  • payload -

    Response message (typically yes or no)

  • seq -

    sequence number? (?)

ModuleMessage AreYouThere()

Creates a AYT, used for Timeout.

GMAgent::AreYouThere

Description:
Creates a new AreYouThere message from this node
Precondition:
This node is in a group.
Postcondition:
No Change.
Return
A GroupManagementMessage with the contents of an AreYouThere message

ModuleMessage AreYouThereResponse(std::string payload, int seq)

Creates A Response message.

GMAgent::AreYouThereResponse

Description:
Creates a response message (Yes/No) message from this node
Precondition:
This node has a UUID.
Postcondition:
No change.
Return
A GroupManagementMessage with the contents of a Response message
Parameters
  • payload -

    Response message (typically yes or no)

  • seq -

    sequence number? (?)

void Check(const boost::system::error_code & err)

Checks for other up leaders.

GMAgent::Check

Description:
This method queries all nodes to Check and see if any of them consider themselves to be Coordinators.
Precondition:
This node is in the normal state.
Postcondition:
Output of a system state and sent messages to all nodes to Check for Coordinators.
Citation:
GroupManagement (Check).
Parameters
  • err -

    Error associated with calling timer.

std::string Coordinator() const

Returns the coordinators uuid.

CPeerNode GetPeer(const std::string & uuid)

Gets a pointer to a peer from UUID.

GMAgent::GetPeer

Description:
Gets a peer from All Peers by uuid
Precondition:
None
Postcondition:
None
Return
A pointer to the requested peer, or a null pointer if the peer could not be found.
Parameters
  • uuid -

    The uuid of the peer to fetch

int GetStatus() const

Gets the status of a node.

Description:
returns the status stored in the node as an an integer. This means that in an inherited class, you may either define integer constants or an enumeration to generate status values.

void HandleAccept(const AcceptMessage & msg, CPeerNode peer)

Handles recieving accept messsages.

GMAgent::HandleAccept

Description:
Handles receiveing the invite accept
Key:
gm.Accept
Precondition:
In an election, and invites have been sent out.
Postcondition:
The sender is added to the tenative list of accepted peers for the group, if the accept messag is for the correct group identifier.
Interaction Peers:
Any node which has received an invite. (This can be any selection of the global peerlist, not just the ones this specific node sent the message to.)

void HandleAreYouCoordinator(const AreYouCoordinatorMessage & msg, CPeerNode peer)

Handles recieving are you coordinator messages.

GMAgent::HandleAreYouCoordinator

Description:
Handles receiveing the AYC message
Key:
gm.AreYouCoordinator
Precondition:
None
Postcondition:
The node responds yes or no to the request.
Interaction Peers:
Any node in the system is eligible to receive this at any time.

void HandleAreYouThere(const AreYouThereMessage & msg, CPeerNode peer)

Handles recieving are you there messsages.

GMAgent::HandleAreYouThere

Description:
Handles recieving the AYT message
Key:
gm.AreYouThere
Precondition:
None
Postcondition:
The node responds yes or no to the request.
Interaction Peers:
Any node in the system is eligible to receive this message at any time.

void HandleEcnMessage(const ecn::EcnMessage & msg, CPeerNode peer)

Handles incoming ECN message.

GMAgent::HandleEcnMessage

Description:
Processes ECN messages delivered to this module. If the message is of the “HARD” type, then the message indicates to the DGI that a group division should be performed to reduce the number of messages sent by the DGI and its algorithm. If the message is of the “SOFT” type, then the message indicates that the DGI should change its operating mode to reduce the amount of traffic it sends. DGI will update its next PeerList message to indicate that the other Processes in the group should change modes It will also set a timer to clear the flag indicating processes should change modes.
Precondition:
None
Postcondition:
The DGI changes modes based on the type of message received.
Return
none
Interaction Peers:
The ECN message originates from the router.
Parameters
  • msg -

    the details of the received ECN message peer Always the current process.

virtual void HandleIncomingMessage(boost::shared_ptr< const ModuleMessage > msg, CPeerNode peer)

Handles received messages.

“Downcasts” incoming messages into a specific message type, and passes the message to an appropriate handler.

Parameters
  • msg -

    the incoming message

  • peer -

    the node that sent this message (could be this DGI)

void HandleInvite(const InviteMessage & msg, CPeerNode peer)

Handles recieving invite messages.

GMAgent::HandleInvite

Description:
Handles recieving the invite
Key:
gm.Invite
Precondition:
The system is in the NORMAL state
Postcondition:
The system updates its group to accept the invite and switches to reorganization mode; it will wait for a timeout. If the Ready/Peerlist has not arrived, it will enter recovery. Otherwise, it will resume work as part of the group it was invited to.
Interaction Peers:
Any node could send an invite at any time.

void HandlePeerList(const PeerListMessage & msg, CPeerNode peer)

Hadles recieving peerlists.

GMAgent::HandlePeerList

Description:
Handles receiveing the peerlist.
Key:
any.PeerList
Precondition:
The node is in the reorganization or normal state
Postcondition:
The node’s peerlist is updated to match that of the incoming message if the message has come from his coordinator.
Interaction Peers:
Coordinator only.

void HandlePeerListQuery(const PeerListQueryMessage & msg, CPeerNode peer)

Handles recieving peerlist requests.

GMAgent::HandlePeerListQuery

Description:
Handles responding to peerlist queries.
Key:
gm.PeerListQuery
Precondition:
None
Postcondition:
Dispatched a message to the requester with a peerlist.
Interaction Peers:
Any. (Local or remote, doesn’t have to be a member of group)

void HandleResponseAYC(const AreYouCoordinatorResponseMessage & msg, CPeerNode peer)

Handles recieving AYC responses.

GMAgent::HandleResponseAYC

Description:
Handles recieving the Response
Key:
gm.Response.AreYouCoordinator
Precondition:
The timer for the AYC request batch the original request was a part of has not expired
Postcondition:
The response from the sender is noted, if yes, the groups will attempt to merge in the future.
Interaction Peers:
A node the AYC request was sent to.

void HandleResponseAYT(const AreYouThereResponseMessage & msg, CPeerNode peere)

Handles recieving AYT responses.

GMAgent::HandleResponseAYT

Description:
Handles recieving the Response
Key:
gm.Response.AreYouThere
Precondition:
The timer for the AYT request batch (typically 1) the original request was a part of has not expired
Postcondition:
The response from the sender is noted. If the answer is no, then this node will enter recovery, if the recovery timer has not already been set, otherwise we will allow it to expire.

ModuleMessage Invitation()

Creates Group Invitation Message.

GMAgent::Invitation

Description:
Creates a new invation message from the leader of this node’s current leader, to join this group.
Precondition:
The node is currently in a group.
Postcondition:
No change
Return
A GroupManagementMessage with the contents of a Invitation message.

void InviteGroupNodes(const boost::system::error_code & err, PeerSet p_tempSet)

Sends invitations to all group members.

GMAgent::InviteGroupNodes

Description:
This function will invite all the members of a node’s old group to join it’s new group.
Precondition:
None
Postcondition:
Invitations have been sent to members of this node’s old group. if this node is a Coordinator, this node sets a timer for Reorganize
Citation:
Group Management (Merge)
Parameters
  • err -

    The error message associated with the calling thimer

  • p_tempSet -

    The set of nodes that were members of the old group.

bool IsCoordinator() const

Returns true if this node considers itself a coordinator.

void Merge(const boost::system::error_code & err)

Sends invitations to all known nodes.

GMAgent::Merge

Description:
If this node is a Coordinator, this method sends invites to join this node’s group to all Coordinators, and then makes a call to second function to invite all current members of this node’s old group to the new group.
Precondition:
This node has waited for a Premerge.
Postcondition:
If this node was a Coordinator, this node has been placed in an election state. Additionally, invitations have been sent to all Coordinators this node is aware of. Lastly, this function has called a function or set a timer to invite its old group nodes.
Citation:
Group Management (Merge)
Parameters
  • err -

    A error associated with the calling timer.

ModuleMessage PeerList(std::string requester = "all")

Generates a peer list.

GMAgent::PeerList

Description:
Packs the group list (Up_Nodes) in GroupManagementMessage
Precondition:
This node is a leader.
Postcondition:
No Change.
Return
A GroupManagementMessage with the contents of group membership
Parameters
  • requester -

    The module to send the final message to

  • ps -

    The set of peers to put in th peerlist

  • coord -

    The leader of the

void Premerge(const boost::system::error_code & err)

Waits a time period determined by UUID for merge.

GMAgent::Premerge

Description:
Handles a proportional wait prior to calling Merge
Precondition:
Check has been called and responses have been collected from other nodes. This node is a Coordinator.
Postcondition:
A timer has been set based on this node’s UUID to break up ties before merging.
Citation:
GroupManagement (Merge)
Parameters
  • err -

    An error associated with the clling timer.

void PushPeerList()

Sends the peer list to all group members.

GMAgent::PushPeerList

Description:
Sends the membership list to other modules of this node and other nodes
Precondition:
This node is new group leader
Postcondition:
A peer list is pushed to the group members
Return
Nothing

void Recovery()

Resets the algorithm to the default startup state.

GMAgent::Recovery

Description:
The method used to set or reset a node into a “solo” state where it is its own leader. To do this, it forms an empty group, then enters a normal state.
Precondition:
None
Postcondition:
The node enters a NORMAL state.
Citation:
Group Management Algorithmn (Recovery).

void Recovery(const boost::system::error_code & err)

Handles no response from timeout message.

GMAgent::Recovery

Description:
Recovery function extension for handling timer expirations.
Precondition:
Some timer leading to this function has expired or been canceled.
Postcondition:
If the timer has expired, Recovery begins, if the timer was canceled and this node is NOT a Coordinator, this will cause a Timeout Check using Timeout()
Parameters
  • err -

    The error code associated with the calling timer.

void Reorganize(const boost::system::error_code & err)

Puts the system into the working state.

GMAgent::Reorganize

Description:
Organizes the members of the group and prepares them to do their much needed work!
Precondition:
The node is a leader of group.
Postcondition:
The group has received work assignments, ready messages have been sent out, and this node enters the NORMAL state.
Citation:
Group Management Reorganize

void SetStatus(int status)

Sets the status of the node.

Description:
Sets the internal status variable, m_status based on the paramter status. See GetStatus for tips on establishing the status code numbers.
Precondition:
None
Postcondition:
The modules status code has been set to “status”
Parameters
  • status -

    The status code to set

void StartMonitor(const boost::system::error_code & err)

Start the monitor after transient is over.

void SystemState()

Outputs information about the current state to the logger.

GMAgent::SystemState

Description:
Puts the system state to the logger.
Precondition:
None
Postcondition:
None

void Timeout(const boost::system::error_code & err)

Checks that the leader is still alive and working.

GMAgent::Timeout

Description:
Sends an AreYouThere message to the coordinator and sets a timer if the timer expires, this node goes into recovery.
Precondition:
The node is a member of a group, but not the leader
Postcondition:
A timer for recovery is set.
Citation:
Group Managment Timeout

Private Members

boost::posix_time::time_duration AYC_RESPONSE_TIMEOUT

How long to wait for responses from other nodes.

boost::posix_time::time_duration AYT_RESPONSE_TIMEOUT

How long to wait for responses from other nodes.

boost::posix_time::time_duration CHECK_TIMEOUT

How long between AYC checks.

boost::posix_time::time_duration FID_TIMEOUT

How long to wait before checking attached FIDs.

boost::posix_time::time_duration INVITE_RESPONSE_TIMEOUT

How long to wait for responses from other nodes.

TimedPeerSet m_AYCResponse

Nodes expecting AYC response from.

TimedPeerSet m_AYTResponse

Nodes expecting AYT response from.

PeerSet m_Coordinators

Known Coordinators.

std::map< std::string, bool > m_fidstate

A store for the state of attached FIDs.

CBroker::TimerHandle m_fidtimer

Timer for checking FIDs.

google::protobuf::uint32 m_GroupID

The ID number of the current group (Never initialized for fun)

std::string m_GroupLeader

The uuid of the group leader.

int m_groupsbroken

Number of groups broken.

int m_groupselection

Number of elections started.

int m_groupsformed

Number of groups formed.

int m_groupsjoined

Number of accepts sent.

unsigned int m_GrpCounter

The number of groups being formed.

boost::asio::io_service m_localservice

The io_service used.

int m_membership

Total size of groups after all checks.

int m_membershipchecks

Number of membership checks.

int m_status

A store for the status of this node.

CBroker::TimerHandle m_timer

A timer for stepping through the election process.

PeerSet m_UpNodes

Nodes In My Group.

boost::posix_time::time_duration TIMEOUT_TIMEOUT

How long beteween AYT checks.

Private Static Functions

ModuleMessage PeerListQuery(std::string requester)

Generates a CMessage that can be used to query for the group.

GMAgent::PeerListQuery

Description:
Generates a GroupManagementMessage that can be used to query the peerlist of a node.
Precondition:
: None
Postcondition:
: No change
Return
A GroupManagementMessage which can be used to query for
Parameters
  • requester -

    The module who the response should be addressed to.

ModuleMessage PrepareForSending(const GroupManagementMessage & message, std::string recipient = "gm")

Wraps a GroupManagementMessage in a ModuleMessage.

Wraps a GroupManagementMessage in a ModuleMessage.

Return
a ModuleMessage containing a copy of the GroupManagementMessage
Parameters
  • message -

    the message to prepare. If any required field is unset, the DGI will abort.

  • recipient -

    the module (sc/lb/gm/clk etc.) the message should be delivered to