CBroker Reference

CBroker is a part of CBroker.hpp

class

Scheduler for the DGI modules.

Public Functions

~CBroker()

De-allocates the timers when the CBroker is destroyed.

Description:
Cleans up all the timers from this module since the timers are stored as pointers.
Precondition:
None
Postcondition:
All the timers are destroyed and their handles no longer point at valid resources.

TimerHandle AllocateTimer(ModuleIdent module)

Allocate a timer to a specified module.

Description:
Returns a handle to a timer to use for scheduling tasks. Timer handles are used in Schedule() calls.
Precondition:
The module is registered
Postcondition:
A handle to a timer is returned.
Return
A CBroker::TimerHandle that can be used to schedule tasks.
Parameters
  • module -

    the module the timer should be allocated to

CClockSynchronizer & GetClockSynchronizer()

Returns the synchronizer.

Description:
Returns a reference to the ClockSynchronizer object.
Precondition:
None
Postcondition:
Any changes to the ClockSynchronizer will affect the object owned by the broker.
Return
A reference to the Broker’s ClockSynchronizer object.

boost::asio::io_service & GetIOService()

Return a reference to the boost::ioservice.

Description:
returns a refernce to the ioservice used by the broker.
Return
The ioservice used by this broker.

void HandleSignal(const boost::system::error_code & error, int parameter)

Handle signals from the operating system (ie, Control-C)

Description:
Handle signals that terminate the program.
Precondition:
None
Postcondition:
The broker is scheduled to be stopped.

void HandleStop(unsigned int signum = 0)

Handles the stop signal from the operating System.

Description:
Handles closing all the sockets connection managers and Services. Should probably only be called by CBroker::Stop().
Precondition:
The ioservice is running but all agents have been stopped.
Postcondition:
The Broker has been cleanly shut down.
Postcondition:
The devices subsystem has been cleanly shut down.
Parameters
  • signum -

    positive if called from a signal handler, or 0 otherwise

bool IsModuleRegistered(ModuleIdent m)

Checks to see if a module is registered with the scheduler.

Description:
Checks to see if a module is registered with the scheduler.
Precondition:
None
Postcondition:
None
Return
true if the module is registered with the broker.
Parameters
  • m -

    the identifier for the module.

void RegisterModule(ModuleIdent m, boost::posix_time::time_duration phase)

Registers a module for the scheduler.

Description:
Places the module in to the list of schedulable phases. The scheduler cycles through registered modules to do real-time round robin scheduling.
Precondition:
None
Postcondition:
The module is registered with a phase duration specified by the phase parameter.
Parameters
  • m -

    the identifier for the module.

  • phase -

    the duration of the phase.

void Run()

Starts the DGI Broker scheduler.

Description:
Starts the adapter factory. Runs the ioservice until it is out of work. Runs the clock synchronizer.
Precondition:
The ioservice has some schedule of jobs waiting to be performed (so it doesn’t exit immediately).
Postcondition:
The ioservice has stopped.
Error Handling:
Could raise arbitrary exceptions from anywhere in the DGI.

int Schedule(TimerHandle h, boost::posix_time::time_duration wait, Scheduleable x)

Schedules a task that will run after a timer expires.

Description:
Given a scheduleable task that should be run in the future. The task will be scheduled to run by the Broker after the timer expires and during the module that owns the timer’s phase. The attempt to schedule may be rejected if the Broker is stopping, indicated by the return value.
Precondition:
The module is registered
Postcondition:
If the Broker is not stopping, the function is scheduled to be called in the future. If a next time function is scheduled, its timer will expire as soon as its round ends. If the Broker is stopping the task will not be scheduled.
Return
0 on success, -1 if rejected
Parameters
  • h -

    The handle to the timer being set.

  • wait -

    the amount of the time to wait. If this value is “not_a_date_time” The wait is converted to positive infinity and the time will expire as soon as it is not the module that owns the timer’s phase.

  • x -

    A schedulable, a functor, that expects a single boost::system::error_code parameter and returns void, created via boost::bind()

int Schedule(ModuleIdent m, BoundScheduleable x, bool start_worker = true)

Schedule a task to be run as soon as the module is active.

Description:
Given a module and a task, put that task into that modules job queue. The attempt to schedule will be rejected if the Broker is stopping.
Precondition:
The module is registered.
Postcondition:
The task is placed in the work queue for the module m. If the start_worker parameter is set to true, the module’s worker will be activated if it isn’t already.
Return
0 on success, -1 if rejected because the Broker is stopping.
Parameters
  • m -

    The module the schedulable should be run as.

  • x -

    The method that will be run. A functor that expects no parameters and returns void. Created via boost::bind()

  • start_worker -

    tells the worker to begin processing again, if it is currently idle. The worker may be idle if the work queue is currently empty

void Stop(unsigned int signum = 0)

Requests that the Broker stops execution to exit the DGI.

Description:
Registers a stop command into the io_service’s job queue. when scheduled, the stop operation will terminate all running modules and cause the ioservice.run() call to exit.
Precondition:
The ioservice is running and processing tasks.
Postcondition:
The command to stop the ioservice has been placed in the service’s task queue.
Parameters
  • signum -

    A signal identifier if the call came from a signal, or 0 otherwise

boost::posix_time::time_duration TimeRemaining()

Returns how much time the current module has left in its phase.

Description:
Returns how much time is remaining in the active module’s phase. The result can be negative if the module has exceeded its allotted execution time.
Precondition:
The Change Phase function has been called at least once. This should have occured by the time the first module is ready to look at the remaining time.
Postcondition:
None
Return
A time_duration describing the amount of time remaining in the phase.

Public Static Functions

CBroker & Instance()

Get the singleton instance of this class.

Access the singleton Broker instance.

See Also

It may be useful to start with Scheduling DGI Modules.

The Scheduler

The non-RT DGI code made heavy use of the boost provided io_service. This library allows for easy asynchronous calls to methods and very easy socket handling. Tasks are inserted into the io_services run list and executed in FIFO order. The RT scheduler takes advantage of the io_service, while trying to apply constraints on the ordering of execution.

Review: Non-RT Processing

To review, the modules and message processing (which are the two major components of the system) relied on a structure very similar to this:

../_images/non-realtime.png
  • Sleeps and Thread Relinquishment - Since most of the processing for the FREEDM system currently runs on the single io_service thread used by the broker, multi-threading is emulated through the use of the io_service’s async wait routines which allow a module to sleep while other modules (or the broker itself) continues doing processing on a single thread. When a task requires a break for I/O or other similar services, it will call async_wait and wait for the io_service to execute the callback it requests. There are many examples of this in the codebase.
  • Message Processing - When a message is received for processing, it will be taken into the CListener module where it is processed to determine if it should be accepted or rejected. If accepted, it is passed to the dispatcher which examines the contents to determine which modules if any should be given the message. The module responsible for receiving the message is immediately called and allowed to act on the message.

Real-time Processing

To create a real-time scheduler, I’ve added a layer between the modules and the io_service in the broker. This layer defers the expired async_waits until it is a modules “phase.” To do this, the broker now distributes timers (or more specifically, handles to the timers) and provides a function “Schedule” which is used instead of async_wait. The result is a scheduling system like this:

../_images/realtime.png
  • Sleeps and Thread Relinquishment - The behavior remains similar to how it did before, except the async_wait call is replaced by a schedule call. When the timer expires, it calls the ScheduledTask method, rather than directly calling the scheduled task. This method enters the readied task into a per-module “ready queue.” Once the scheduler enters that modules phase (or it is already in it) a worker method will process the ready queue in FIFO order, calling the scheduled methods.
  • Message Processing - This also remains similar, except now instead of the dispatcher directly calling the module who should receive the incoming message the call to the module is placed in that module’s ready queue. Then, as before, when the scheduler enters that module’s phase, a worker method handles the actual call to the receive method.

Phase Behavior

../_images/example.png

Phases proceed in a round robin fashion. However, there are some observations to make a for this implementation. Consider the diagram above.

  • A The group management phase begins. It spends some time authoring messages for the check function, requests them to be sent and then idles. Send, which is handled by the broker operates outside of the module scheduler and begins work immediately.
  • B Since group management has no work to do while waiting for replies, the system is idle.
  • C Message(s) arrive. Since one of them is addressed to group management and it is currently GroupManagement’s phase, the worker immediately fulfills the request.
  • D Processing the message makes GroupManagement over-run its phase.
  • E There is a “No-man’s” land where the group management task is completing work outside of its phase, but the scheduler is ready to switch phases to state collection. Because phases are aligned in order to form groups, state collection is penalized for by the overflow.
  • F The scheduler has changed phases, but the broker does work on a received message before calling state collection’s readied method. Note that the message is put into the ready queue for its intended process, so the message processing time is only spent by the Broker and communication stack
  • G Finally, the state collection module begins.