jactorch.parallel.comm#

Classes

`SlavePipe`	Pipe for master-slave communication.
`SyncMaster`	An abstract SyncMaster object.

Class SlavePipe

class SlavePipe[source]#

Bases: _SlavePipeBase

Pipe for master-slave communication.

__add__(value, /)#: Return self+value.

__init__()#

__mul__(value, /)#: Return self*value.

static __new__(_cls, identifier, queue, result)#: Create new instance of _SlavePipeBase(identifier, queue, result)

count(value, /)#: Return number of occurrences of value.

index(value, start=0, stop=9223372036854775807, /)#

Return first index of value.

Raises ValueError if the value is not present.

run_slave(msg)[source]#

Parameters:: msg (Any)

identifier#: Alias for field number 0

queue#: Alias for field number 1

result#: Alias for field number 2

Class SyncMaster

class SyncMaster[source]#

Bases: object

An abstract SyncMaster object.

During the replication, as the data parallel will trigger an callback of each module, all slave devices should call register(id) and obtain an SlavePipe to communicate with the master.
During the forward pass, master device invokes run_master, all messages from slave devices will be collected, and passed to a registered callback.
After receiving the messages, the master device should gather the information and determine to message passed back to each slave devices.

__init__(master_callback)[source]#

Initialize the master.

Parameters:: master_callback (Callable) – a callback to be invoked after having collected messages from slave devices.

__new__(**kwargs)#

register_slave(identifier)[source]#

Parameters:: identifier – an identifier, usually is the device id.
Returns:: a SlavePipe object which can be used to communicate with the master device.

run_master(master_msg)[source]#

Main entry for the master device in each forward pass. The messages were first collected from each devices (including the master device), and then an callback will be invoked to compute the message to be sent back to each devices (including the master device).

Parameters:

master_msg (Any) – the message that the master want to send to itself. This will be placed as the first
usage (message when calling master_callback. For detailed)
example. (see _SynchronizedBatchNorm for an)

Returns:

the message to be sent back to the master device.

property nr_slaves: int#: The number of slave devices.