Faasm Collective Communication

To support efficient parallel processing, Faasm exposes shared memory regions between functionscolocated on the same host. These in-memory regions are referred to collectively as the local tier.Faasm synchronises these regions across hosts through the global tier, which is independent of anygiven host, and holds a single authoritative copy of the underlying data. Data held in the global tiercan outlive any given function execution, and is useful for holding larger shared datasets which may beused as part of a composition of several functions.

This two-tier architecture is flexible and works well for a range of applications. Synchronisationbetween the local tier and the global tier is crucial in enabling this flexibility, but doesintroduce latency and network overheads. Certain classes of applications have well defined patternsfor sharing data, and don't require this much flexibility. Accordingly, we can introduce optimisationsin Faasm's state management specifically for such applications, hence avoiding unnecessary overheads incertain cases.

Collective communication is one such class of application. It involves several well-defined operationssuch as broadcast, reduce and gather, all of which have fixed modes of sharing immutable data between anumber of distributed functions. When these operations are performed, the Faasm runtime knows exactly whichfunctions need to access which chunks of data, and can optimise their placement accordingly.

To do this, Faasm uses point-to-point replication. This is a special type of state synchronisationthat occurs only through the local tier of each host, and does not involve the global tier.

Point-to-point replication

Whenever Faasm functions access shared state, they do so through their local tier, i.e. each host has asingle replica of the data held in memory which is shared between all functions on that host. In acollective communication operation, Faasm knows exactly which functions need to access which pieces of data,hence can deduce which hosts need replicas in their local tier. To ensure minimal replication, the hostinitiating the operation can send one or more point-to-point messages, transmitting the required data tothe target hosts. Hosts receiving such messages will copy this data into their local tier, thus making itaccessible to every function on that host.

If many functions on a given host need to access the same data, only one point-to-point replication is requiredto ensure they all have access.

Collective operations

Collective operations can be categorised by the number of messages that must be sent, the level of datasharing, and hence how many replications they require. If we assume a naive implementation of eachoperation, with a total of n_funcs involved across n_hosts, we get the following table:

TODO - finish these notes