Choosing a Replication Manager acknowledgement policy

Replication Manager allows the user to choose from a variety of acknowledgement policies. There are two characteristics that should be considered when choosing the policy: consistency and durability. Consistency means making sure some number of clients have applied all available master transactions. Durability, in this context, means only indicating success only if enough clients have applied a transaction. The issue of how many is enough depends on the application’s requirements and varies per acknowledgement policy. For example, DB_REPMGR_ACKS_QUORUM means the data will survive a change in master or a network partition. In most cases, the number of sites for consistency is equal to the number of sites for durability. Replication Manager uses the consistency value to decide whether or not to wait for acknowledgements. Replication manager uses the durability value to decide either the transaction was successfully processed or that a DB_EVENT_REP_PERM_FAILED event should be generated.

Replication Manager also strives to give the application the answer and return to the application as quickly as possible. Therefore, if it knows that the number of sites connected is insufficient to meet the consistency value, then it does not wait for any acknowledgements and if it knows that the durability value cannot be met, it returns DB_EVENT_REP_PERM_FAILED immediately to the user.

With one exception, discussed below, all acknowledgement policies combine the consistency and durability values. For most policies the primary purpose is the durability of the data. For example, the DB_REPMGR_ACKS_QUORUM policy ensures that, if successful, the transaction’s data is safe in the event of a network partition so that a majority of the sites in the group have the data. The DB_REPMGR_ACKS_NONE policy does not consider either consistency or durability, and it is very fast because it does not wait for any acknowledgements and it does not ever trigger the DB_EVENT_REP_PERM_FAILED event. Other policies, DB_REPMGR_ACKS_ALL and DB_REPMGR_ACKS_ALL_PEERS, have a primary purpose of consistency. These two policies wait for acknowledgements from all (or all electable) sites in the group.

In the face of failure, however, the DB_REPMGR_ACKS_ALL and DB_REPMGR_ACKS_ALL_PEERS policies can result in a surprising lack of consistency due to the fact that Replication Manager strives to give the answer back to the application as fast as it can. So, for example, with DB_REPMGR_ACKS_ALL, and one site down, Replication Manager knows that disconnected site can never acknowledge, so it immediately triggers DB_EVENT_REP_PERM_FAILED. An unfortunate side effect of this policy is that existing, running sites may fall further and further behind the master if the master site is sending a fast, busy stream of transactions and never waiting for any site to send an acknowledgement. The master does not wait because the consistency value cannot be met, and it does trigger the DB_EVENT_REP_PERM_FAILED event because the durability value cannot be met, but those actions now affect the consistency of the other running sites.

In order to counteract this unfortunate side effect, the DB_REPMGR_ACKS_ALL_AVAILABLE acknowledgement policy focuses on the consistency aspect, but also considers durability. This policy uses all sites for consistency, and a quorum of sites for its decision about durability. As long as there is a non-zero number of client replicas to send to, the master will wait for all available sites to acknowledge the transaction. As long as any client site is connected, this policy will prevent the master from racing ahead if one or more sites is down. On the master, this policy will then consider the transaction durable if the number of acknowledgements meets quorum for the group.

The following acknowledgement policies determine durability using acknowledgements from electable peers only: DB_REPMGR_ACKS_QUORUM, DB_REPMGR_ACKS_ONE_PEER, DB_REPMGR_ACKS_ALL_PEERS. An electable peer is a site where the priority value is greater than zero. In replication groups using these policies, an unelectable site does not send acknowledgements and cannot contribute to transaction durability.

If your application needs to associate a DB_EVENT_REP_PERM_FAILED event with the specific transaction and thread that caused it, this can be done by using your platform’s thread-specific data facility. You can create a thread-specific data structure that contains a durability indicator. You would set the durability indicator in the DB_EVENT_REP_PERM_FAILED portion of your event notification callback function. You can then check and clear the durability indicator after your transaction commit operation completes. You must be careful to coordinate this approach across all application threads performing transaction commits or checkpoints because any of these threads can generate a DB_EVENT_REP_PERM_FAILED event. The ex_rep_mgr sample application demonstrates one way to implement this capability. See Ex_rep: a replication example for more information.