Running replication using the db_replicate utility

Replication Manager supports shared access to a database environment from multiple processes. Berkeley DB provides a replication-aware utility, db_replicate, that enables you to upgrade an existing Transactional Data Store application, as discussed in the Transactional Data Store introduction section, to a replication application with minor modifications. While the db_replicate utility simplifies the use of replication with a Transactional Data Store application, you must still understand replication and its impact on the application.

One replication process and multiple subordinate processes

Based on the terminology introduced in the Running Replication Manager in multiple processes section, application processes are subordinate processes and the db_replicate utility is the replication process.

You must consider the following items when planning to use the db_replicate utility in combination with a Transactional Data Store application.

• Memory regions

  The db_replicate utility requires shared memory access among separate processes, and therefore cannot be used with DB_PRIVATE.

• Multi-process implications

  You must understand and accept all of the Transactional Data Store implications of multi-process use as specified in Architecting Transactional Data Store applications. Special attention should be paid to the coordination needed for unrelated processes to start up correctly.

• Replication configuration

  Several configuration settings are required for replication. You must set the DB_INIT_REP and DB_THREAD flags for the DB_ENV->open() method. Another required configuration item is the local address. You identify this by creating a DB_SITE handle and then setting the DB_LOCAL_SITE parameter using the DB_SITE->set_config() method. You also tell sites how to contact other sites by creating DB_SITE handles for those sites. Most replication configuration options start with reasonable defaults, but applications have to customize at least some of them. You can set all replication related configuration options either programmatically or in the DB_CONFIG file.

• Starting the application and replication

  The db_replicate utility assumes that an environment exists and that the application has run recovery, if necessary, and created and configured the environment. The startup flow of a typical Transactional Data Store application may not be the best flow for a replication application and you must understand the issues involved. For instance, if an application starts, runs recovery, and performs update operations before starting the db_replicate utility, then if that site becomes a client when replication starts, those update operations will be rolled back.

• Handling events

  Almost all of the replication-specific events are handled by the db_replicate utility process, and therefore the application process does not see them. If the application needs to know the information from those replication-specific events, such as role changes, the application must call the rep_stat() method method. The one replication-specific event the application can now receive is the DB_EVENT_REP_PERM_FAILED event. See Choosing a Replication Manager acknowledgement policy for additional information about this event.

• Handling errors

  There are some error return values that relate only to replication. Specifically, the DB_REP_HANDLE_DEAD error should now be handled by the application. Also, if master leases are in use, then the application also needs to consider the DB_REP_LEASE_EXPIRED error.

• Flexibility tradeoff

  You are giving up flexibility for the ease of use of the utility. Application complexity or requirements may eventually dictate integrating replication calls into the application instead of using the db_replicate utility.

• Read-only client application

  The application requires additional changes to manage the read-only status when the application takes on the role of a client.

Common use case

This section lists the steps needed to get replication running for a common use case of the db_replicate utility. The use case presented is an existing Transactional Data Store application that already has its environment and databases created and is up and running. At some point, replication is considered because failover protection or balancing the read load may now be desired.

1. To understand the issues involved in a replication application, see the db_replicate utility section in the API Reference Guide, the Replication Chapter in the Programmer’s Reference Guide, and the source code of the ex_rep_mgr example program.

2. Make a local hot backup of the current application environment to a new location to use as a testing area.

3. Add the DB_INIT_REP and DB_THREAD flags (if not already being used) to the application or the DB_CONFIG file.

4. Modify the DB_CONFIG file to add the necessary replication configuration values. At a minimum, the local host and port information must be added using the repmgr_site method parameter. As more sites are added to the group, remote host and port information can optionally also be added by adding more repmgr_site method parameters to the DB_CONFIG file.

5. Rebuild the application and restart it in the current testing directory.

6. Start the db_replicate utility on the master site with the -M option and any other options needed such as -h for the home directory. At this point you have a lone master site running in an environment with no other replicated sites in the group.

7. Optionally, prepare to start a client site by performing a manual hot backup of the running master environment to initialize a client target directory. While replication can make its own copy, the hot backup will expedite the synchronization process. Also, if the application assumes the existence of a database and the client site is started without data, the application may have errors or incorrectly attempt to create the database.

8. Copy the application to the client target.

9. Modify the client environment’s DB_CONFIG file to set the client’s local host and port values and to add remote site information for the master site and any other replication configuration choices necessary.

10. Start the application on the client. The client application should not update data at this point, as explained previously.

11. Start the db_replicate utility specifying the client environment’s home directory using the -h option. Omit the -M option in this case, because the utility defaults to starting in the client role.

Once the initial replication group is established, do not use the -M option with the db_replicate utility. After the initial start, the db_replicate utility assumes the use of elections. If a site crashes, it should rejoin the group as a client so that it can synchronize with the rest of the group.

Avoiding rollback

Depending on how an application is structured, transactional rollback can occur. If this is possible, then you must make application changes or be prepared for successful transactions to disappear. Consider a common program flow where the application first creates and opens the environment with recovery. Then, immediately after that, the application opens up the databases it expects to use. Often an application will use the DB_CREATE flag so that if the database does not exist it is created, otherwise the existing one is used automatically. Then the application begins servicing transactions to write and read data.

When replication is introduced, particularly via the db_replicate utility, the possibility of rollback exists unless the application takes steps to prevent it. In the situation described above, if all of the above steps occur before the db_replicate utility process starts, and the site is started as a client, then all the operations will be rolled back when the site finds the master. The client site will synchronize with the log and operations on the master site, so any operations that occurred in the client application before it knew it was a client will be discarded.

One way to reduce the possibility of rollback is to modify the application so that it only performs update operations (including creation of a database) if it is the master site. If the application refrains from updating until it is the master, then it will not perform operations when it is in the undefined state before replication has been started. The event indicating a site is master will be delivered to the db_replicate utility process, so the application process must look for that information via the rep_stat() method. A site that is expecting to perform updates may need to poll via the rep_stat() method to see the state change from an undefined role to either the master or client role. Similarly, since a client site cannot create a database, it may need to poll for the database’s existence while the client synchronizes with the master until the database is created at the client site.

When to consider an integrated replication application

The db_replicate utility provides the means to achieve a replicated application quickly. However, the trade-off for this rapid implementation is that the full flexibility of replication is not available. Some applications may eventually need to consider integrating directly with replication rather than using the db_replicate utility if greater flexibility is desired.

One likely reason for considering integration would be the convenience of receiving all replication-related events in the application process and gaining direct knowledge of such things as role changes. Using the event callback is cleaner and easier than polling for state changes via the rep_stat() method.

A second likely reason for integrating replication directly into the application is the multi-process aspect of the utility program. The developer may find it easier to insert the start of replication directly into the code once the environment is created, recovered, or opened, and avoid the scenario where the application is running in the undefined state. Also it may simply be easier to start the application once than to coordinate different processes and their startup order in the system.