Orca: Redis to SQL Migration

Migrate Orca from Redis to SQL, for resilience.

If you are not migrating an existing Orca deployment, refer to Orca SQL Setup instead.

Migrate from Redis to SQL

Migrating without downtime from Redis to SQL is a three-step process:

  1. Deploy Orca with the <code>DualExecutionRepository</code> writing to both Redis and SQL.
  2. Deploy a new Orca cluster with migrators enabled and queue processing disabled.
  3. Once all executions have been migrated, delete migration cluster and disable <code>DualExecutionRepository</code>.

When DualExecutionRepository is running, writes will be routed to either Redis or SQL. Executions will only be migrated to SQL once they’ve completed (either successfully or terminally): This keeps the migration story simple. As such, the migration agents will need to run for awhile. At Netflix, we ran the migration cluster for two weeks, as we had long pipeline executions due to canaries. You may only need to run the migration cluster for an hour.

NOTE: Deploying the migrators as a separate cluster is optional, however the migration process is memory hungry, so you may need to devote more resources to the Orca process.

Enable DualExecutionRepository

Building atop the baseline configuration above, add the following to orca.yml:

  1. executionRepository:
  2.   dual:
  3.     enabled: true
  4.     primaryName: sqlExecutionRepository
  5.     previousName: redisExecutionRepository
  6.   sql:
  7.     enabled: true
  8.   redis:
  9.     enabled: true

Note that both repositories are enabled. Orca will fail to start up if the DualExecutionRepository is misconfigured.

Deploy a Migration Cluster

At Netflix, we deployed orca-main, the cluster that serves our production traffic, as well as orca-main-sqlmigration, which does not receive API traffic nor process the work queue. Its sole purpose is to shovel bits from Redis to SQL.

To perform a deploy a migration cluster, add the following configuration to orca.yml:

  1. ---
  2. spring:
  3.   profiles: sqlmigration
  5. pollers:
  6.   orchestrationMigrator:
  7.     enabled: true
  8.     intervalMs: 1800000
  9.   pipelineMigrator:
  10.     enabled: true
  11.     intervalMs: 1800000
  13. queue:
  14.   redis:
  15.     enabled: false
  17. keiko:
  18.   queue:
  19.     enabled: false

You will need to launch this migration cluster with -Dspring.profiles.active=sqlmigration. Spring Profiles allow you to configure a service to startup with different configurations.

Disable DualExecutionRepository

Once all executions have been migrated, you can deploy Orca once again without DualExecutionRepository and delete the migration cluster.

SQL-specific Metrics

Orca emits a few handfuls of metrics that are specific to SQL.

The SqlHealthcheckQueueActivator class, which will disable work queue processing if SQL connectivity goes unhealthy, emits one metric: sql.queueActivator.invocations with a tag of status (disabled or enabled).

The ExecutionRepository will emit a bunch of invocation and timing metrics with the following patterns:

  • sql.executionRepository.$method.timing
  • sql.executionRepository.$method.invocations

If you are using the default HikariCP connection pool:

  • sql.pool.$poolName.connectionAcquiredTiming
  • sql.pool.$poolName.connectionUsageTiming
  • sql.pool.$poolName.connectionTimeout
  • sql.pool.$poolName.idle
  • sql.pool.$poolName.active
  • sql.pool.$poolName.total
  • sql.pool.$poolName.blocked

If you are using the MariaDB driver extension:

  • sql.pool.$poolName.active
  • sql.pool.$poolName.idle
  • sql.pool.$poolName.total
  • sql.pool.$poolName.blocked

Last modified May 4, 2021: rest of migration (700781a)