Build

Background

For systems running on a single database that urgently need to securely and simply migrate data to a horizontally sharded database.

Prerequisites

  • Proxy is developed in JAVA, and JDK version 1.8 or later is recommended.
  • Data migration adopts the cluster mode, and ZooKeeper is currently supported as the registry.

Procedure

  1. Run the following command to compile the ShardingSphere-Proxy binary package:
  1. git clone --depth 1 https://github.com/apache/shardingsphere.git
  2. cd shardingsphere
  3. mvn clean install -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Drat.skip=true -Djacoco.skip=true -DskipITs -DskipTests -Prelease

Release package：

  • /shardingsphere-distribution/shardingsphere-proxy-distribution/target/apache-shardingsphere-${latest.release.version}-shardingsphere-proxy-bin.tar.gz

Or you can get the installation package through the Download Page

  1. Decompress the proxy release package and modify the configuration file conf/config-sharding.yaml. Please refer to proxy startup guide for details.

  2. Modify the configuration file conf/server.yaml. Please refer to mode configuration for details.

Currently, mode must be Cluster, and the corresponding registry must be started in advance.

Configuration sample:

  1. mode:
  2.   type: Cluster
  3.   repository:
  4.     type: ZooKeeper
  5.     props:
  6.       namespace: governance_ds
  7.       server-lists: localhost:2181
  8.       retryIntervalMilliseconds: 500
  9.       timeToLiveSeconds: 60
  10.       maxRetries: 3
  11.       operationTimeoutMilliseconds: 500
  1. Introduce JDBC driver.

Proxy has included JDBC driver of PostgreSQL.

If the backend is connected to the following databases, download the corresponding JDBC driver jar package and put it into the ${shardingsphere-proxy}/ext-lib directory.

DatabaseJDBC DriverReference
MySQLmysql-connector-java-5.1.47.jarConnector/J Versions
openGaussopengauss-jdbc-3.0.0.jar

If you are migrating to a heterogeneous database, then you could use more types of database, e.g. Oracle. Introduce JDBC driver as above too.

  1. Start ShardingSphere-Proxy:
  1. sh bin/start.sh
  1. View the proxy log logs/stdout.log. If you see the following statements:
  1. [INFO ] [main] o.a.s.p.frontend.ShardingSphereProxy - ShardingSphere-Proxy start success

The startup will have been successful.

  1. Configure and migrate on demand.

7.1. Query configuration.

  1. SHOW MIGRATION PROCESS CONFIGURATION;

The default configuration is as follows.

  1. +--------------------------------------------------------------+--------------------------------------+------------------------------------------------------+
  2. | read                                                         | write                                | stream_channel                                       |
  3. +--------------------------------------------------------------+--------------------------------------+------------------------------------------------------+
  4. | {"workerThread":40,"batchSize":1000,"shardingSize":10000000} | {"workerThread":40,"batchSize":1000} | {"type":"MEMORY","props":{"block-queue-size":10000}} |
  5. +--------------------------------------------------------------+--------------------------------------+------------------------------------------------------+

7.2. New configuration (Optional).

A default value is available if there is no configuration.

A completely configured DistSQL is as follows.

  1. CREATE MIGRATION PROCESS CONFIGURATION (
  2. READ(
  3.   WORKER_THREAD=40,
  4.   BATCH_SIZE=1000,
  5.   SHARDING_SIZE=10000000,
  6.   RATE_LIMITER (TYPE(NAME='QPS',PROPERTIES('qps'='500')))
  7. ),
  8. WRITE(
  9.   WORKER_THREAD=40,
  10.   BATCH_SIZE=1000,
  11.   RATE_LIMITER (TYPE(NAME='TPS',PROPERTIES('tps'='2000')))
  12. ),
  13. STREAM_CHANNEL (TYPE(NAME='MEMORY',PROPERTIES('block-queue-size'='10000')))
  14. );

Configuration item description:

  1. CREATE MIGRATION PROCESS CONFIGURATION (
  2. READ( -- Data reading configuration. If it is not configured, part of the parameters will take effect by default.
  3.   WORKER_THREAD=40, -- Obtain the thread pool size of all the data from the source side. If it is not configured, the default value is used.
  4.   BATCH_SIZE=1000, -- The maximum number of records returned by a query operation. If it is not configured, the default value is used.
  5.   SHARDING_SIZE=10000000, -- Sharding size of all the data. If it is not configured, the default value is used.
  6.   RATE_LIMITER ( -- Traffic limit algorithm. If it is not configured, traffic is not limited.
  7.   TYPE( -- Algorithm type. Option: QPS
  8.   NAME='QPS',
  9.   PROPERTIES( -- Algorithm property
  10.   'qps'='500'
  11.   )))
  12. ),
  13. WRITE( -- Data writing configuration. If it is not configured, part of the parameters will take effect by default.
  14.   WORKER_THREAD=40, -- The size of the thread pool on which data is written into the target side. If it is not configured, the default value is used.
  15.   BATCH_SIZE=1000, -- The maximum number of records for a batch write operation. If it is not configured, the default value is used.
  16.   RATE_LIMITER ( -- Traffic limit algorithm. If it is not configured, traffic is not limited.
  17.   TYPE( -- Algorithm type. Option: TPS
  18.   NAME='TPS',
  19.   PROPERTIES( -- Algorithm property.
  20.   'tps'='2000'
  21.   )))
  22. ),
  23. STREAM_CHANNEL ( -- Data channel. It connects producers and consumers, used for reading and writing procedures. If it is not configured, the MEMORY type is used by default.
  24. TYPE( -- Algorithm type. Option: MEMORY
  25. NAME='MEMORY',
  26. PROPERTIES( -- Algorithm property
  27. 'block-queue-size'='10000' -- Property: blocking queue size.
  28. )))
  29. );

DistSQL sample: configure READ for traffic limit.

  1. CREATE MIGRATION PROCESS CONFIGURATION (
  2. READ(
  3.   RATE_LIMITER (TYPE(NAME='QPS',PROPERTIES('qps'='500')))
  4. )
  5. );

Configure data reading for traffic limit. Other configurations use default values.

7.3. Modify configuration.

ALTER MIGRATION PROCESS CONFIGURATION, and its internal structure is the same as that of CREATE MIGRATION PROCESS CONFIGURATION.

DistSQL sample: modify traffic limit parameter

  1. ALTER MIGRATION PROCESS CONFIGURATION (
  2. READ(
  3.   RATE_LIMITER (TYPE(NAME='QPS',PROPERTIES('qps'='1000')))
  4. )
  5. );
  6. ---
  7. ALTER MIGRATION PROCESS CONFIGURATION (
  8. READ(
  9.   RATE_LIMITER (TYPE(NAME='QPS',PROPERTIES('qps'='1000')))
  10. ), WRITE(
  11.   RATE_LIMITER (TYPE(NAME='TPS',PROPERTIES('tps'='1000')))
  12. )
  13. );

7.4. Clear configuration.

DistSQL sample: clear the configuration of READ and restore it to the default value.

  1. DROP MIGRATION PROCESS CONFIGURATION '/READ';

DistSQL sample: clear the configuration of READ/RATE_LIMITER.

  1. DROP MIGRATION PROCESS CONFIGURATION '/READ/RATE_LIMITER';