YAML configuration

5.0.0-alpha

Data Sharding

Configuration Item Explanation

  1. dataSources: # Omit the data source configuration, please refer to the usage
  3. rules:
  4. - !SHARDING
  5.   tables: # Sharding table configuration
  6.     <logic-table-name> (+): # Logic table name
  7.       actualDataNodes (?): # Describe data source names and actual tables (refer to Inline syntax rules)
  8.       databaseStrategy (?): # Databases sharding strategy, use default databases sharding strategy if absent. sharding strategy below can choose only one.
  9.         standard: # For single sharding column scenario
  10.           shardingColumn: # Sharding column name
  11.           shardingAlgorithmName: # Sharding algorithm name
  12.         complex: # For multiple sharding columns scenario
  13.           shardingColumns: # Sharding column names, multiple columns separated with comma
  14.           shardingAlgorithmName: # Sharding algorithm name
  15.         hint: # Sharding by hint
  16.           shardingAlgorithmName: # Sharding algorithm name
  17.         none: # Do not sharding
  18.       tableStrategy: # Tables sharding strategy, same as database sharding strategy
  19.       keyGenerateStrategy: # Key generator strategy
  20.         column: # Column name of key generator
  21.         keyGeneratorName: # Key generator name
  22.   autoTables: # Auto Sharding table configuration
  23.     t_order_auto: # Logic table name
  24.       actualDataSources (?): # Data source names
  25.       shardingStrategy: # Sharding strategy
  26.         standard: # For single sharding column scenario
  27.           shardingColumn: # Sharding column name
  28.           shardingAlgorithmName: # Auto sharding algorithm name
  29.   bindingTables (+): # Binding tables
  30.     - <logic_table_name_1, logic_table_name_2, ...> 
  31.     - <logic_table_name_1, logic_table_name_2, ...> 
  32.   broadcastTables (+): # Broadcast tables
  33.     - <table-name>
  34.     - <table-name>
  35.   defaultDatabaseStrategy: # Default strategy for database sharding
  36.   defaultTableStrategy: # Default strategy for table sharding
  37.   defaultKeyGenerateStrategy: # Default Key generator strategy
  39.   # Sharding algorithm configuration
  40.   shardingAlgorithms:
  41.     <sharding-algorithm-name> (+): # Sharding algorithm name
  42.       type: # Sharding algorithm type
  43.       props: # Sharding algorithm properties
  44.       # ...
  46.   # Key generate algorithm configuration
  47.   keyGenerators:
  48.     <key-generate-algorithm-name> (+): # Key generate algorithm name
  49.       type: # Key generate algorithm type
  50.       props: # Key generate algorithm properties
  51.       # ...
  53. props:
  54.   # ...

Replica Query

Configuration Item Explanation

  1. dataSources: # Omit the data source configuration, please refer to the usage
  3. rules:
  4. - !REPLICA_QUERY
  5.   dataSources:
  6.     <data-source-name> (+): # Logic data source name of replica query
  7.       primaryDataSourceName: # Primary data source name
  8.       replicaDataSourceNames: 
  9.         - <replica-data_source-name> (+) # Replica data source name
  10.       loadBalancerName: # Load balance algorithm name
  12.   # Load balance algorithm configuration
  13.   loadBalancers:
  14.     <load-balancer-name> (+): # Load balance algorithm name
  15.       type: # Load balance algorithm type
  16.       props: # Load balance algorithm properties
  17.         # ...
  19. props:
  20.   # ...

Please refer to Built-in Load Balance Algorithm List for more details about type of algorithm.

Encryption

Configuration Item Explanation

  1. dataSource: # Omit the data source configuration, please refer to the usage
  3. rules:
  4. - !ENCRYPT
  5.   tables:
  6.     <table-name> (+): # Encrypt table name
  7.       columns:
  8.         <column-name> (+): # Encrypt logic column name
  9.           cipherColumn: # Cipher column name
  10.           assistedQueryColumn (?):  # Assisted query column name
  11.           plainColumn (?): # Plain column name
  12.           encryptorName: # Encrypt algorithm name
  14.   # Encrypt algorithm configuration
  15.   encryptors:
  16.     <encrypt-algorithm-name> (+): # Encrypt algorithm name
  17.       type: # Encrypt algorithm type
  18.       props: # Encrypt algorithm properties
  19.         # ...
  21.   queryWithCipherColumn: # Whether query with cipher column for data encrypt. User you can use plaintext to query if have

Please refer to Built-in Encrypt Algorithm List for more details about type of algorithm.

Shadow DB

Configuration Item Explanation

  1. dataSources: # Omit the data source configuration, please refer to the usage
  3. rules:
  4. - !SHADOW
  5.   column: # Shadow column name
  6.   sourceDataSourceNames: # Source Data Source names
  7.      # ...
  8.   shadowDataSourceNames: # Shadow Data Source names
  9.      # ... 
  11. props:
  12.   # ...

Governance

Configuration Item Explanation

  1. governance:
  2.   name: # Governance name
  3.   registryCenter: # Registry center
  4.     type: # Governance instance type. Example:Zookeeper, etcd
  5.     serverLists: # The list of servers that connect to governance instance, including IP and port number; use commas to separate 
  6.   overwrite: # Whether to overwrite local configurations with config center configurations; if it can, each initialization should refer to local configurations

ShardingSphere-4.x

Data Sharding

Configuration Item Explanation

  1. dataSources:
  2.   ds0: !!org.apache.commons.dbcp.BasicDataSource
  3.     driverClassName: com.mysql.jdbc.Driver
  4.     url: jdbc:mysql://localhost:3306/ds0
  5.     username: root
  6.     password: 
  7.   ds1: !!org.apache.commons.dbcp.BasicDataSource
  8.     driverClassName: com.mysql.jdbc.Driver
  9.     url: jdbc:mysql://localhost:3306/ds1
  10.     username: root
  11.     password: 
  13. shardingRule:  
  14.   tables:
  15.     t_order: 
  16.       actualDataNodes: ds${0..1}.t_order${0..1}
  17.       databaseStrategy:
  18.         inline:
  19.           shardingColumn: user_id
  20.           algorithmExpression: ds${user_id % 2}
  21.       tableStrategy: 
  22.         inline:
  23.           shardingColumn: order_id
  24.           algorithmExpression: t_order${order_id % 2}
  25.       keyGenerator:
  26.         type: SNOWFLAKE
  27.         column: order_id
  28.     t_order_item:
  29.       actualDataNodes: ds${0..1}.t_order_item${0..1}
  30.       databaseStrategy:
  31.         inline:
  32.           shardingColumn: user_id
  33.           algorithmExpression: ds${user_id % 2}
  34.       tableStrategy:
  35.         inline:
  36.           shardingColumn: order_id
  37.           algorithmExpression: t_order_item${order_id % 2}  
  38.   bindingTables:
  39.     - t_order,t_order_item
  40.   broadcastTables:
  41.     - t_config
  43.   defaultDataSourceName: ds0
  44.   defaultTableStrategy:
  45.     none:
  46.   defaultKeyGenerator:
  47.     type: SNOWFLAKE
  48.     column: order_id
  50. props:
  51.   sql.show: true

Read-Write Split

Configuration Item Explanation

  1. dataSources:
  2.   ds_master: !!org.apache.commons.dbcp.BasicDataSource
  3.     driverClassName: com.mysql.jdbc.Driver
  4.     url: jdbc:mysql://localhost:3306/ds_master
  5.     username: root
  6.     password: 
  7.   ds_slave0: !!org.apache.commons.dbcp.BasicDataSource
  8.     driverClassName: com.mysql.jdbc.Driver
  9.     url: jdbc:mysql://localhost:3306/ds_slave0
  10.     username: root
  11.     password:
  12.   ds_slave1: !!org.apache.commons.dbcp.BasicDataSource
  13.     driverClassName: com.mysql.jdbc.Driver
  14.     url: jdbc:mysql://localhost:3306/ds_slave1
  15.     username: root
  16.     password: 
  18. masterSlaveRule:
  19.   name: ds_ms
  20.   masterDataSourceName: ds_master
  21.   slaveDataSourceNames: [ds_slave0, ds_slave1]
  23. props:
  24.   sql.show: true

Create a DataSource through the YamlMasterSlaveDataSourceFactory factory class:

  1. DataSource dataSource = YamlMasterSlaveDataSourceFactory.createDataSource(yamlFile);

Data Masking

Configuration Item Explanation

  1. dataSource:  !!org.apache.commons.dbcp2.BasicDataSource
  2.   driverClassName: com.mysql.jdbc.Driver
  3.   url: jdbc:mysql://127.0.0.1:3306/encrypt?serverTimezone=UTC&useSSL=false
  4.   username: root
  5.   password:
  7. encryptRule:
  8.   encryptors:
  9.     encryptor_aes:
  10.       type: aes
  11.       props:
  12.         aes.key.value: 123456abc
  13.     encryptor_md5:
  14.       type: md5
  15.   tables:
  16.     t_encrypt:
  17.       columns:
  18.         user_id:
  19.           plainColumn: user_plain
  20.           cipherColumn: user_cipher
  21.           encryptor: encryptor_aes
  22.         order_id:
  23.           cipherColumn: order_cipher
  24.           encryptor: encryptor_md5
  25. props:
  26.   query.with.cipher.column: true # use ciphertext column query

Orchestration

Configuration Item Explanation

  1. # Omit data sharding, Read-Write split, and Data masking configuration.
  3. orchestration:
  4.   name: orchestration_ds
  5.   overwrite: true
  6.   registry:
  7.     type: zookeeper
  8.     namespace: orchestration
  9.     serverLists: localhost:2181

ShardingSphere-3.x

Data Sharding

Configuration Item Explanation

  1. dataSources:
  2.   ds0: !!org.apache.commons.dbcp.BasicDataSource
  3.     driverClassName: com.mysql.jdbc.Driver
  4.     url: jdbc:mysql://localhost:3306/ds0
  5.     username: root
  6.     password: 
  7.   ds1: !!org.apache.commons.dbcp.BasicDataSource
  8.     driverClassName: com.mysql.jdbc.Driver
  9.     url: jdbc:mysql://localhost:3306/ds1
  10.     username: root
  11.     password: 
  13. shardingRule:  
  14.   tables:
  15.     t_order: 
  16.       actualDataNodes: ds${0..1}.t_order${0..1}
  17.       databaseStrategy:
  18.         inline:
  19.           shardingColumn: user_id
  20.           algorithmExpression: ds${user_id % 2}
  21.       tableStrategy: 
  22.         inline:
  23.           shardingColumn: order_id
  24.           algorithmExpression: t_order${order_id % 2}
  25.       keyGeneratorColumnName: order_id
  26.     t_order_item:
  27.       actualDataNodes: ds${0..1}.t_order_item${0..1}
  28.       databaseStrategy:
  29.         inline:
  30.           shardingColumn: user_id
  31.           algorithmExpression: ds${user_id % 2}
  32.       tableStrategy:
  33.         inline:
  34.           shardingColumn: order_id
  35.           algorithmExpression: t_order_item${order_id % 2}  
  36.   bindingTables:
  37.     - t_order,t_order_item
  38.   broadcastTables:
  39.     - t_config
  41.   defaultDataSourceName: ds0
  42.   defaultTableStrategy:
  43.     none:
  44.   defaultKeyGeneratorClassName: io.shardingsphere.core.keygen.DefaultKeyGenerator
  46. props:
  47.   sql.show: true

Read-Write Split

Configuration Item Explanation

  1. dataSources:
  2.   ds_master: !!org.apache.commons.dbcp.BasicDataSource
  3.     driverClassName: com.mysql.jdbc.Driver
  4.     url: jdbc:mysql://localhost:3306/ds_master
  5.     username: root
  6.     password: 
  7.   ds_slave0: !!org.apache.commons.dbcp.BasicDataSource
  8.     driverClassName: com.mysql.jdbc.Driver
  9.     url: jdbc:mysql://localhost:3306/ds_slave0
  10.     username: root
  11.     password:
  12.   ds_slave1: !!org.apache.commons.dbcp.BasicDataSource
  13.     driverClassName: com.mysql.jdbc.Driver
  14.     url: jdbc:mysql://localhost:3306/ds_slave1
  15.     username: root
  16.     password: 
  18. masterSlaveRule:
  19.   name: ds_ms
  20.   masterDataSourceName: ds_master
  21.   slaveDataSourceNames: [ds_slave0, ds_slave1]
  22.   props:
  23.       sql.show: true
  24.   configMap:
  25.       key1: value1

Create a DataSource through the YamlMasterSlaveDataSourceFactory factory class:

  1. DataSource dataSource = MasterSlaveDataSourceFactory.createDataSource(yamlFile);

Orchestration

Configuration Item Explanation

  1. # Omit data sharding, Read-Write split configuration.
  3. orchestration:
  4.   name: orchestration_ds
  5.   overwrite: true
  6.   registry:
  7.     namespace: orchestration
  8.     serverLists: localhost:2181

ShardingSphere-2.x

Data Sharding

Configuration Item Explanation

  1. dataSources:
  2.   db0: !!org.apache.commons.dbcp.BasicDataSource
  3.     driverClassName: org.h2.Driver
  4.     url: jdbc:h2:mem:db0;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MYSQL
  5.     username: sa
  6.     password: 
  7.     maxActive: 100
  8.   db1: !!org.apache.commons.dbcp.BasicDataSource
  9.     driverClassName: org.h2.Driver
  10.     url: jdbc:h2:mem:db1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MYSQL
  11.     username: sa
  12.     password: 
  13.     maxActive: 100
  15. shardingRule:
  16.   tables:
  17.     config:
  18.       actualDataNodes: db${0..1}.t_config
  19.     t_order: 
  20.       actualDataNodes: db${0..1}.t_order_${0..1}
  21.       databaseStrategy: 
  22.         standard:
  23.           shardingColumn: user_id
  24.           preciseAlgorithmClassName: io.shardingjdbc.core.yaml.fixture.SingleAlgorithm
  25.       tableStrategy: 
  26.         inline:
  27.           shardingColumn: order_id
  28.           algorithmInlineExpression: t_order_${order_id % 2}
  29.       keyGeneratorColumnName: order_id
  30.       keyGeneratorClass: io.shardingjdbc.core.yaml.fixture.IncrementKeyGenerator
  31.     t_order_item:
  32.       actualDataNodes: db${0..1}.t_order_item_${0..1}
  33.       # The strategy of binding the rest of the tables in the table is the same as the strategy of the first table
  34.       databaseStrategy: 
  35.         standard:
  36.           shardingColumn: user_id
  37.           preciseAlgorithmClassName: io.shardingjdbc.core.yaml.fixture.SingleAlgorithm
  38.       tableStrategy: 
  39.         inline:
  40.           shardingColumn: order_id
  41.           algorithmInlineExpression: t_order_item_${order_id % 2}
  42.   bindingTables:
  43.     - t_order,t_order_item
  44.   # Default database sharding strategy
  45.   defaultDatabaseStrategy:
  46.     none:
  47.   defaultTableStrategy:
  48.     complex:
  49.       shardingColumns: id, order_id
  50.       algorithmClassName: io.shardingjdbc.core.yaml.fixture.MultiAlgorithm
  51.   props:
  52.     sql.show: true

Read-Write Split

concept

In order to relieve the pressure on the database, the write and read operations are separated into different data sources. The write library is called the master library, and the read library is called the slave library. One master library can be configured with multiple slave libraries.

Supported

  1. Provides a read-write separation configuration with one master and multiple slaves, which can be used independently or with sub-databases and sub-meters.
  2. Independent use of read-write separation to support SQL transparent transmission.
  3. In the same thread and the same database connection, if there is a write operation, subsequent read operations will be read from the main library to ensure data consistency.
  4. Spring namespace.
  5. Hint-based mandatory main library routing.

Unsupported

  1. Data synchronization between the master library and the slave library.
  2. Data inconsistency caused by the data synchronization delay of the master library and the slave library.
  3. Double writing or multiple writing in the main library.

rule configuration

  1. dataSources:
  2.   db_master: !!org.apache.commons.dbcp.BasicDataSource
  3.     driverClassName: org.h2.Driver
  4.     url: jdbc:h2:mem:db_master;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MYSQL
  5.     username: sa
  6.     password: 
  7.     maxActive: 100
  8.   db_slave_0: !!org.apache.commons.dbcp.BasicDataSource
  9.     driverClassName: org.h2.Driver
  10.     url: jdbc:h2:mem:db_slave_0;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MYSQL
  11.     username: sa
  12.     password: 
  13.     maxActive: 100
  14.   db_slave_1: !!org.apache.commons.dbcp.BasicDataSource
  15.       driverClassName: org.h2.Driver
  16.       url: jdbc:h2:mem:db_slave_1;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=false;MODE=MYSQL
  17.       username: sa
  18.       password: 
  19.       maxActive: 100
  21. masterSlaveRule:
  22.   name: db_ms
  23.   masterDataSourceName: db_master
  24.   slaveDataSourceNames: [db_slave_0, db_slave_1]
  25.   configMap:
  26.     key1: value1

Create a DataSource through the MasterSlaveDataSourceFactory factory class:

  1. DataSource dataSource = MasterSlaveDataSourceFactory.createDataSource(yamlFile);

Orchestration

Configuration Item Explanation

Zookeeper sharding table and database Orchestration Configuration Item Explanation

  1. dataSources: Data sources configuration
  3. shardingRule: Sharding rule configuration
  5. orchestration: Zookeeper Orchestration Configuration
  6.   name: Orchestration name
  7.   overwrite: Whether to overwrite local configurations with config center configurations; if it can, each initialization should refer to local configurations
  8.   zookeeper: Registry center Configuration
  9.     namespace: Registry center namespace
  10.     serverLists: The list of servers that connect to governance instance, including IP and port number, use commas to separate, such as: host1:2181,host2:2181
  11.     baseSleepTimeMilliseconds: The initial millisecond value of the interval to wait for retry
  12.     maxSleepTimeMilliseconds: The maximum millisecond value of the interval to wait for retry
  13.     maxRetries: The maximum retry count
  14.     sessionTimeoutMilliseconds: The session timeout milliseconds
  15.     connectionTimeoutMilliseconds: The connecton timeout milliseconds
  16.     digest: Permission token to connect to Zookeeper. default no authorization is required

Etcd sharding table and database Orchestration Configuration Item Explanation

  1. dataSources: Data sources configuration
  3. shardingRule: Sharding rule configuration
  5. orchestration: Etcd Orchestration Configuration
  6.   name: Orchestration name
  7.   overwrite: Whether to overwrite local configurations with config center configurations; if it can, each initialization should refer to local configurations
  8.   etcd: Registry center Configuration
  9.     serverLists: The list of servers that connect to governance instance, including IP and port number, use commas to separate, such as: http://host1:2379,http://host2:2379
  10.     timeToLiveSeconds: Time to live seconds for ephemeral nodes
  11.     timeoutMilliseconds: The request timeout milliseconds
  12.     maxRetries: The maximum retry count
  13.     retryIntervalMilliseconds: The retry interval milliseconds

Sharding table and database Data source construction method

  1. DataSource dataSource = OrchestrationShardingDataSourceFactory.createDataSource(yamlFile);

Read-Write split Data source construction method

  1. DataSource dataSource = OrchestrationMasterSlaveDataSourceFactory.createDataSource(yamlFile);