版本信息

本文所研究的MySQL版本： —> MySQL 8.0.13 : https://github.com/mysql/mysql-server/releases/tag/mysql-8.0.13

协议变化

1. PARTIAL_UPDATE_ROWS_EVENT 新的binlog对象

MySQL 8.0.3之后新增了支持json局部更新的binlog事件(主要是基于性能考虑，在after事件中只会记录json的变更项，而不是完整镜像)，目前canal已经完全支持partial事件的完整解析 参考文档：https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_row_value_options

  1. # 关闭patital_json的事件
  2. set binlog_row_value_options=""
  4. # 开启patital_json的事件
  5. set binlog_row_value_options="PARTIAL_JSON"
  7. # 测试内容
  8. CREATE TABLE `test_json` (
  9.   `id` bigint(20) NOT NULL AUTO_INCREMENT,
  10.   `c_json` json DEFAULT NULL,
  11.   PRIMARY KEY (`id`)
  12. ) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COMMENT='10000000';
  14. insert into test_json values(1,'{"k1": "v1", "k2": 10}');
  15. update test_json set c_json = JSON_SET(c_json, '$.k1', 'v1') where id = 1;
  16. --> 输出：JSON_REPLACE(@1, $.k1, "v2")
  18. update test_json set c_json = JSON_SET(c_json, '$.k3', 'k3') where id = 1;
  19. --> 输出：JSON_INSERT(@1, $.k3, "k3")
  21. update test_json set c_json = JSON_REPLACE(c_json, '$.k3', CAST('[1,2]' AS JSON )) where id = 1;
  22. --> 输出：JSON_REPLACE(@1, $.k3, [1, 2])
  24. update test_json set c_json = JSON_REMOVE(c_json, '$.k4') where id = 1;
  25. --> 输出：JSON_REMOVE(@1, $.k4)
  27. update test_json set c_json = JSON_SET(c_json, '$.k4', CAST('[1,2]' AS JSON )) where id = 1;
  28. --> 输出：JSON_INSERT(@1, $.k4, [1, 2])
  31. insert into test_json values(2,'["a", {"b": [1, 2]}');
  32. update test_json set c_json = JSON_SET(c_json, '$[1]', 'v2') where id = 2;
  33. --> 输出：JSON_REPLACE(@1, $[1], "v1")
  35. update test_json set c_json = JSON_SET(c_json, '$[2]', 'v2' , '$[3]', 'v2') where id = 2;
  36. --> 输出：JSON_ARRAY_INSERT(@1, $[2], "v2", $[3], "v2")

2. QueryLogEvent 新增变量解析 ，需要修改QueryLogEvent.unpackVariables()方法，处理8.0新增的变量

  1.     /**
  2.      * The variable carries xid info of 2pc-aware (recoverable) DDL queries.
  3.      */
  4.     public static final int Q_DDL_LOGGED_WITH_XID             = 17;
  5.     /**
  6.      * This variable stores the default collation for the utf8mb4 character set.
  7.      * Used to support cross-version replication.
  8.      */
  9.     public static final int Q_DEFAULT_COLLATION_FOR_UTF8MB4   = 18;
  11.     /**
  12.      * Replicate sql_require_primary_key.
  13.      */
  14.     public static final int Q_SQL_REQUIRE_PRIMARY_KEY         = 19;

3. binlog_row_metadata=FULL特性支持

MySQL8.0.1版本之后新增在binlog里记录更多的column metadata信息，比如列名、主键、编码、SET/ENUM/GEO类型等信息，默认是MINIMAL信息 (只记录基本的type/meta/unsigned等信息)

参考文档：https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_row_metadata

  1. # 开启metadata全记录
  2. set @@global.binlog_row_metadata='FULL';

考虑binlog记录的信息，相比于canal之前定义的表元数据有缺失，比如uk/mysql type信息，即使开启binlog_row_metadata=FULL模式，更多是和TableMetaTSDB得到的元数据进行强校验.

image.png | left | 747x254