TiDB 4.0.15 Release Notes

TiDB 4.0.15 Release Notes

Release Date: September 27, 2021

TiDB version: 4.0.15

Compatibility changes

TiDB
- Fix the issue that executing SHOW VARIABLES in a new session is slow. This fix reverts some changes made in #21045 and might cause compatibility issues. #24326
- The following bug fixes change execution results, which might cause upgrade incompatibilities:
  - Fix the issue that greatest(datetime) union null returns empty string #26532
  - Fix the issue that the having clause might not work correctly #26496
  - Fix the wrong execution results that occur when the collations around the between expression are different #27146
  - Fix the result wrong that occurs when the argument of the extract function is a negative duration #27236
  - Fix the wrong execution results that occur when the column in the group_concat function has a non-bin collation #27429
  - Fix the issue that column information is missed when converting the Apply operator to Join #27233
  - Fix the issue of unexpected behavior when casting the invalid string to DATE #26762
  - Fix a bug that the count distinct result on multiple columns is wrong when the new collation is enabled #27091

Feature enhancement

TiKV
- Support changing TiCDC configurations dynamically #10645

Improvements

TiDB
- Trigger auto-analyze based on the histogram row count #24237
TiKV
- Handle read ready and write ready separately to reduce read latency #10475
- The slow log of TiKV coprocessor only considers the time spent on processing requests. #10841
- Drop log instead of blocking threads when the slogger thread is overloaded and the queue is filled up #10841
- Reduce the size of Resolved TS messages to save network bandwidth #2448
PD
- Improve the performance of synchronizing Region information between PDs #3932
Tools
- Backup & Restore (BR)
  - Split and scatter Regions concurrently to improve restore speed #1363
  - Retry BR tasks when encountering the PD request error or the TiKV I/O timeout error #27787
  - Reduce empty Regions when restoring many small tables to avoid affecting cluster operations after the restore #1374
  - Perform the rebase auto id operation while creating tables, which saves the separate rebase auto id DDL operation and speeds up restore #1424
- Dumpling
  - Filter the skipped databases before getting the table information to improve the filtering efficiency of SHOW TABLE STATUS #337
  - Use SHOW FULL TABLES to get table information for tables to be exported, because SHOW TABLE STATUS cannot work properly in some MySQL versions #322
  - Support backing up MySQL-compatible databases that do not support the START TRANSACTION ... WITH CONSISTENT SNAPSHOT or the SHOW CREATE TABLE syntax #309
  - Refine the Dumpling warning log to avoid the misleading information that a dump fails #340
- TiDB Lightning
  - Support importing data into tables that have expression index or the index that depends on virtual generated columns #1404
- TiCDC
  - Always pulls old values from TiKV internally to improve usability #2397
  - Reduce the goroutine usage when a table’s Regions are all transferred away from a TiKV node #2284
  - Optimize workerpool for fewer goroutines when concurrency is high #2211
  - Execute DDL statements asynchronously to avoid affecting other changefeeds #2295
  - Add a global gRPC connection pool and share gRPC connections among KV clients #2531
  - Fail fast for unrecoverable DML errors #1724
  - Optimize memory management when the Unified Sorter is using memory to sort data #2553
  - Add Prometheus metrics for DDL executions #2595 #2669
  - Prohibit operating TiCDC clusters across major or minor versions #2601
  - Remove file sorter #2325
  - Clean up changefeed metrics when a changefeed is removed, and clean up processor metrics when a processor exits #2156
  - Optimize the lock-resolving algorithm after a Region is initialized #2188

Bug fixes

TiDB
- Fix a bug that collation is incorrectly set for binary literals when building ranges #23672
- Fix the “index out of range” error that occurs when a query includes both GROUP BY and UNION #26553
- Fix the issue that TiDB might fail to send requests if TiKV has tombstone stores #23676 #24648
- Remove the undocumented /debug/sub-optimal-plan HTTP API #27264
- Fix the issue of wrong character set and collation for the case when expression #26662
TiKV
- Fix the issue that BR reports the “file already exists” error when TDE is enabled during data restore #1179
- Fix the potential disk full issue caused by corrupted snapshot files #10813
- Fix the issue that TiKV deletes stale Regions too frequently #10680
- Fix the issue that TiKV frequently reconnects the PD client #9690
- Check stale file information from the encryption file dictionary #9115
PD
- Fix the issue that PD does not fix the down peers in time #4077
- Fix a bug that PD might panic when scaling out TiKV #3868
TiFlash
- Fix the potential issue of data inconsistency that occurs when TiFlash is deployed on multiple disks
- Fix a bug of incorrect results that occurs when queries contain filters like CONSTANT, <, <=, >, >=, or COLUMN
- Fix the issue that the store size in metrics is inaccurate under heavy writing
- Fix a potential bug that TiFlash cannot restore data when deployed on multiple disks
- Fix the potential issue that TiFlash cannot garbage-collect the delta data after running for a long time
Tools
- Backup & Restore (BR)
  - Fix a bug that the average speed is inaccurately calculated for backup and restore #1405
- TiCDC
  - Fix the ErrSchemaStorageTableMiss error that occurs when the DDL Job duplication is encountered in the integrated test #2422
  - Fix a bug that a changefeed cannot be removed if the ErrGCTTLExceeded error occurs #2391
  - Fix the issue that outdated capture might appear in the output of the capture list command #2388
  - Fix the deadlock issue in the TiCDC processor #2017
  - Fix a data inconsistency issue that occurs because multiple processors might write data to the same table when this table is being re-scheduled #2230
  - Fix a bug that the EtcdWorker snapshot isolation is violated in metadata management #2557
  - Fix the issue that the changefeed cannot be stopped due to the DDL sink error #2552
  - Fix the issue of TiCDC Open Protocol: TiCDC outputs an empty value when there is no change in a transaction #2612
  - Fix a bug that causes TiCDC to panic on the unsigned TINYINT type #2648
  - Decrease the gRPC window size to avoid the OOM that occurs when TiCDC captures too many Regions #2202
  - Fix the OOM issue that occurs when TiCDC captures too many Regions #2673
  - Fix the issue of process panic that occurs when encoding the data types such as mysql.TypeString, mysql.TypeVarString, mysql.TypeVarchar into JSON #2758
  - Fix the a memory leak issue that might occur when creating a new changefeed #2389
  - Fix a bug that DDL handling fails when a changefeed starts at the finish TS of a schema change #2603
  - Fix the issue of potential DDL loss when the owner crashes when executing DDL statements #1260
  - Fix the issue of insecure concurrent access to the map in SinkManager #2298