When upgrading has modify column DDL and the cluster has more than 1 TiDB, we may block. #47912
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
ref PR: #3993 and #47818
deploy v7.1.0 cluster with 2 tidb.
Then upgrade this cluster to master one by one.
2. What did you expect to see? (Required)
Upgrade successfully.
3. What did you see instead (Required)
tidb> admin show ddl jobs;
+--------+---------+-------------------------+---------------+--------------+-----------+-----------------+-----------+---------------------+---------------------+---------------------+--------+
| JOB_ID | DB_NAME | TABLE_NAME | JOB_TYPE | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | CREATE_TIME | START_TIME | END_TIME | STATE |
+--------+---------+-------------------------+---------------+--------------+-----------+-----------------+-----------+---------------------+---------------------+---------------------+--------+
| 97 | mysql | tidb_runaway_queries | modify column | public | 1 | 93 | 0 | 2023-10-23 12:13:40 | 2023-10-23 12:13:40 | NULL | done |
| 96 | mysql | tidb_timers | create table | public | 1 | 95 | 0 | 2023-10-23 12:13:39 | 2023-10-23 12:13:39 | 2023-10-23 12:13:39 | synced |
| 94 | mysql | tidb_runaway_queries | create table | public | 1 | 93 | 0 | 2023-10-23 12:13:38 | 2023-10-23 12:13:39 | 2023-10-23 12:13:39 | synced |
| 92 | mysql | tidb_import_jobs | create table | public | 1 | 91 | 0 | 2023-10-23 12:13:38 | 2023-10-23 12:13:38 | 2023-10-23 12:13:38 | synced |
| 90 | mysql | tidb_background_subtask | add column | public | 1 | 281474976710650 | 0 | 2023-10-23 12:13:37 | 2023-10-23 12:13:37 | 2023-10-23 12:13:38 | synced |
| 89 | test | battle_begin | create table | public | 2 | 88 | 0 | 2023-10-23 12:11:28 | 2023-10-23 12:11:28 | 2023-10-23 12:11:28 | synced |
| 87 | mysql | load_data_jobs | create table | public | 1 | 86 | 0 | 2023-10-23 12:08:37 | 2023-10-23 12:08:37 | 2023-10-23 12:08:37 | synced |
| 85 | mysql | tidb_global_task | create table | public | 1 | 84 | 0 | 2023-10-23 12:08:37 | 2023-10-23 12:08:37 | 2023-10-23 12:08:37 | synced |
| 83 | mysql | tidb_ttl_job_history | create table | public | 1 | 82 | 0 | 2023-10-23 12:08:37 | 2023-10-23 12:08:37 | 2023-10-23 12:08:37 | synced |
| 81 | mysql | tidb_ttl_task | create table | public | 1 | 80 | 0 | 2023-10-23 12:08:37 | 2023-10-23 12:08:37 | 2023-10-23 12:08:37 | synced |
| 79 | mysql | tidb_ttl_table_status | create table | public | 1 | 78 | 0 | 2023-10-23 12:08:36 | 2023-10-23 12:08:36 | 2023-10-23 12:08:36 | synced |
+--------+---------+-------------------------+---------------+--------------+-----------+-----------------+-----------+---------------------+---------------------+---------------------+--------+
11 rows in set (0.00 sec)
And have a lot of logs like the following information:
[INFO] [job_table.go:417] ["[ddl] handle ddl job failed"] [error="[ddl:8213]Version 2 of DDL job is greater than current one: 1"] [job="ID:97, Type:modify column, State:synced, SchemaState:public, SchemaID:1, TableID:93, RowCount:0, ArgLen:0, start time: 2023-10-23 12:13:40.062 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:0, UniqueWarnings:0"]
4. What is your TiDB version? (Required)
master