add index status was always running and can not be cancelled due to dead lock #48303
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
tidb_enable_dist_task='on'
1、run workload
2、add index for one table
3、inject one of tikv network partition
case config
tag: "ha-test-add-index"
workloads:
- type: sysbench
storage: s3://benchmark/sysbench_64_7000w
db: sysbench_64_7000w
tableNum: 64
tableSize: 70000000
threads: 64
subType: oltp_read_write
ignoreErrors: "all"
cases:
- name: "ha_tikv(random)_to_all_network_partition"
warmUpTime: 1m
faultType: network_partition
selector: "tikv(random)_to_all"
period: "@every 5m"
faultDuration: 3m
faultTotalRunTime: 30m
ddl:
addIndex:
- db: "sysbench_64_7000w"
table: "sbtest1"
timeout: 60m
times: 5
index_col_name: "c"
job_type: "ingest"
admin_check: false
count_check: false
2. What did you expect to see? (Required)
add index can success
3. What did you see instead (Required)
add index status was always running due to dead lock
the status of ddl job is not synced after 1h0m0s (now: 2023-11-04 09:14:24, jobId: 475, job type: add index /* ingest /, state: running)
operatorLogs:
[2023-11-04 08:14:10] ###### start adding index
alter table sbtest1 add index index_test_1699056850827 (c)
[2023-11-04 08:14:10] ###### wait for ddl job finish
[2023-11-04 09:14:24] ###### wait for ddl job finish timeout(1h0m0s)
select job_id, job_type, state from information_schema.ddl_jobs where query = 'alter table sbtest1 add index index_test_1699056850827 (c)'
jobId: 475, job type: add index / ingest */, state: running
the status of ddl job is not synced or running (now: 2023-11-04 09:15:32, jobId: 476, job type: add index, state: queueing)
operatorLogs:
[2023-11-04 09:15:02] ###### start adding index
alter table sbtest1 add index index_test_1699060502481 (c)
[2023-11-04 09:15:02] ###### wait for ddl job finish
[2023-11-04 09:15:32] ###### wait for import job to finish failed
select job_id, job_type, state from information_schema.ddl_jobs where query = 'alter table sbtest1 add index index_test_1699060502481 (c)'
jobId: 476, job type: add index, state: queueing
4. What is your TiDB version? (Required)
git hash：aa930910eb2316bec085ca235cb9f11fe8c7ccfa