add index blocked after injection network partition between pd leader and all other pods #53909
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
tidb_enable_dist_task='off'
1、run sysbench
2、add index for one of table
3、inject network partition between pd leader and all other pods
tidb logs:
tidb-0-2024-06-09T22-21-12.334.log.tar.gz
tidb-1-2024-06-09T22-09-07.701.log.tar.gz
2. What did you expect to see? (Required)
add index can success
3. What did you see instead (Required)
add index blocked after injection network partition between pd leader and all other pods
operator logs：
the status of ddl job is not synced after 1h0m0s (now: 2024-06-09 12:39:47, jobId: 491, job type: add index /* ingest /, state: running)
operatorLogs:
[2024-06-09 11:39:29] ###### start adding index
ALTER TABLE sbtest1 ADD INDEX index_test_1717904369842(c)
[2024-06-09 11:39:29] ###### wait for ddl job finish
[2024-06-09 12:39:47] ###### wait for ddl job finish timeout(1h0m0s)
select job_id, job_type, state from information_schema.ddl_jobs where query = 'ALTER TABLE sbtest1 ADD INDEX index_test_1717904369842(c)'
jobId: 491, job type: add index / ingest */, state: running
4. What is your TiDB version? (Required)
./tidb-server -V
Release Version: v8.2.0-alpha
Edition: Community
Git Commit Hash: d75ff82
Git Branch: heads/refs/tags/v8.2.0-alpha
UTC Build Time: 2024-06-08 11:46:12
GoVersion: go1.21.10
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-06-09T11:28:46.842+0800