data inconsistency in table (lightning didn't ingest all kv) after tikv is down longer than 810s

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、tidb_enable_dist_task='off'
2、run sysbench
3、add index for one table
4、tikv rolling restart
5、admin check

logs:
tidb-0.zip
tidb-1.zip

2. What did you expect to see? (Required)

admin check success

3. What did you see instead (Required)

admin check failed with error

admin check failed (Error 8223 (HY000): data inconsistency in table: sbtest1, index: index_test_1725249594568, handle: 428304, index-values:"" != record-values:"handle: 428304, values: [KindString 72115182252-02679645832-08099037427-76043912395-43440959176-93229371484-33119819645-19100075546-83614225936-74898877917]")
operatorLogs:
[2024-09-02 11:59:54] ###### start adding index
ALTER TABLE sbtest1 ADD INDEX index_test_1725249594568(c)
[2024-09-02 11:59:54] ###### wait for ddl job finish
[2024-09-02 12:18:27] ###### ddl job finished
select job_id, db_name, table_name, job_type, create_time, start_time, end_time, state, query from information_schema.ddl_jobs where query = 'ALTER TABLE sbtest1 ADD INDEX index_test_1725249594568(c)'
jobId: 563, job type: add index /* ingest */, state: synced
add index done, it takes: 18m32.781865519s
[2024-09-02 12:18:27] ###### start admin check
admin check index sbtest1 index_test_1725249594568

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v8.4.0-alpha
Edition: Community
Git Commit Hash: 3419bde
Git Branch: heads/refs/tags/v8.4.0-alpha
UTC Build Time: 2024-08-31 11:47:42
GoVersion: go1.21.10
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-09-02T11:56:15.822+0800

[Lily2025]

/assign tangenta

[lance6716]

the problem is lightning forget to set j.lastRetryableErr before line 636

tidb/pkg/lightning/backend/local/region_job.go

Lines 623 to 636 in 01797cb

	for retry := 0; retry < maxRetryTimes; retry++ {
	resp, err := local.doIngest(ctx, j)
	if err == nil && resp.GetError() == nil {
	j.convertStageTo(ingested)
	return nil
	}
	if err != nil {
	if common.IsContextCanceledError(err) {
	return err
	}
	log.FromContext(ctx).Warn("meet underlying error, will retry ingest",
	log.ShortError(err), logutil.SSTMetas(j.writeResult.sstMeta),
	logutil.Region(j.region.Region), logutil.Leader(j.region.Leader))
	continue

when retry exceeded the limit (30), it will return nil error which is not expected

tidb/pkg/lightning/backend/local/local.go

Lines 1463 to 1474 in 01797cb

	case job, ok = <-jobFromWorkerCh:
	}
	if !ok {
	retryer.close()
	return nil
	}
	switch job.stage {
	case regionScanned, wrote:
	job.retryCount++
	if job.retryCount > maxWriteAndIngestRetryTimes {
	job.done(&jobWg)
	return job.lastRetryableErr