multi-schema-change ADD INDEX may cause data-index inconsistency

[lance6716]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

found by CI, failed to build writer does not fail the DDL job

[2024/06/03 19:01:11.373 +08:00] [INFO] [engine_mgr.go:96] ["open an engine for index reorg task"] ["job ID"=130] ["index IDs"="[13,14,15,16,17,18,19,20,21,22,23,24]"] ["current memory usage"=6442452808] ["memory limitation"=8589934592]
...
[2024/06/03 19:01:11.451 +08:00] [WARN] [message.go:80] ["allocate memory failed"] ["job ID"=130] ["index ID"=13] ["current memory usage"=8468302240] ["max memory quota"=8589934592]
[2024/06/03 19:01:11.451 +08:00] [ERROR] [engine.go:200] ["build ingest writer context failed"] [error="[ddl:8247]Ingest failed: memory used up"] ["job ID"=130] ["index ID"=13] ["worker ID"=18]
[2024/06/03 19:01:11.451 +08:00] [ERROR] [backfilling_operators.go:597] ["create index ingest worker failed"] [task-id=1] [subtask-id=1] [error="[ddl:8247]Ingest failed: memory used up"]
[2024/06/03 19:01:11.451 +08:00] [INFO] [backfilling_operators.go:700] ["finish a index ingest task"] [task-id=1] [subtask-id=1] [id=1]
...
[2024/06/03 19:01:11.468 +08:00] [INFO] [backend.go:225] ["acquire distributed flush lock success"] [jobID=130]

just reading the code

tidb/pkg/ddl/backfilling_operators.go

Lines 585 to 588 in b23ac8e

	writer, err := engines[i].CreateWriter(writerID)
	if err != nil {
	logutil.Logger(ctx).Error("create index ingest worker failed", zap.Error(err))
	return nil

we should not ignore the error

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

[lance6716]

After I run lightning CI locally, can verify that data-index inconsistency

MySQL [add_index]> admin check table multi_indexes;
ERROR 8223 (HY000): data inconsistency in table: multi_indexes, index: idx_c2, handle: 1, index-values:"" != record-values:"handle: 1, values: [KindString R001_2]"

[lance6716]

Seems it's a problem of my own PR 😂