Ingest writer memory consumption raised TiDB OOM in the million tables of dml and ddl scenario

[lilinghai]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

3 million tables and dml
ddl(add column/index, change column type) of concurrency 64

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

Release Version: v8.5.0-alpha-410-gd0ea9e50d5
Edition: Community
Git Commit Hash: d0ea9e5
Git Branch: HEAD
UTC Build Time: 2024-12-21 05:26:21
GoVersion: go1.23.4
Race Enabled: false
Check Table Before Drop: false
Store: unistore

[tangenta]

2024/12/29 21:10:43.154 +00:00] [INFO] [task_executor.go:143] ["subtask is scheduled away, cancel running"] [task-id=1587] [task-type=backfill] [subtaskID=1587]
...
[2024/12/29 21:10:43.155 +00:00] [INFO] [task_executor.go:188] ["update subtask row count failed"] [task-id=1587] [task-type=backfill] [error="context canceled"] [errorVerbose="context cancel
ed\ngithub.com...
[2024/12/29 21:10:43.156 +00:00] [WARN] [handle.go:228] ["met retryable error"] [task-id=1587] [task-type=backfill] [retry-count=0] [max-retry=30] [error="context canceled"] [errorVerbose="co
ntext canceled...
[2024/12/29 21:10:43.156 +00:00] [ERROR] [task_executor.go:325] ["run subtask failed"] [task-id=1587] [task-type=backfill] [error="context canceled"]
...
[2024/12/29 21:10:43.460 +00:00] [INFO] [task_executor.go:402] ["subtask in running state and is idempotent"] [task-id=1587] [task-type=backfill] [subtask-id=1587]
[2024/12/29 21:10:43.460 +00:00] [INFO] [task_executor.go:418] ["run subtask start"] [task-id=1587] [task-type=backfill] [subtaskID=1587] [step=read-index]
[2024/12/29 21:10:43.460 +00:00] [INFO] [backfilling_read_index.go:98] ["read index executor run subtask"] [category=ddl] ["use cloud"=false]
[2024/12/29 21:10:43.492 +00:00] [ERROR] [backfilling_operators.go:1006] ["import error"] [task-id=1587] [subtask-id=1587] [category=ddl] [error="context canceled"]
[2024/12/29 21:10:43.492 +00:00] [INFO] [task_executor.go:440] ["run subtask failed"] [task-id=1587] [task-type=backfill] [subtaskID=1587] [step=read-index] [takeTime=32.071207ms] [error="con
text canceled"] [errorVerbose="context canceled\ngithub.com/tikv/pd/client/clients/tso.(*Request).waitCtx\n\t/root/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20241220053006-461b86adc78d/clie
nts/tso/request.go:98\ngithub.com...
[2024/12/29 21:10:43.796 +00:00] [INFO] [task_executor.go:402] ["subtask in running state and is idempotent"] [task-id=1587] [task-type=backfill] [subtask-id=1587]
[2024/12/29 21:10:43.796 +00:00] [INFO] [task_executor.go:418] ["run subtask start"] [task-id=1587] [task-type=backfill] [subtaskID=1587] [step=read-index]
...

If the subtask is scheduled away and it comes back immediately, step executor always uses the old, canceled stepCtx:

tidb/pkg/disttask/framework/taskexecutor/task_executor.go

Lines 142 to 148 in 5934515

	if len(subtasks) == 0 {
	e.logger.Info("subtask is scheduled away, cancel running",
	zap.Int64("subtaskID", e.currSubtaskID.Load()))
	// cancels runStep, but leave the subtask state unchanged.
	e.cancelRunStepWith(nil)
	return
	}