Skip to content

non-dist-task fast-reorg ADD INDEX can't be canceled timely #56017

New issue
New issue
Closed
#56404
Closed
non-dist-task fast-reorg ADD INDEX can't be canceled timely#56017
#56404
Labels
affects-6.5This bug affects the 6.5.x(LTS) versions.affects-7.1This bug affects the 7.1.x(LTS) versions.affects-7.5This bug affects the 7.5.x(LTS) versions.affects-8.1This bug affects the 8.1.x(LTS) versions.component/ddlThis issue is related to DDL of TiDB.report/customerCustomers have encountered this bug.severity/majortype/bugThe issue is confirmed as a bug.
@lance6716

Description

@lance6716
lance6716
opened on Sep 11, 2024

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

It has been stuck in "cancelling" for a long time, and lightning's goroutine didn't exit.

1 @ 0x474cf87 0x474fbea 0x474e93b 0x474e9c5 0x474e9c5 0x472e3cc 0x472e38b 0x472d8aa 0x472dba5 0x476b42b 0x48a8970 0x48a83d3 0x481cb25 0x481c4e9 0x481c1a5 0x492d6f0 0x492b545 0x30f90ef 0x1f6c901
#	0x474cf86	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).flushKVs.func1+0xc6				/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1355
#	0x474fbe9	slices.partitionCmpFunc[...]+0xe49										/usr/local/go/src/slices/zsortanyfunc.go:154
#	0x474e93a	slices.pdqsortCmpFunc[...]+0x49a										/usr/local/go/src/slices/zsortanyfunc.go:114
#	0x474e9c4	slices.pdqsortCmpFunc[...]+0x524										/usr/local/go/src/slices/zsortanyfunc.go:121
#	0x474e9c4	slices.pdqsortCmpFunc[...]+0x524										/usr/local/go/src/slices/zsortanyfunc.go:121
#	0x472e3cb	slices.SortFunc[...]+0x8b											/usr/local/go/src/slices/sort.go:28
#	0x472e38a	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).flushKVs+0x4a					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1355
#	0x472d8a9	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).appendRowsUnsorted+0x5e9				/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1242
#	0x472dba4	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).AppendRows+0x264					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1279
#	0x476b42a	github.com/pingcap/tidb/pkg/ddl/ingest.(*writerContext).WriteRow+0x1aa						/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ingest/engine.go:217
#	0x48a896f	github.com/pingcap/tidb/pkg/ddl.writeOneKVToLocal+0x1ef								/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/index.go:1781
#	0x48a83d2	github.com/pingcap/tidb/pkg/ddl.writeChunkToLocal+0xb92								/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/index.go:1740
#	0x481cb24	github.com/pingcap/tidb/pkg/ddl.(*indexIngestBaseWorker).WriteChunk+0x124					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/backfilling_operators.go:874
#	0x481c4e8	github.com/pingcap/tidb/pkg/ddl.(*indexIngestBaseWorker).HandleTask+0xa8					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/backfilling_operators.go:812
#	0x481c1a4	github.com/pingcap/tidb/pkg/ddl.(*indexIngestLocalWorker).HandleTask+0x124					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/backfilling_operators.go:763
#	0x492d6ef	github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool.(*WorkerPool[...]).handleTaskWithRecover+0x1cf	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool/workerpool.go:142
#	0x492b544	github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool.(*WorkerPool[...]).runAWorker.func1+0x84		/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool/workerpool.go:158
#	0x30f90ee	github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1+0x4e						/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/wait_group_wrapper.go:157

1 @ 0x474cf87 0x474fdc4 0x474e93b 0x474ea38 0x474ea38 0x474ea38 0x474ea38 0x474e9c5 0x474ea38 0x472e3cc 0x472e38b 0x472d8aa 0x472dba5 0x476b42b 0x48a8970 0x48a83d3 0x481cb25 0x481c4e9 0x481c1a5 0x492d6f0 0x492b545 0x30f90ef 0x1f6c901
#	0x474cf86	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).flushKVs.func1+0xc6				/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1355
#	0x474fdc3	slices.partitionCmpFunc[...]+0x1023										/usr/local/go/src/slices/zsortanyfunc.go:157
#	0x474e93a	slices.pdqsortCmpFunc[...]+0x49a										/usr/local/go/src/slices/zsortanyfunc.go:114
#	0x474ea37	slices.pdqsortCmpFunc[...]+0x597										/usr/local/go/src/slices/zsortanyfunc.go:125
#	0x474ea37	slices.pdqsortCmpFunc[...]+0x597										/usr/local/go/src/slices/zsortanyfunc.go:125
#	0x474ea37	slices.pdqsortCmpFunc[...]+0x597										/usr/local/go/src/slices/zsortanyfunc.go:125
#	0x474ea37	slices.pdqsortCmpFunc[...]+0x597										/usr/local/go/src/slices/zsortanyfunc.go:125
#	0x474e9c4	slices.pdqsortCmpFunc[...]+0x524										/usr/local/go/src/slices/zsortanyfunc.go:121
#	0x474ea37	slices.pdqsortCmpFunc[...]+0x597										/usr/local/go/src/slices/zsortanyfunc.go:125
#	0x472e3cb	slices.SortFunc[...]+0x8b											/usr/local/go/src/slices/sort.go:28
#	0x472e38a	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).flushKVs+0x4a					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1355
#	0x472d8a9	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).appendRowsUnsorted+0x5e9				/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1242
#	0x472dba4	github.com/pingcap/tidb/pkg/lightning/backend/local.(*Writer).AppendRows+0x264					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine.go:1279
#	0x476b42a	github.com/pingcap/tidb/pkg/ddl/ingest.(*writerContext).WriteRow+0x1aa						/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ingest/engine.go:217
#	0x48a896f	github.com/pingcap/tidb/pkg/ddl.writeOneKVToLocal+0x1ef								/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/index.go:1781
#	0x48a83d2	github.com/pingcap/tidb/pkg/ddl.writeChunkToLocal+0xb92								/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/index.go:1740
#	0x481cb24	github.com/pingcap/tidb/pkg/ddl.(*indexIngestBaseWorker).WriteChunk+0x124					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/backfilling_operators.go:874
#	0x481c4e8	github.com/pingcap/tidb/pkg/ddl.(*indexIngestBaseWorker).HandleTask+0xa8					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/backfilling_operators.go:812
#	0x481c1a4	github.com/pingcap/tidb/pkg/ddl.(*indexIngestLocalWorker).HandleTask+0x124					/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/backfilling_operators.go:763
#	0x492d6ef	github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool.(*WorkerPool[...]).handleTaskWithRecover+0x1cf	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool/workerpool.go:142
#	0x492b544	github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool.(*WorkerPool[...]).runAWorker.func1+0x84		/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/resourcemanager/pool/workerpool/workerpool.go:158
#	0x30f90ee	github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1+0x4e						/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/wait_group_wrapper.go:157

Then it begins to generateAndSendJob. Seems the ctx is not cancelled

4. What is your TiDB version? (Required)

+-----------------------------------------------------------+
| tidb_version()                                            |
+-----------------------------------------------------------+
| Release Version: v8.5.0-alpha                             |
| Edition: Community                                        |
| Git Commit Hash: b97baba693964ef0625eb7f0b4b321d40a16b3fe |
| Git Branch: heads/refs/tags/v8.5.0-alpha                  |
| UTC Build Time: 2024-09-09 04:40:51                       |
| GoVersion: go1.21.10                                      |
| Race Enabled: false                                       |
| Check Table Before Drop: false                            |
| Store: tikv                                               |
+-----------------------------------------------------------+

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects-6.5This bug affects the 6.5.x(LTS) versions.affects-7.1This bug affects the 7.1.x(LTS) versions.affects-7.5This bug affects the 7.5.x(LTS) versions.affects-8.1This bug affects the 8.1.x(LTS) versions.component/ddlThis issue is related to DDL of TiDB.report/customerCustomers have encountered this bug.severity/majortype/bugThe issue is confirmed as a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions