tidb panic after inject one of AZ network partition with pitr running #52597
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
1、run pitr
2、run tpcc with 2w warehouse and 32 threads
3、inject network partition between one of az and other az last for 10mins
fault start time：2024-04-12 23:16:25
fault end time：2024-04-12 23:26:32
2. What did you expect to see? (Required)
no panic
3. What did you see instead (Required)
tidb panic after fault recover
2024-04-12 23:27:44 log="/tidb-server --store=tikv --advertise-address=tc-tidb-1.tc-tidb-peer.endless-ha-test-oltp-pitr-tps-7504383-1-172.svc --host=0.0.0.0 --path=tc-pd:2379 --config=/etc/tidb/tidb.toml\n" 2024-04-12 23:27:44 log="start tidb-server ...\n" 2024-04-12 23:27:33 log="\t/workspace/source/tidb/br/pkg/streamhelper/flush_subscriber.go:150 +0x78\n" 2024-04-12 23:27:33 log="created by github.com/pingcap/tidb/br/pkg/streamhelper.spawnJoinable in goroutine 16842\n" 2024-04-12 23:27:33 log="\t/workspace/source/tidb/br/pkg/streamhelper/flush_subscriber.go:152 +0x4d\n" 2024-04-12 23:27:33 log="github.com/pingcap/tidb/br/pkg/streamhelper.spawnJoinable.func1()\n" 2024-04-12 23:27:33 log="\t/workspace/source/tidb/br/pkg/streamhelper/flush_subscriber.go:237 +0x27\n" 2024-04-12 23:27:33 log="github.com/pingcap/tidb/br/pkg/streamhelper.(*subscription).doConnect.func1()\n" 2024-04-12 23:27:33 log="\t/workspace/source/tidb/br/pkg/streamhelper/flush_subscriber.go:280 +0xac5\n" 2024-04-12 23:27:33 log="github.com/pingcap/tidb/br/pkg/streamhelper.(*subscription).listenOver(0xc0b24bf600, {0x66287d8, 0xc086a1e420}, {0x663d180, 0xc057c9afc0})\n" 2024-04-12 23:27:33 log="goroutine 1090341181 [running]:\n" 2024-04-12 23:27:33 log="\n" 2024-04-12 23:27:33 log="panic: send on closed channel\n"
4. What is your TiDB version? (Required)
./tidb-server -V
Release Version: v8.1.0
Edition: Community
Git Commit Hash: 6bf6755
Git Branch: HEAD
UTC Build Time: 2024-04-12 11:10:02
GoVersion: go1.21.4
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-04-12T23:06:17.157+0800