CDC panic: send on closed channel

[pingyu]

What did you do?

1. Start up the cluster.
2. Create changefeed.
3. Start workload.
4. Stop workload.
5. Pause the changefeed by HTTP API.
6. Restart the PD (only 1 PD instance).

What did you expect to see?

Nothing wrong.

What did you see instead?

TiCDC panic: send on closed channel

goroutine 603 [running]:
github.com/pingcap/tiflow/pkg/chann.(*Chann[...]).Close(...)
	github.com/pingcap/tiflow/pkg/chann/chann.go:171
github.com/pingcap/tiflow/pkg/chann.(*DrainableChann[...]).CloseAndDrain(...)
	github.com/pingcap/tiflow/pkg/chann/drainable_chann.go:46
github.com/pingcap/tiflow/cdc/kv.(*SharedClient).Close(0xc0008fe3c0)
	github.com/pingcap/tiflow/cdc/kv/shared_client.go:264 +0x70
github.com/pingcap/tiflow/cdc/puller.(*MultiplexingPuller).Close(0xc002dbac40)
	github.com/pingcap/tiflow/cdc/puller/multiplexing_puller.go:283 +0x35
github.com/pingcap/tiflow/cdc/puller.(*ddlJobPullerImpl).Close(0x0?)
	github.com/pingcap/tiflow/cdc/puller/ddl_puller.go:106 +0x1d
github.com/pingcap/tiflow/cdc/puller.(*ddlPullerImpl).Close(0xc0036d05a0)
	github.com/pingcap/tiflow/cdc/puller/ddl_puller.go:831 +0x48
github.com/pingcap/tiflow/cdc/owner.(*changefeed).releaseResources(0xc0010a8000, {0x58150b0?, 0xc0050e52a8})
	github.com/pingcap/tiflow/cdc/owner/changefeed.go:770 +0x115
github.com/pingcap/tiflow/cdc/owner.(*changefeed).tick(0xc0010a8000, {0x58150b0?, 0xc0050e52a8}, 0xc00410f3e8?)
	github.com/pingcap/tiflow/cdc/owner/changefeed.go:336 +0x1528
github.com/pingcap/tiflow/cdc/owner.(*changefeed).Tick(0xc0010a8000, {0x5815100, 0xc0050e5278}, 0x7?)
	github.com/pingcap/tiflow/cdc/owner/changefeed.go:254 +0x2bc
github.com/pingcap/tiflow/cdc/owner.(*ownerImpl).Tick(0xc0008eaaf0, {0x7f34d8f6d118?, 0xc003712640}, {0x57e7190?, 0xc0008e3a40})
	github.com/pingcap/tiflow/cdc/owner/owner.go:182 +0x385
github.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run(0xc003153080, {0x7f34d8f6d118?, 0xc003712640}, 0xc00370a450, 0x2faf080, {0x4c28355, 0x5})
	github.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:290 +0x1069
github.com/pingcap/tiflow/cdc/capture.(*captureImpl).runEtcdWorker(0xc003332d80, {0x5814f60?, 0xc003712640}, {0x7f34d8e79f90, 0xc0008eaaf0}, {0x57e7190, 0xc0008e3a40}, 0x9ab474?, {0x4c28355, 0x5})
	github.com/pingcap/tiflow/cdc/capture/capture.go:596 +0x185
github.com/pingcap/tiflow/cdc/capture.(*captureImpl).campaignOwner.func4()
	github.com/pingcap/tiflow/cdc/capture/capture.go:532 +0x2ea
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync@v0.9.0/errgroup/errgroup.go:78 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 525
	golang.org/x/sync@v0.9.0/errgroup/errgroup.go:75 +0x96

cdc-1.log.zip
rep-pd-1.log.zip

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

TiDB: 7.5 (CSE)
TiKV: 7.5 (CSE)
PD: 8.1 (CSE)

Upstream TiKV version (execute tikv-server --version):

TiDB: 7.5 (CSE), unistore.
PD: 8.1 (CSE)

TiCDC version (execute cdc version):

latest commit of release-7.5 041c9fb

[pingyu]

cc @flowbehappy @tenfyzhong

[wk989898]

It should be fixed by #12245

[pingyu]

It should be fixed by #12245

Great ! Can you help to merge the cherry-pick #12263 to release-7.5 ?