dead chanel in UpdateNewAndDoneWatch

[xhebox]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

It started here

tidb/pkg/domain/runaway.go

Lines 75 to 80 in e601725

	err = do.runawayManager.UpdateNewAndDoneWatch()
	if err == nil {
	logutil.BgLogger().Info("preparations for the runaway manager are finished and start runaway manager loop")
	do.wg.Run(do.runawayManager.RunawayRecordFlushLoop, "runawayRecordFlushLoop")
	do.wg.Run(do.runawayManager.RunawayWatchSyncLoop, "runawayWatchSyncLoop")
	do.runawayManager.MarkSyncerInitialized()

will insert record into channels here

tidb/pkg/resourcegroup/runaway/manager.go

Lines 424 to 426 in 6a12f31

	} else if ttl <= 0 {
	rm.staleQuarantineRecord <- record
	return

The receive loop however is started later than the first call of UpdateNewAndDoneWatch(RunawayRecordFlushLoop)

tidb/pkg/resourcegroup/runaway/manager.go

Line 211 in 6a12f31

case r := <-staleQuarantineRecordCh:

If the channel exceeds the maximum(1024), it blocks.

introduced by #52283

2. What did you expect to see? (Required)

non blocking.

3. What did you see instead (Required)

runawayStartLoop blocks forever

4. What is your TiDB version? (Required)

v8.1.0 and later