changefeed lag increase after scale-in and then scale-out the TiKV cluster

[3AceShowHand]

What did you do?

• scale-in the TiKV cluster
• scale-out the TiKV cluster

What did you expect to see?

The changefeed still keeps normal

What did you see instead?

The resolved-ts and checkpoint-ts lag increase

Address              Old ID    New ID
10.107.2.95:20160    8         2945361
10.107.3.90:20160    4         2945362
10.107.4.96:20160    6         2945363

[2025/05/01 15:59:07.296 +08:00] [INFO] [shared_stream.go:93] ["event feed check store version fails"] [namespace=default] [changefeed=xxxx] [streamID=15] [storeID=6] [addr=10.107.4.96:20160] [error="[CDC:ErrGetAllStoresFailed]get stores from pd failed: type:UNKNOWN message:"invalid store ID 6, not found" "]

After the TiKV scale-out, the changefeed still access the TiKV store by using the store id before the scale-in, which is outdate.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

v7.5.6

[3AceShowHand]

This is similar to the issue #11766