Fail to scan regions when using Follower Handle

[3pointer]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

During a restore from a checkpoint in a large-scale cluster(3M regions), I observed that BR continuously reported the following error:

[2025/03/14 05:58:30.092 +00:00] [WARN] [split.go:254] ["failed to scan region, retrying"] [error="region 15872724's endKey not equal to next region 15872732's startKey, endKey: 

However, upon checking with pd-ctl, I found that the next region after 15872724 was actually 15872728, and there was no specific region role.

This issue started after merging pingcap/tidb#59783, which introduced the use of followerHandle for pd client scanning regions.

After disabling the followerHandle option, the error no longer occurred.

This suggests that the root cause is likely related to inconsistencies in the region tree synchronization between the PD leader and follower. It appears that the follower’s region information is not fully up-to-date, causing discrepancies during the region scanning process. This synchronization issue needs to be addressed.

2. What did you expect to see? (Required)

No error

3. What did you see instead (Required)

keep scan keep fail

4. What is your TiDB version? (Required)

master

[rleungx]

I think PD needs to fallback to the leader in this case, if the follower does have the hole. Or we can disable it in BR because it uses ScanRegions right now. /cc @okJiang

[okJiang]

@3pointer Will this issue cause the BR restore to fail? Can you upload the BR logs?

[3pointer]

It's will keep trying to scan regions for at least 30minutes.

[2025/03/10 **11:51:06.**341 +00:00] [WARN] [split.go:254] ["failed to scan region, retrying"] [error="region 15043270's endKey not equal to next region 19117209's startKey, endKey: 7480000000000000FFA75F728000000196FFB51F0A0000000000FA, startKey: 7480000000000000FFA75F728000000196FFB5383A0000000000FA, region epoch: conf_ver:101 version:24411  conf_ver:89 version:24411 : [BR:PD:ErrPDBatchScanRegion]batch scan region"] [regionLength=3101044]
...
[2025/03/10 **12:19:01**.553 +00:00] [WARN] [split.go:254] ["failed to scan region, retrying"] [error="region 15043270's endKey not equal to next region 19117209's startKey, endKey: 7480000000000000FFA75F728000000196FFB51F0A0000000000FA, startKey: 7480000000000000FFA75F728000000196FFB5383A0000000000FA, region epoch: conf_ver:101 version:24411  conf_ver:89 version:24411 : [BR:PD:ErrPDBatchScanRegion]batch scan region"] [regionLength=3101044]

[okJiang]

There are two points of doubt:

1. The ScanRegion has been evenly distributed across all PD nodes during this period. Due to the existence of three PDs in the cluster, it is possible that BR hits a PD follower for this region every time it retries. It is possible in the current code of BR retry Scanregion.
2. It is unreasonable for RegionSyncer to experience such a long region inconsistency.

Further investigation needed.

[okJiang]

pd-2 is leader. Its region sync index is less than follower. It is unexpected.

[okJiang]

ref tikv/pd#9154