Progress total on BR-restore v8.5 is overestimated, causing the progress bar to be stuck at a very low value during checksum

[kennytm]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Backup a cluster with large amount of small tables.
2. Restore it.
3. Observe the [progress] numbers in the log

2. What did you expect to see? (Required)

The progress increases smoothly from 0% to 100%

3. What did you see instead (Required)

The grows "smoothly" from 0% to maybe about 40%, and the keep unchanged during checksum, and suddenly jump to 100% at the end.

4. What is your TiDB version? (Required)

BR v8.5.1

[kennytm]

(internal reference TICKET-6117)

The issue should not affect v9 because of the refactoring #56612. But I suggest double-checking for sure.

The issue seems to be caused by #55662 so it should only affect v8.5 and not lower LTS versions, but again I suggest double-checking for sure.

Originally, the total progress is computed as a sum of:

1. Total *_write.sst files, to estimate the number of regions, and thus the number of split+scatter operations
2. Total *.sst files, to estimate the number of download+ingest operations
3. Total number of tables, to estimate the number of checksum operations

tidb/br/pkg/task/restore.go

Lines 1081 to 1086 in af5544d

	// Split/Scatter + Download/Ingest
	progressLen := int64(rangeSize + len(files))
	if cfg.Checksum {
	progressLen += int64(len(tables))
	}
	if cfg.WaitTiflashReady {

However, items 1 and 2 overestimated the number of operations, because

1. Since br: restore merge split ranges with small data #55662 multiple ranges may be pre-merged, thus reducing the total number of regions
2. The number of download+ingest operations is actually equal to the number of regions, not the number of SSTs, as we download/ingest them in a group.
   Both of these significantly reduces the total count of increment, and thus even when the actual progress is 99% complete, to visible value has just grown to ~40%.

This should be fixed by adjusting the amount of .Inc() when counting progress so it can catch up with the original estimation.

1. tidb/br/pkg/restore/snap_client/client.go

   Lines 1081 to 1083 in af5544d

   	log.Info("import sst files done", logutil.Files(files))
   	updateCh.Inc()
   	}()

   
   This line should be changed to updateCh.IncBy(len(files))

2. tidb/br/pkg/restore/snap_client/tikv_sender.go

   Lines 335 to 340 in af5544d

   	splitClientOpts = append(splitClientOpts, split.WithOnSplit(func(keys [][]byte) {
   	for range keys {
   	updateCh.Inc()
   	}
   	}))
   	if isRawKv {

   
   This line should be changed to updateCh.IncBy(/* the total number of original ranges between those keys */).