[Lightning]: Panic on invalid Mertic countter operation

[0xPoe]

Bug Report

Version: 6.1.0

Lightning panic on https://github.com/pingcap/tidb/blob/master/br/pkg/lightning/restore/restore.go#L2333

I think it's because float64(currOffset - startOffset) < 0.

panic: counter cannot decrease in value

goroutine 202374 [running]:
github.com/prometheus/client_golang/prometheus.(*counter).Add(0x0?, 0xc069674c80?)
/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/counter.go:109 +0xf4
github.com/pingcap/tidb/br/pkg/lightning/restore.(*chunkRestore).deliverLoop(0xc0618805a0,
{0x445dbd8, 0xc0017c6040}

, 0xc06c80a3c0, 0xc0012af740, 0x3, 0xc002a8ff18?, 0xc002a8ffd0?, 0xc0016e5080)
/mnt/tidb/sql/br/pkg/lightning/restore/restore.go:2255 +0x7a3
github.com/pingcap/tidb/br/pkg/lightning/restore.(*chunkRestore).restore.func1()
/mnt/tidb/sql/br/pkg/lightning/restore/restore.go:2509 +0xab
created by github.com/pingcap/tidb/br/pkg/lightning/restore.(*chunkRestore).restore
/mnt/tidb/sql/br/pkg/lightning/restore/restore.go:2507 +0x31b

[D3Hunter]

value of currOffset comes from parser.pos which increase monotonically. the init value of parser.pos comes from chunk.Chunk.Offset. so it should happen that currOffset - startOffset < 0

need to reproduce it.

[D3Hunter]

user changed lightning code, cannot find code related to hash b3ee7aef716e39dd5c6744e4dac0fbbb3529aca7

not sure the root cause is from 6.1 code base or from user's customization