unstable bank test

[CharlesCheung96]

Which jobs are flaking?

cdc-integration-mysql-test

Which test(s) are flaking?

bank test

Jenkins logs or GitHub Actions link

[2024/11/27 10:41:24.960 +08:00] [ERROR] [case.go:266] ["bank test verify failed"] [error="verify balance failed, accounts2 expect 100000, but got 100056, endTs: 2024-11-27 10:41:15"] [errorVerbose="verify balance failed, accounts2 expect 100000, but got 100056, endTs: 2024-11-27 10:41:15\nmain.(*bankTest).verify.func1\n\t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/tests/integration_tests/bank/case.go:290\[ngithub.com/pingcap/tiflow/pkg/retry.run\n\t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/pkg/retry/retry_with_opt.go:34\nmain.(*bankTest).verify\n\t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/tests/integration_tests/bank/case.go:262\nmain.run.func2\n\t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/tests/integration_tests/bank/case.go:594\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594](http://ngithub.com/pingcap/tiflow/pkg/retry.run/n/t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/pkg/retry/retry_with_opt.go:57/ngithub.com/pingcap/tiflow/pkg/retry.Do/n/t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/pkg/retry/retry_with_opt.go:34/nmain.(*bankTest).verify/n/t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/tests/integration_tests/bank/case.go:262/nmain.run.func2/n/t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/tests/integration_tests/bank/case.go:594/ngolang.org/x/sync/errgroup.(*Group).Go.func1/n/t/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78/nruntime.goexit/n/t/usr/local/go/src/runtime/asm_amd64.s:1594)"] [stack="main.(*bankTest).verify.func1.1\n\t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/tests/integration_tests/bank/case.go:266\nmain.(*bankTest).verify.func1\n\t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/tests/integration_tests/bank/case.go:290\[ngithub.com/pingcap/tiflow/pkg/retry.run\n\t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/pkg/retry/retry_with_opt.go:34\nmain.(*bankTest).verify\n\t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/tests/integration_tests/bank/case.go:262\nmain.run.func2\n\t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/tests/integration_tests/bank/case.go:594\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78](http://ngithub.com/pingcap/tiflow/pkg/retry.run/n/t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/pkg/retry/retry_with_opt.go:57/ngithub.com/pingcap/tiflow/pkg/retry.Do/n/t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/pkg/retry/retry_with_opt.go:34/nmain.(*bankTest).verify/n/t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/tests/integration_tests/bank/case.go:262/nmain.run.func2/n/t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/tests/integration_tests/bank/case.go:594/ngolang.org/x/sync/errgroup.(*Group).Go.func1/n/t/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78)"]
[2024/11/27 10:41:24.961 +08:00] [PANIC] [case.go:595] ["downstream verify failed"] [error="[CDC:ErrReachMaxTry]reach maximum try: 10, error: verify balance failed, accounts2 expect 100000, but got 100056, endTs: 2024-11-27 10:41:15: verify balance failed, accounts2 expect 100000, but got 100056, endTs: 2024-11-27 10:41:15"] [stack="main.run.func2\n\t/home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/tests/integration_tests/bank/case.go:595\[ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78](http://ngolang.org/x/sync/errgroup.(*Group).Go.func1/n/t/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78)"]
panic: downstream verify failed

goroutine 109 [running]:
[go.uber.org/zap/zapcore.CheckWriteAction.OnWrite](http://go.uber.org/zap/zapcore.CheckWriteAction.OnWrite)(0x1?, 0x1?, {0x0?, 0x0?, 0xc00020db60?})
        /go/pkg/mod/[go.uber.org/zap@v1.26.0/zapcore/entry.go:196](http://go.uber.org/zap@v1.26.0/zapcore/entry.go:196) +0x65
[go.uber.org/zap/zapcore.(*CheckedEntry).Write](http://go.uber.org/zap/zapcore.(*CheckedEntry).Write)(0xc00031c0d0, {0xc000686440, 0x1, 0x1})
        /go/pkg/mod/[go.uber.org/zap@v1.26.0/zapcore/entry.go:262](http://go.uber.org/zap@v1.26.0/zapcore/entry.go:262) +0x3ec
[go.uber.org/zap.(*Logger).Panic](http://go.uber.org/zap.(*Logger).Panic)(0xc000686440?, {0x916acc?, 0xc000032180?}, {0xc000686440, 0x1, 0x1})
        /go/pkg/mod/[go.uber.org/zap@v1.26.0/logger.go:284](http://go.uber.org/zap@v1.26.0/logger.go:284) +0x59
[github.com/pingcap/log.Panic](http://github.com/pingcap/log.Panic)({0x916acc?, 0x13?}, {0xc000686440?, 0xa0da08?, 0xc0007e0de0?})
        /go/pkg/mod/[github.com/pingcap/log@v1.1.1-0.20230317032135-a0d097d16e22/global.go:54](http://github.com/pingcap/log@v1.1.1-0.20230317032135-a0d097d16e22/global.go:54) +0x8b
main.run.func2()
        /home/jenkins/agent/workspace/pingcap/tiflow/release-6.5/pull_cdc_integration_test/tiflow/tests/integration_tests/bank/case.go:595 +0xa85
[golang.org/x/sync/errgroup.(*Group).Go.func1()](http://golang.org/x/sync/errgroup.(*Group).Go.func1())
        /go/pkg/mod/[golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78](http://golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78) +0x64
created by [golang.org/x/sync/errgroup.(*Group).Go](http://golang.org/x/sync/errgroup.(*Group).Go)
        /go/pkg/mod/[golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75](http://golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75) +0xa5
exit status 2
+ stop_tidb_cluster
script returned exit code 1

Anything else we need to know

• Does this test exist for other branches as well?

• Has there been a high frequency of failure lately?