Skip to content

tidb restart failed cause by panic #58378

@crazycs520

Description

@crazycs520

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

# create a tidb cluster
tiup cluster deploy csdb nightly topo.yaml -y
tiup cluster start csdb -R tidb
# use br to load data
tiup br restore db --checksum=false --db=oltp_12x1e7 --pd=http://10.202.0.41:2379/ --storage=gs://oltp-bench-us-east5/dataset/v8.1.0/oltp_12x1e7
# restart tidb cluster
tiup cluster restart csdb  # tidb restart failed cause by panic
Error: failed to start: failed to start tidb: failed to start: 10.202.0.38 tidb-4000.service, please check the instance's log(/data/deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s

Related tidb_stderr.log log:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0x48b880b]

goroutine 713 [running]:
[github.com/pingcap/tidb/pkg/statistics/handle.(*Handle).initStatsHistograms4Chunk](http://github.com/pingcap/tidb/pkg/statistics/handle.(*Handle).initStatsHistograms4Chunk)(0xc00179a400, {0x7148938, 0xc001187c28}, {0x712ee10, 0xc000015590}, 0xc003a2bd88, 0x0)
        /workspace/source/tidb/pkg/statistics/handle/bootstrap.go:233 +0x4ab
[github.com/pingcap/tidb/pkg/statistics/handle.(*Handle).initStatsHistogramsByPaging](http://github.com/pingcap/tidb/pkg/statistics/handle.(*Handle).initStatsHistogramsByPaging)(0xc00179a400, {0x7148938, 0xc001187c28}, {0x712ee10, 0xc000015590}, {0xc003468100?, 0xc003a1d6a8?}, 0xf92af2000)
        /workspace/source/tidb/pkg/statistics/handle/bootstrap.go:360 +0x425
[github.com/pingcap/tidb/pkg/statistics/handle.(*Handle).initStatsHistogramsConcurrency.func1](http://github.com/pingcap/tidb/pkg/statistics/handle.(*Handle).initStatsHistogramsConcurrency.func1)({0x0?, 0x0?})
        /workspace/source/tidb/pkg/statistics/handle/bootstrap.go:369 +0x37
[github.com/pingcap/tidb/pkg/statistics/handle/initstats.(*RangeWorker).loadStats(0xc00342b1a0)](http://github.com/pingcap/tidb/pkg/statistics/handle/initstats.(*RangeWorker).loadStats(0xc00342b1a0))
        /workspace/source/tidb/pkg/statistics/handle/initstats/load_stats_page.go:91 +0x70
[github.com/pingcap/tidb/pkg/statistics/handle/initstats.(*RangeWorker).LoadStats.func1()](http://github.com/pingcap/tidb/pkg/statistics/handle/initstats.(*RangeWorker).LoadStats.func1())
        /workspace/source/tidb/pkg/statistics/handle/initstats/load_stats_page.go:84 +0x17
[github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1()](http://github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1())
        /workspace/source/tidb/pkg/util/wait_group_wrapper.go:157 +0x4c
created by [github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run](http://github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run) in goroutine 647
        /workspace/source/tidb/pkg/util/wait_group_wrapper.go:155 +0x73

2. What did you expect to see? (Required)

cluster restart success.

3. What did you see instead (Required)

Error: failed to start: failed to start tidb: failed to start: 10.202.0.38 tidb-4000.service, please check the instance's log(/data/deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s

4. What is your TiDB version? (Required)

> select tidb_version();
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version()                                                                                                                                                                                                                                                     |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v8.5.0-alpha-373-g177a03c8e5
Edition: Community
Git Commit Hash: 177a03c8e51caf844491d8a93edda29295510e8d
Git Branch: HEAD
UTC Build Time: 2024-12-18 05:08:08
GoVersion: go1.23.4
Race Enabled: false
Check Table Before Drop: false
Store: tikv |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions