Skip to content

The priority queue panic when refreshing the last analysis time #57648

@0xPoe

Description

@0xPoe

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

We found this problem on a very large cluster with a heavy workload.

2. What did you expect to see? (Required)

No panic

3. What did you see instead (Required)

tidb-2024-11-22T11-43-27.742.log:[2024/11/22 10:00:45.192 +00:00] [ERROR] [queue.go:302] ["Priority queue panicked"] [category=stats] [recover="runtime error: invalid memory address or nil pointer dereference"] [stack="github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*AnalysisPriorityQueue).run.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/queue.go:302\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:785\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:262\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:917\ngithub.com/pingcap/tidb/pkg/statistics.(*Table).IsAnalyzed\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/table.go:720\ngithub.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*AnalysisJobFactory).FindLastAnalyzeTime\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/analysis_job_factory.go:209\ngithub.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*AnalysisJobFactory).GetTableLastAnalyzeDuration\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/analysis_job_factory.go:197\ngithub.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*AnalysisPriorityQueue).RefreshLastAnalysisDuration.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/queue.go:656\ngithub.com/pingcap/tidb/pkg/statistics/handle/util.WrapTxn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/util/util.go:206\ngithub.com/pingcap/tidb/pkg/statistics/handle/util.CallWithSCtx\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/util/util.go:104\ngithub.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*AnalysisPriorityQueue).RefreshLastAnalysisDuration\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/queue.go:624\ngithub.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*AnalysisPriorityQueue).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/queue.go:323\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/wait_group_wrapper.go:157"]
tidb-2024-11-22T11-43-27.742.log:[2024/11/22 10:04:17.666 +00:00] [INFO] [cache.go:119] ["reset the old infoschema after v1 v2 switch, using the stale object will panic"]
tidb-2024-11-22T11-43-27.742.log:[2024/11/22 10:20:00.976 +00:00] [ERROR] [queue.go:302] ["Priority queue panicked"] [category=stats] [recover="runtime error: invalid memory address or nil pointer dereference"] [stack="github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*AnalysisPriorityQueue).run.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/queue.go:302\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:785\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:262\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:917\ngithub.com/pingcap/tidb/pkg/statistics.(*Table).IsAnalyzed\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/table.go:720\ngithub.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*AnalysisJobFactory).FindLastAnalyzeTime\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/analysis_job_factory.go:209\ngithub.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*AnalysisJobFactory).GetTableLastAnalyzeDuration\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/analysis_job_factory.go:197\ngithub.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*AnalysisPriorityQueue).RefreshLastAnalysisDuration.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/queue.go:656\ngithub.com/pingcap/tidb/pkg/statistics/handle/util.WrapTxn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/util/util.go:206\ngithub.com/pingcap/tidb/pkg/statistics/handle/util.CallWithSCtx\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/util/util.go:104\ngithub.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*AnalysisPriorityQueue).RefreshLastAnalysisDuration\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/queue.go:624\ngithub.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*AnalysisPriorityQueue).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/queue.go:323\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/wait_group_wrapper.go:157"]
tidb.log:[2024/11/23 03:03:22.126 +00:00] [ERROR] [worker.go:87] ["Auto analyze job execution failed"] [category=stats] [recover="assignment to entry in nil map"] [stack="github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/refresher.(*worker).SubmitJob.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/refresher/worker.go:87\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).RunWithRecover.func1.1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/wait_group_wrapper.go:185\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:785\nruntime.mapassign_fast64\n\t/usr/local/go/src/runtime/map_fast64.go:115\ngithub.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*AnalysisPriorityQueue).Pop.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/queue.go:763\ngithub.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*DynamicPartitionedTableAnalysisJob).Analyze.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/dynamic_partitioned_table_analysis_job.go:110\ngithub.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue.(*DynamicPartitionedTableAnalysisJob).Analyze\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/priorityqueue/dynamic_partitioned_table_analysis_job.go:115\ngithub.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/refresher.(*worker).processJob\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/refresher/worker.go:102\ngithub.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/refresher.(*worker).SubmitJob.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/statistics/handle/autoanalyze/refresher/worker.go:83\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).RunWithRecover.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/wait_group_wrapper.go:189"]

4. What is your TiDB version? (Required)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions