improve tidb go runtime metrics

[XuHuaiyu]

Enhancement

• Background
  Currently, TiDB's monitoring related to the Golang runtime lacks sufficient detail to effectively identify runtime-related issues. To address this, we need to enhance the existing Grafana monitoring panels and introduce additional monitoring metrics focused on the Golang runtime.
• Objectives
  The primary goal is to refine and expand the TiDB monitoring capabilities to provide more insightful data, which will aid in diagnosing runtime-related problems. The improvements will be implemented by optimizing the existing monitoring panels and adding new ones to cover critical runtime aspects.
• Tasks
• Enhance the readability and clarity of the tidb-runtime -> memory usage monitoring panel. pkg/metrics: improve the TiDB-Runtime->Memory Usage panel #55391
• Introduce a new panel to monitor CPU time of runtime jobs. pkg/metrics: add a panel for estimated portion of cpu time #55401
• Refine the GC STW monitoring panel to provide more actionable insights. pkg/metrics: introduce a more detailed metrics for Go STW Latency #55982
• Implement new panels to monitor scheduler activity and mutex contention, offering better visibility into potential bottlenecks. pkg/metrics: add a panel for sync mutex wait #56022
• Introduce new panels for golang gc rate and GOGC/GOMEMLIMIT. pkg/metrics: add panels for golang gc rate and GOGC/GOMEMLIMIT #56033 pkg/metrics: make the golang gc mpanel more readable #56039
• Introduce new panels for heap alloc and heap free. pkg/metrics: add panels for heap alloc and heap free #56050