TiKV does not scale beyond 70,000 tpmC in 8-node on-prem cluster

[mamiyaakira0804]

TiKV scalability issue in on-premises environment

Hi TiKV team,

We're currently evaluating TiKV in an on-premises environment and are encountering issues with scaling under load. I'd like to ask for your insights regarding this behavior.

---

📌 Test Environment

• TiDB version: v8.5.0
• Deployment: On-premises (virtual machines)
• Cluster topology:
  • PD: 3 nodes
  • TiDB: 3 nodes
  • TiKV: 8 nodes
• VM specifications:
  • TiKV: 16 vCPU / 64GB RAM
  • TiDB: 8 vCPU / 32GB RAM
  • PD: 4 vCPU / 16GB RAM
• Benchmark tool: go-tpc (TPC-C workload)
• Warehouse count: 10,000

---

🔬 Observed Behavior

• When running the TPC-C workload with increasing thread and client counts, tpmC performance reaches a ceiling at around 70,000, and does not improve further.
• TiKV nodes show only around 20% CPU utilization, even under heavy load.
• No obvious bottlenecks have been observed in disk I/O or network throughput.

---

❓ Questions

1. What could be potential causes for this early saturation in TiKV throughput?
2. Is TiKV expected to scale linearly with node count, or are there limitations due to Region count, schedulers, or internal coordination?
3. Could the low CPU utilization be related to thread pool configuration, NUMA effects, or other system-level scheduling issues?

---

Any advice or guidance on how to further investigate or resolve this would be greatly appreciated.

Thank you in advance!