Data and loads are not well distributed in my benchmark

[jaltabike]

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   If possible, provide a recipe for reproducing the error.

I tried to insert tuples into TiDB by using YCSB (Yahoo! Cloud Serving Benchmark).
My TiDB settings were as follows:

• I used 9 AWS EC2 instances (each instance is i3.2xlarge: 8 vCPUs, 61GiB RAM , 1 x 1900GB nvme disk)
  . node#1~store: fix boltdb package import. #3: 1 x TiDB & 1 x PD ware deployed on each instance (i.e., total 3 x TiDBs & 3 x PDs)
  . node#4~Update README.md #9: 1 x TiKV was deployed on each instance (i.e., total 6 x TiKVs)
• tidb_version: v2.0.5
• I used the default TiDB config. Exception is as follows:
  . prepared_plan_cache: enabled: true
• I used the default PD config.
• I used the default TiKV config. Exceptions are as follows:
  . raftstore: sync-log: false
  . rocksdb: bytes-per-sync: "1MB"
  . rocksdb: wal-bytes-per-sync: "512KB"
  . raftdb: bytes-per-sync: "1MB"
  . raftdb: wal-bytes-per-sync: "512KB"
  . storage: scheduler-concurrency: 1024000
  . storage: scheduler-worker-pool-size: 6
  . *: max-write-buffer-number: 10
• I created one table like follows:
  CREATE TABLE usertable (
  YCSB_KEY VARCHAR(255) PRIMARY KEY,
  FIELD0 TEXT, FIELD1 TEXT,
  FIELD2 TEXT, FIELD3 TEXT,
  FIELD4 TEXT, FIELD5 TEXT,
  FIELD6 TEXT, FIELD7 TEXT,
  FIELD8 TEXT, FIELD9 TEXT
  );

And YCSB settings were as follows:

• I used YCSB "load" command to insert tuples into TiDB
• I used the following config for YCSB "load"
  . maxexecutiontime: 3600 (i.e., insert tuples for 3600 seconds)
  . target: 32000 (i.e., insert tuples at a rate 32000 insertions per sec)
  . threads: 512 (i.e., use 512 client threads to insert tuples)
  . db.batchsize: 100 & jdbc.autocommit: false (i.e., commit every 100 inserts)

2. What did you expect to see?
   I expected the data and loads to be well distributed.

3. What did you see instead?
   Data and loads were not well distributed. I think the performance (TPS and latency) was limited by this.

Data were not well distributed as follows:

• Sore size => store size of tikv_1 and tikv7 was 3 times larger than that of the others
  
• Leader and region distribution => It was unbalanced
  
  

Loads for TiKV were not well distributed as follows:
(In overall, tikv1 seemed to be overloaded.)

• CPU => tikv_1 (also tikv_7) used about 200% (100%) more CPU than other tikvs
  
• Load => load received by tikv1(=tikv_1) was twice more than that of the others
  
• IO Util => IO util of tikv1(=tikv_1) and tikv5(=tikv_7) was six times (twice for tikv5) more than that of the others
  
• Network traffic Inbound => Inbound of tikv1 and tikv5 was twice more than that of the others
  
• Network traffic Outbound => Outbound of tikv1 was very larger than that of the others
  
• Region average written keys => most of keys seemd to be written into tikv_1
  
• Scheduler pending commands => most of scheduler pending commands were issued from tikv_1
  

4. Questions
   Why were data not well-distributed in my situation?
   Why were loads (CPU, IO, Network) not well-distributed in my situation?
   What is "scheduler pending commands"? Is it related to unbalanced load or data?
   Can the region size exceed region-max-size? Following graph shows average region size was 20.3GiB which is too big! It seems that there was one very large size region. If there was such a region, why did not this large size region split?
   

5. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?
   v2.0.5

[XuHuaiyu]

Hi, @jaltabike Thank you for your feedback.
We'll reply to you soon.

[XuHuaiyu]

PTAL @liukun4515

[liukun4515]

Can you provide more detailed ycsb workload file and load script?
@jaltabike

[jaltabike]

@liukun4515
My YCSB properties(configs) are as follows:

• maxexecutiontime=3600
• recordcount=1000000000
• operationcount=1000000000
• workload=com.yahoo.ycsb.workloads.CoreWorkload
• readallfields=true
• readproportion=0
• updateproportion=0
• scanproportion=0
• insertproportion=1
• requestdistribution=zipfian
• insertstart=0
• fieldcount=10
• fieldlength=100
• reportlatencyforeacherror=true
• db.driver=com.mysql.jdbc.Driver
• db.url=jdbc:mysql://tidb-proxy:4001/ycsb?useSSL=false&useServerPrepStmts=true
• db.user=???
• db.passwd=???
• db.batchsize=100
• jdbc.batchupdateapi=false
• jdbc.autocommit=false

And, my load script is as follows:
ycsb -jvm-args="${JVM_ARGS}" load jdbc -P ${PROPERTY_FILE} -s -target 32000 -threads 512
where

• JVM_ARGS="-Xms8G -Xmx8G -XX:PermSize=512m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSWaitDuration=10000 -XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways -XX:+CMSClassUnloadingEnabled"
• PROPERTY_FILE contains properties I listed above.

*** My first comment contained incorrect information, so I've corrected the information: I've changed the "target" value of the YCSB config from 20000 to 32000.

[jaltabike]

@liukun4515
I performed another benchmark.

All the configs of TiDB and YCSB were identical to that of the above benchmark except:

• (for YCSB) maxexecutiontime = 0 (i.e., the benchmark was finished when the predefined number of tuples (specified by recordcount (=1000000000)) were inserted.
• (for YCSB) target = 20000 (i.e., insert tuples at a rate 20000 insertions per sec)

As the result of this benchmark, the trends were similar to that of the above benchmark's result but more severe.

• Data were not well distributed as follows:
  
  
  

• Loads for TiKV were not well distributed as follows:
  
  
  
  
  
  
  

• Approximate region size continued to increase; I suspect that split worked properly.
  

(Maybe, due to the above result) the performance of the TiDB benchmark was worse than expected, as follows:
(*from 22:00, the performance was very unstable.)

Please let me know why this is happening.

Thank you.

[liukun4515]

This situation is caused by hot region.
You can refer to _tidb_rowid and SHARD_ROW_ID_BITS in this doc
@jaltabike

[jaltabike]

Thank you, @liukun4515
I will change the value of SHARD_ROW_ID_BITS and run the ycsb benchmark again.
By the way, can the region size exceed region-max-size?
"approximate region size", the graph I attached above, shows that average region size is 20.3GiB which is far larger than region-max-size (default: 144MB).
Why do not regions whose size is far larger than region-max-size be splitted?

[liukun4515]

@jaltabike
The region size may exceed the region-max-size.
The range split is based on the scanning data in this region, it will cost some time.
When most of the data load into the hot region, the speed of splitting region can't catch up that of loading data, the region will become larger.
You can set the SHARD_ROW_ID_BITS and the data will be written into multiple different regions.
The split region by keys will speed up the split without scanning the data.

[liukun4515]

Any update for your benchmark result?

[dbjoa]

@liukun4515 Can TiDB split the region automatically if the user changes SHARD_ROW_ID_BITS via ALTER TABLE?

[liukun4515]

The value of SHARD_ROW_ID_BITS doesn't affect the region split. @dbjoa

[jaltabike]

Thank you for your answer, @liukun4515
I currently can not access my workspace until next Tuesday (Aug. 21st).
I will run the benchmark next Wednesday (Aug. 22nd), and I will share the results as soon as possible.

[jaltabike]

@liukun4515 @XuHuaiyu
I would like to share my new benchmark results.

I just set the value of SHARD_ROW_ID_BITS to 4,

and performed the same benchmark as the first one mentioned above.

As the result, it seemed that data and loads were well-distributed as follows:

• Data were well-distributed:
  
  
  

• Loads for TiKV were relatively well-distributed:
  
  
  
  
  
  (*For outbound network traffic, tikv2 was twice as much as the others.)
  
  (*For "region average written keys", tikv_4(=tikv2) was up to twice as much as the others.)
  

• Region size seemed normal:
  

Now, i'm performing the same benchmark as the second one mentioned above, which will take about a day.

I'll also share the results as soon as possible.

By the way, why might "setting a large value of SHARD_ROW_ID_BITS" lead to a large number of RPC requests?

And, what does "scheduler pending commands" means?

[winoros]

@jaltabike
When there's no int primary key and you performed a range query on table's primary key and try to get data that not only containing index key but also other columns, TiDB will first scan index and get its corresponding __tidb_rowid then use the __tidb_rowid to seek the table data. You can explain the sql to see its execution plan.
SHARD_ROW_ID_BITS won't affect the process of scanning index, but will affect the process of scanning table.
Table's data is stored in __tidb_rowid's order. So when there's no SHARD_ROW_ID_BITS, the continuous index values' corresponding __tidb_rowid is more likely stored in same region. But it will stored in multiple region to avoid hot spot when there's SHARD_ROW_ID_BITS.
One RPC task sending to tikv will only contain one region. So the more region one query involved, the more RPC request is sent.
Thus the larger SHARD_ROW_ID_BITS is, the more RPC requests are sent.