Ligthtning : Increase io size(BlockSize) of pebble DB

[AricSu]

Enhancement

Well, recently, I got a question that's if we have any option to increase io size of pebble DB.

1. I found here is the only area changing something about LevelOptions. So, untill now, Tidb-lightning hasn't had a way to control it.

2. What the user want is when they are hiting the IOPS limit of the EBS drives in read data from pebble into SSTs of TiKV. In this way, "Pebble could relatively read more data at once to reduce iops"

type LevelOptions struct {
// BlockRestartInterval is the number of keys between restart points
// for delta encoding of keys.
//
// The default value is 16.
BlockRestartInterval int

// BlockSize is the target uncompressed size in bytes of each table block.
//
// The default value is 4096.
BlockSize int

more info --> https://pingcap.zendesk.com/agent/tickets/1896

[mittalrishabh]

I am trying changing the lightning code and see if it improves the performance.

[lance6716]

It's a good suggestion! Increasing the block size may have other effects such as higher memory consumption, we will think about it thoroughly.

[mittalrishabh]

Hi, I have done the experiments internally by making the code changes. Performance improved 2X times. Bottleneck moved from network to disk. Will it be ok to you if i make the changes and upstream it. I will add it as an option controlled through configuration.toml file

[D3Hunter]

@mittalrishabh can you upload logs before/after change the code? and what's the iops/throughput config of lighting EBS?

and can you share info about your data set:

• total size
• table column count, index count
• average row length

[mittalrishabh]

I can run the experiment again to collect the logs. I will share the document with you for other details

[lance6716]

Hi, I have done the experiments internally by making the code changes. Performance improved 2X times. Bottleneck moved from network to disk. Will it be ok to you if i make the changes and upstream it. I will add it as an option controlled through configuration.toml file

Hi, we think we can add a configuration item like

[tikv-importer]
# block-size controls the local sorting file IO lock size in physical import (local backend). You can increase this value when IOPS  is a bottleneck.
block-size = '16KiB'