bad_alloc + segfault with 10k partitions

[grsmith-projects]

Version & Environment

Redpanda version: 21.11.5
OS: Kubernetes using the :latest tag
Using default config options from the custom config example
only change was storage_read_buffer_size set to 32768
Kernel: Linux 4.19.0-0.bpo.5-amd64 #1 SMP Debian 4.19.37-4~bpo9+1 (2019-06-19) x86_64 GNU/Linux

we have 6 fairly beefy nodes with 384 CPU + 24 SSD (2TB)

each pod (24) has 40G of ram allocated, with 32G for RedPanda itself.

What went wrong?

ERROR 2022-02-15 03:31:53,794 [shard 0] seastar_memory - Dumping seastar memory diagnostics Used memory: 2G Free memory: 0B Total memory: 2G

libc++abi: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc Aborting on shard 0.

Segmentation fault on shard 0.

What should have happened instead?

We were looking to scale to 25k partitions, we were able to scal

How to reproduce the issue?

1. create cluster with size described above
2. once confirmed healthy start running a load test against it [using custom tools, with various sized payloads]
3. the LT will hit one topic
4. change topic to 10,000 partitions
5. witness crashing

[jcsp]

Thanks for opening this, can you tell us a bit more about the client workload to help us craft a reproducer? Interesting dimensions are:

• How many consumers, how many producers?
• Are producers all in the same consumer group (i.e. sharing out partitions across consumers), or are there consumers that are reading the whole topic from one client?
• Are clients using transactions?
• What's the batch size of clients? (or what's the largest batch size if clients are using a mixture)

[ebbydw]

We are using confluent-kafka (1.7.0)

1. 100 consumers processes trying to join with group.instance.id set, this seems like it's not supported yet so we end up with rebalance in progress (I'll open a separate issue for this).
2. Sorry I don't understand this question - We have close to 48000 confluent_kafka processes in k8 pods sending messages to a topic with 10k partitions(~250k req/sec), the consumers have not been setup up yet (see 1).
3. We are not using transactions.
4. We are trying to optimize for a low latency use case
   (We didn't enable compression)
   linger.ms=0
   request.required.acks=-1
   message.send.max.retires=3

FWIW we also tried to set a smaller retention period for the topic (10 minutes) to test how it worked, and it was the followed by the crash as reported. (Update: we repeated the loadtest today and setting retention worked fine, it only crashes when we increased the partitions to 10k)

[grsmith-projects]

Some added info

we are seeing high I/O wait on all of the nodes up to 40% -- it looks like CPU wise we are hitting some limits at 7k partitions

01:37:52 up 5 days, 5:08, 1 user, load average: 3027.30, 2891.06, 2844.81

sde            1438.56       105.57      4199.17   47566409 1892038532
sdf            1345.51        69.90      3920.07   31494798 1766284440
sdb               0.00         0.00         0.00       2164          0
sdc            1376.51       309.26      4155.24  139343628 1872245708

I am wondering if we are just under provisioned for using SSDs instead of NVME?

[grsmith-projects]

We also saw this just before the crash..

Reactor stalled for 263 ms on shard 13. Backtrace: 0x3878c39 0x387aa79 0x2a07995ce9ff 0x33b368b 0x33a6f1f 0x326d2cb 0x38169dc 0x381ca43 0x382b681 0x1a71cf7 0x1a77880 0x1a82828 0x196aaaa 0x389e244 0x38a1707 0x38f154d 0x384f22f 0x9298 0x1006a2

the cluster was working fine at 8000 partitions, but under high load -- as soon as we swapped to 10k, it flatlined.

[dotnwat]

Thanks @grsmith-projects we are still looking at this.

[grsmith-projects]

interesting thing:

under very minimal load <-> very high load when adding 20k partitions is quite quick and cluster remains healthy

under very minimal load <-> very high load when adding 22k partitions the cluster begins to fail

it seems independent of request rate and is specific to partition count only how we are causing the cluster to fail.

Also, it seems to scale with hardware

6 nodes = (80 core / 384G ram) = 10k partitions
18 nodes = (80 core / 384G ram) = 20k partitions

we are running 72 pods in kube (16 core, 32G ram), each one has its own dedicated SSD

so total test cluster size is 1152 CPU / 2304G Ram / 72 x 2TB SSD

it seems to handle traffic very well even with the higher partition count - if we're on the edge of failure, our application fails before RP, but the moment there is 1 too many partitions the cluster will fail with the errors above.

[travisdowns]

Thanks for all the observations so far, @grsmith-projects. I'll be looking at this issue and hopefully we'll get to the partition counts you are looking for.

A few questions:

I think your sentence got truncated in the original report:

we have 6 fairly beefy nodes with 384 CPU + 24 SSD (2TB)

each pod (24) has 40G of ram allocated, with 32G for RedPanda itself.

To clarify the setup, is that 24 pods, with 384/24 = 16 cores for each pod?

So we can expect 2GB (32G for redpanda, per pod / 16 cores) per shard (core) for redpanda.

Each pod has a dedicated SSD?

Each node supports 4 pods and the nodes have 64 cores?

under very minimal load <-> very high load when adding 20k partitions is quite quick and cluster remains healthy

under very minimal load <-> very high load when adding 22k partitions the cluster begins to fail

To clarify, the <-> here means everywhere from minimal to high load, right? That is, at 20k with high load, the cluster is healthy, while at 22k even with only minimal load, there are failures, right?

This behavior isn't entirely surprising since it's not a "soft" resource overload like CPU where you'd expect degraded performance, but an out of memory scenario which is more like a hard wall: it either works or it doesn't.

[grsmith-projects]

haha.. so it does seem that it did get truncated.

1. There are now 1152 CPUs
2. I've increased the memory to nearly 4GB per core, there is more than 2GB per core now
3. Each pod had a dedicated SSD
4. Each node supports 4 pods, and the nodes have 80 cores
5. <-> does mean that! That is correct.. however in version .8 it looks like this is fixed, we were able to scale to nearly 70k partitions with a minimal request rate.. once we ramped up though, the cluster failed.

[travisdowns]

Haha, the truncated sentence was actually:

We were looking to scale to 25k partitions, we were able to scal

but I neglected to paste that. I think you were going say "... we were only able to scale to 10k" or something like that?

Thanks for the updated note, I'm continuing to look at this and will reach out soon.

[emaxerrno]

curious if it is some sort of delta-delta soa issue: https://github.com/redpanda-data/redpanda/blob/dev/src/v/raft/types.cc#L335-L403 - i tested this in the 100's of K and it yielded OK perf. Maybe there is a growth vector we didn't take into account and the payload takes longer to heartbeat than the next round which would vote-storm