cache: split filter and index cache when sharing cache

[galaio]

Hi guys, while using Pebble, we've discovered that sharing the cache across multiple instances can be a better solution. However, we're based on Pebble 1.0, and the current pull request is a proof-of-concept. If you agree on the changes, I'll continue optimizing it to meet the merge requirements.

In our scenario, we've been using a single database, accumulating around 1.5TB of data. To reduce the pressure on the single database, we've used parallel commits and other methods. We plan to run on shards in the future, so we'll split the single database into eight shards, which will of course reduce the cache size and handles proportionally.

Although parallel commits work well, read performance and cache hit rate have decreased slightly. We've tried using a shared cache across eight shards.

Version 1:

The performance has deteriorated significantly. The flame graph shows a significant drop in the cache hit rate for readFilter. Analysis suggests this may be due to frequent evicts of the filter cache when using a shared cache. see #5242

Version 2:

We forked the Pebble code to isolate the filter and index caches. The performance difference is minimal compared to a single-DB cache.

Thank you for your review.

[cockroach-teamcity]

This change is 

[RaduBerinde]

Thank you! These findings are interesting. I am reluctant of going down the separate caches route, as that will involve a non-trivial sizing decision. It is worth investigating why the cache policy didn't appropriately keep the filter blocks in the cache (we may benefit from a better replacement algorithm, or perhaps we need to somehow make certain misses more expensive than others).

[RaduBerinde]

Do you by chance have a program that reproduces this behavior which you can share (so we can investigate more)?

[RaduBerinde]

It would also be useful to see if you can repro the same improvement with a more recent Pebble (preferably 2.1)