[enhancement](cloud) refine block file cache evict policy #42451
Conversation
Signed-off-by: freemandealer <[email protected]>
Signed-off-by: freemandealer <[email protected]>
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
TPC-H: Total hot run time: 41625 ms |
TPC-DS: Total hot run time: 192350 ms |
ClickBench: Total hot run time: 32.8 s |
freemandealer
changed the title
|
run p0 |
1 similar comment
|
run p0 |
Signed-off-by: freemandealer <[email protected]>
|
clang-tidy review says "All clean, LGTM! 👍" |
|
This PR introduce some behavior change, pls describe what it is |
Signed-off-by: freemandealer <[email protected]>
Signed-off-by: freemandealer <[email protected]>
Signed-off-by: freemandealer <[email protected]>
|
clang-tidy review says "All clean, LGTM! 👍" |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| context.cache_type = io::FileCacheType::TTL; | ||
| context.expiration_time = UnixSeconds() + 3600; | ||
|
|
||
| auto sp = SyncPoint::get_instance(); |
There was a problem hiding this comment.
warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]
| auto sp = SyncPoint::get_instance(); | |
| s);auto * |
| ; | ||
| EXPECT_EQ(cache._cur_cache_size, 30); |
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
| ; | |
| EXPECT_EQ(cache._cur_cache_size, 30); | |
| he) { | |
| ; | |
| } |
| if (fs::exists(cache_base_path)) { | ||
| fs::remove_all(cache_base_path); | ||
| } | ||
| fs::create_directories(cache_base_path); | ||
| auto sp = SyncPoint::get_instance(); |
There was a problem hiding this comment.
warning: 'auto sp' can be declared as 'auto *sp' [readability-qualified-auto]
| auto sp = SyncPoint::get_instance(); | |
| h);auto * |
| @@ -5493,4 +5283,400 @@ | |||
| } | |||
| } | |||
|
|
|||
| TEST_F(BlockFileCacheTest, populate_empty_cache_with_disposable) { | |||
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5285: 103 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, disposable_seize_after_normal) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5390: 115 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, evict_privilege_order) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5507: 172 lines including whitespace and comments (threshold 80)
^
freemandealer
force-pushed
the
evict-policy
branch
from
c10721b to
a91968a
Compare
| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, populate_empty_cache_with_normal) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5390: 102 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, populate_empty_cache_with_index) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5494: 102 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, populate_empty_cache_with_ttl) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5598: 102 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, disposable_seize_after_normal) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5702: 115 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_disposable) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5819: 172 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_normal) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5993: 171 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_index) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:6166: 171 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_ttl) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:6339: 171 lines including whitespace and comments (threshold 80)
^Signed-off-by: freemandealer <[email protected]>
freemandealer
force-pushed
the
evict-policy
branch
from
a91968a to
8e005c2
Compare
| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, seize_after_full) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5819: 155 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_disposable) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5976: 172 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_normal) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:6150: 171 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_index) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:6323: 171 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_ttl) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:6496: 171 lines including whitespace and comments (threshold 80)
^Signed-off-by: freemandealer <[email protected]>
| @@ -5493,4 +5284,1388 @@ TEST_F(BlockFileCacheTest, file_cache_path_storage_parse) { | |||
| } | |||
| } | |||
|
|
|||
| TEST_F(BlockFileCacheTest, populate_empty_cache_with_disposable) { | |||
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5286: 103 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, populate_empty_cache_with_normal) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5391: 102 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, populate_empty_cache_with_index) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5495: 102 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, populate_empty_cache_with_ttl) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5599: 102 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, disposable_seize_after_normal) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5703: 115 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, seize_after_full) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5820: 155 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_disposable) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:5977: 172 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_normal) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:6151: 171 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_index) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:6324: 171 lines including whitespace and comments (threshold 80)
^| } | ||
| } | ||
|
|
||
| TEST_F(BlockFileCacheTest, evict_privilege_order_for_ttl) { |
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
^Additional context
be/test/io/cache/block_file_cache_test.cpp:6497: 171 lines including whitespace and comments (threshold 80)
^
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 41575 ms |
TPC-DS: Total hot run time: 196015 ms |
ClickBench: Total hot run time: 32.62 s |
|
run cloud_p0 |
Signed-off-by: freemandealer <[email protected]>
|
run buildall |
TPC-H: Total hot run time: 41512 ms |
TPC-DS: Total hot run time: 190963 ms |
ClickBench: Total hot run time: 31.68 s |
|
TeamCity be ut coverage result: |
| @@ -86,6 +86,94 @@ BlockFileCache::BlockFileCache(const std::string& cache_base_path, | |||
| _total_evict_size_metrics = std::make_shared<bvar::Adder<size_t>>( | |||
| _cache_base_path.c_str(), "file_cache_total_evict_size"); | |||
|
|
|||
| _evict_by_heat_metrics_matrix[FileCacheType::DISPOSABLE][FileCacheType::NORMAL] = | |||
| std::make_shared<bvar::Adder<size_t>>(_cache_base_path.c_str(), | |||
| "file_cache_evict_by_heat_disposable_to_normal"); | |||
There was a problem hiding this comment.
"file_cache_evict_by_heat" -> "file_cache_evict_by_time"
| @@ -109,14 +197,16 @@ BlockFileCache::BlockFileCache(const std::string& cache_base_path, | |||
| "file_cache_hit_ratio_5m", 0.0); | |||
| _hit_ratio_1h = std::make_shared<bvar::Status<double>>(_cache_base_path.c_str(), | |||
| "file_cache_hit_ratio_1h", 0.0); | |||
| _disk_limit_mode_metrics = | |||
| std::make_shared<bvar::Status<size_t>>(_cache_base_path.c_str(), "disk_limit_mode", 0); | |||
There was a problem hiding this comment.
"disk_limit_mode" -> "file_cache_disk_limit_mode"
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
|
PR approved by anyone and no changes requested. |
- Enhance cache performance and reduce cache misses by:
- Prevent starvation of any particular cache.
- Improve disk space utilization.
- Maintain strict and systematic eviction priorities.
- Enhance the observability of caching strategies, allowing cache
behavior to be intuitively self-explanatory through monitoring.
- Improve code comprehensibility: unify the cache framework to avoid
ad-hoc handling of TTL, increasing the modularity of the code.
- Reduce operational complexity by eliminating and standardizing
configuration items.
---------
Signed-off-by: freemandealer <[email protected]>
…43201) - Enhance cache performance and reduce cache misses by: - Prevent starvation of any particular cache. - Improve disk space utilization. - Maintain strict and systematic eviction priorities. - Enhance the observability of caching strategies, allowing cache behavior to be intuitively self-explanatory through monitoring. - Improve code comprehensibility: unify the cache framework to avoid ad-hoc handling of TTL, increasing the modularity of the code. - Reduce operational complexity by eliminating and standardizing configuration items. Signed-off-by: freemandealer <[email protected]>
dataroaring
added
dev/3.0.3-merged
and removed
cherry-pick-conflict-in-3.0
dev/3.0.x
labels
gavinchou
added
cir
and removed
approved
Uh oh!
There was an error while loading. Please reload this page.