Skip to content

Commit 5c55a35

Browse files
sollhuidataroaring
authored andcommitted
[fix](multi table) fix single stream multi table memory leak (#38255)
We meet OOM when using single stream multi table ![image](https://github.com/user-attachments/assets/748e9914-d591-4f41-8b28-412d3cecc841) It exist memory leak, and heap profile like: ![image](https://github.com/user-attachments/assets/af30c593-88ea-44f6-bba1-82436b13f99f) The stream load context will not release in some exception conditions as plan failed for high concurrency causing timeout when obtaining read lock. It is introduced by #35458 The solution effect is shown in the following figure, which can run stably with a small amount of memory ![image](https://github.com/user-attachments/assets/4483e0a5-6c0c-4cdc-b8ed-3408da6a86b2)
1 parent 70c308c commit 5c55a35

File tree

1 file changed

+13
-0
lines changed

1 file changed

+13
-0
lines changed

be/src/io/fs/multi_table_pipe.cpp

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -326,6 +326,19 @@ void MultiTablePipe::_handle_consumer_finished() {
326326
_ctx->number_filtered_rows = _number_filtered_rows;
327327
_ctx->number_unselected_rows = _number_unselected_rows;
328328
_ctx->commit_infos = _tablet_commit_infos;
329+
330+
// remove ctx to avoid memory leak.
331+
for (const auto& pair : _planned_tables) {
332+
if (pair.second) {
333+
doris::ExecEnv::GetInstance()->new_load_stream_mgr()->remove(pair.second->id);
334+
}
335+
}
336+
for (const auto& pair : _unplanned_tables) {
337+
if (pair.second) {
338+
doris::ExecEnv::GetInstance()->new_load_stream_mgr()->remove(pair.second->id);
339+
}
340+
}
341+
329342
LOG(INFO) << "all plan for multi-table load complete. number_total_rows="
330343
<< _ctx->number_total_rows << " number_loaded_rows=" << _ctx->number_loaded_rows
331344
<< " number_filtered_rows=" << _ctx->number_filtered_rows

0 commit comments

Comments
 (0)