planner: optimize return type of partial sum for tiflash #59709

guo-shaoge · 2025-02-24T08:10:57Z

What problem does this PR solve?

Issue Number: close #59708

Problem Summary:
the type infer of avg(col_decimal_15_2):

In plan builder, call typeInfer4Avg, the return type of avg becomes decimal(19, 6)
In task.go, trying to convert normal avg to mpp avg, which will be spilit to count() and sum(col_decimal_15_2)(1 phase) and sum/count(2 phase). And the return type of 1 phase sum(col_decimal_15_2) is decimal(41, 6) because of calling TypeInfer4AvgSum

decimal(41, 6) will use Int256 to compute, which is slow. So we need to optimize type infer.

What changed and how does it work?

optimize the partial sum of avg for tiflash. So appropriate type will be used to compute sum.
the final avg ret type should not be changed.

Workload: tpch50G
Query: explain analyze select avg(l_quantity) from lineitem group by l_orderkey;
performance: 4-> 2.9 (+26%)
before:

mysql> mysql> explain analyze select avg(l_quantity) avgres from lineitem group by l_orderkey having avgres > 100000;
+--------------------------------------+--------------+-----------+--------------+----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+---------+------+
| id                                   | estRows      | actRows   | task         | access object  | execution info                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | operator info                                                                                                                       | memory  | disk |
+--------------------------------------+--------------+-----------+--------------+----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+---------+------+
| TableReader_65                       | 59126579.20  | 0         | root         |                | time:3.98s, open:1.53ms, close:6.64µs, loops:1, RU:113028.94, cop_task: {num: 2, max: 0s, min: 0s, avg: 0s, p95: 0s, copr_cache_hit_ratio: 0.00}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | MppVersion: 3, data:ExchangeSender_64                                                                                               | 1.50 KB | N/A  |
| └─ExchangeSender_64                  | 59126579.20  | 0         | mpp[tiflash] |                | tiflash_task:{proc max:3.98s, min:3.92s, avg: 3.95s, p80:3.98s, p95:3.98s, iters:0, tasks:2, threads:32}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | ExchangeType: PassThrough                                                                                                           | N/A     | N/A  |
|   └─Selection_63                     | 59126579.20  | 0         | mpp[tiflash] |                | tiflash_task:{proc max:3.98s, min:3.92s, avg: 3.95s, p80:3.98s, p95:3.98s, iters:0, tasks:2, threads:32}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | gt(Column#17, 100000)                                                                                                               | N/A     | N/A  |
|     └─Projection_57                  | 73908224.00  | 75000000  | mpp[tiflash] |                | tiflash_task:{proc max:3.94s, min:3.88s, avg: 3.91s, p80:3.94s, p95:3.94s, iters:4096, tasks:2, threads:32}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | div(Column#17, cast(case(eq(Column#31, 0), 1, Column#31), decimal(20,0) BINARY))->Column#17                                         | N/A     | N/A  |
|       └─HashAgg_58                   | 73908224.00  | 75000000  | mpp[tiflash] |                | tiflash_task:{proc max:2.82s, min:2.81s, avg: 2.82s, p80:2.82s, p95:2.82s, iters:4096, tasks:2, threads:32}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | group by:test.lineitem.l_orderkey, funcs:sum(Column#32)->Column#31, funcs:sum(Column#33)->Column#17, stream_count: 16               | N/A     | N/A  |
|         └─ExchangeReceiver_60        | 73908224.00  | 75000093  | mpp[tiflash] |                | tiflash_task:{proc max:2.19s, min:2.15s, avg: 2.17s, p80:2.19s, p95:2.19s, iters:7226, tasks:2, threads:32}, tiflash_wait: {pipeline_queue_wait: 511ms}, tiflash_network: {inner_zone_receive_bytes: 561417293}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | stream_count: 16                                                                                                                    | N/A     | N/A  |
|           └─ExchangeSender_59        | 73908224.00  | 75000093  | mpp[tiflash] |                | tiflash_task:{proc max:2.61s, min:0s, avg: 1.3s, p80:2.61s, p95:2.61s, iters:1536, tasks:2, threads:144}, tiflash_network: {inner_zone_send_bytes: 561417293}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | ExchangeType: HashPartition, Compression: FAST, Hash Cols: [name: test.lineitem.l_orderkey, collate: binary], stream_count: 16      | N/A     | N/A  |
|             └─HashAgg_55             | 73908224.00  | 75000093  | mpp[tiflash] |                | tiflash_task:{proc max:1.9s, min:0s, avg: 951.2ms, p80:1.9s, p95:1.9s, iters:1536, tasks:2, threads:144}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | group by:test.lineitem.l_orderkey, funcs:count(test.lineitem.l_quantity)->Column#32, funcs:sum(test.lineitem.l_quantity)->Column#33 | N/A     | N/A  |
|               └─TableFullScan_35     | 300005811.00 | 300005811 | mpp[tiflash] | table:lineitem | tiflash_task:{proc max:786.5ms, min:0s, avg: 393.2ms, p80:786.5ms, p95:786.5ms, iters:4757, tasks:2, threads:144}, tiflash_wait: {pipeline_queue_wait: 772ms}, tiflash_scan:{mvcc_input_rows:0, mvcc_input_bytes:0, mvcc_output_rows:0, local_regions:217, remote_regions:0, tot_learner_read:6ms, region_balance:{instance_num: 2, max/min: 109/108=1.009259}, delta_rows:0, delta_bytes:0, segments:434, stale_read_regions:0, tot_build_snapshot:0ms, tot_build_bitmap:25ms, tot_build_inputstream:226ms, min_local_stream:323ms, max_local_stream:780ms, dtfile:{data_scanned_rows:300005811, data_skipped_rows:682517, mvcc_scanned_rows:0, mvcc_skipped_rows:0, lm_filter_scanned_rows:0, lm_filter_skipped_rows:0, tot_rs_index_check:27ms, tot_read:5806ms}} | keep order:false                                                                                                                    | N/A     | N/A  |
+--------------------------------------+--------------+-----------+--------------+----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+---------+------+
9 rows in set (4.00 sec)

after:

mysql> mysql> explain analyze select avg(l_quantity) avgres from lineitem group by l_orderkey having avgres > 100000;
+--------------------------------------+--------------+-----------+--------------+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+---------+------+
| id                                   | estRows      | actRows   | task         | access object  | execution info                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | operator info                                                                                                                       | memory  | disk |
+--------------------------------------+--------------+-----------+--------------+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+---------+------+
| TableReader_65                       | 59126579.20  | 0         | root         |                | time:2.91s, open:1.53ms, close:5.11µs, loops:1, RU:102874.94, cop_task: {num: 2, max: 0s, min: 0s, avg: 0s, p95: 0s, copr_cache_hit_ratio: 0.00}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | MppVersion: 3, data:ExchangeSender_64                                                                                               | 1.50 KB | N/A  |
| └─ExchangeSender_64                  | 59126579.20  | 0         | mpp[tiflash] |                | tiflash_task:{proc max:2.91s, min:2.88s, avg: 2.89s, p80:2.91s, p95:2.91s, iters:0, tasks:2, threads:32}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | ExchangeType: PassThrough                                                                                                           | N/A     | N/A  |
|   └─Selection_63                     | 59126579.20  | 0         | mpp[tiflash] |                | tiflash_task:{proc max:2.91s, min:2.88s, avg: 2.89s, p80:2.91s, p95:2.91s, iters:0, tasks:2, threads:32}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | gt(Column#17, 100000)                                                                                                               | N/A     | N/A  |
|     └─Projection_57                  | 73908224.00  | 75000000  | mpp[tiflash] |                | tiflash_task:{proc max:2.87s, min:2.85s, avg: 2.86s, p80:2.87s, p95:2.87s, iters:4096, tasks:2, threads:32}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | div(Column#17, cast(case(eq(Column#31, 0), 1, Column#31), decimal(20,0) BINARY))->Column#17                                         | N/A     | N/A  |
|       └─HashAgg_58                   | 73908224.00  | 75000000  | mpp[tiflash] |                | tiflash_task:{proc max:2.42s, min:2.41s, avg: 2.42s, p80:2.42s, p95:2.42s, iters:4096, tasks:2, threads:32}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | group by:test.lineitem.l_orderkey, funcs:sum(Column#32)->Column#31, funcs:sum(Column#33)->Column#17, stream_count: 16               | N/A     | N/A  |
|         └─ExchangeReceiver_60        | 73908224.00  | 75000091  | mpp[tiflash] |                | tiflash_task:{proc max:1.87s, min:1.83s, avg: 1.85s, p80:1.87s, p95:1.87s, iters:7238, tasks:2, threads:32}, tiflash_wait: {pipeline_queue_wait: 356ms}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | stream_count: 16                                                                                                                    | N/A     | N/A  |
|           └─ExchangeSender_59        | 73908224.00  | 75000091  | mpp[tiflash] |                | tiflash_task:{proc max:2.21s, min:0s, avg: 1.11s, p80:2.21s, p95:2.21s, iters:1536, tasks:2, threads:144}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | ExchangeType: HashPartition, Compression: FAST, Hash Cols: [name: test.lineitem.l_orderkey, collate: binary], stream_count: 16      | N/A     | N/A  |
|             └─HashAgg_55             | 73908224.00  | 75000091  | mpp[tiflash] |                | tiflash_task:{proc max:2.04s, min:0s, avg: 1.02s, p80:2.04s, p95:2.04s, iters:1536, tasks:2, threads:144}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | group by:test.lineitem.l_orderkey, funcs:count(test.lineitem.l_quantity)->Column#32, funcs:sum(test.lineitem.l_quantity)->Column#33 | N/A     | N/A  |
|               └─TableFullScan_35     | 300005811.00 | 300005811 | mpp[tiflash] | table:lineitem | tiflash_task:{proc max:806ms, min:0s, avg: 403ms, p80:806ms, p95:806ms, iters:4757, tasks:2, threads:144}, tiflash_wait: {pipeline_queue_wait: 785ms}, tiflash_scan:{mvcc_input_rows:0, mvcc_input_bytes:0, mvcc_output_rows:0, local_regions:217, remote_regions:0, tot_learner_read:8ms, region_balance:{instance_num: 2, max/min: 109/108=1.009259}, delta_rows:0, delta_bytes:0, segments:434, stale_read_regions:0, tot_build_snapshot:0ms, tot_build_bitmap:13ms, tot_build_inputstream:233ms, min_local_stream:317ms, max_local_stream:799ms, dtfile:{data_scanned_rows:300005811, data_skipped_rows:682517, mvcc_scanned_rows:0, mvcc_skipped_rows:0, lm_filter_scanned_rows:0, lm_filter_skipped_rows:0, tot_rs_index_check:33ms, tot_read:6088ms}} | keep order:false                                                                                                                    | N/A     | N/A  |
+--------------------------------------+--------------+-----------+--------------+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+---------+------+
9 rows in set (2.93 sec)

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

create table tt1(c1 decimal(15, 2) not null, c2 int not null);
insert into tt1 values(1, 1), (2, 2);
alter table tt1 set tiflash replica 1;
explain analyze select /*+ mpp_2phase_agg() */ avg(c1) from tt1 group by c2;

before: need to cast sum(table_scan_0) (which is Decimal(37, 2)) to Decimal(41, 6) in task1.

task1
[2024/07/24 11:19:42.698 +08:00] [INFO] [executeQuery.cpp:424] ["Query pipeline:
ExchangeSender: <enable fine grained shuffle> header: ExchangeSender_27_CAST(count()_collator , Int64_String)_collator_0  0 Int64 Int64(size = 0), ExchangeSender_27_CAST(sum(table_scan_0)_collator_0 , Nullable(Decimal(41,6))_String)_collator_0  0 Nullable(Decimal(41,6)) Nullable(size = 0, Decimal256(size = 0), UInt8(size = 0)), ExchangeSender_27_table_scan_1 2 Int32 Int32(size = 0)
 Expression: <final projection> header: ExchangeSender_27_CAST(count()_collator , Int64_String)_collator_0  0 Int64 Int64(size = 0), ExchangeSender_27_CAST(sum(table_scan_0)_collator_0 , Nullable(Decimal(41,6))_String)_collator_0  0 Nullable(Decimal(41,6)) Nullable(size = 0, Decimal256(size = 0), UInt8(size = 0)), ExchangeSender_27_table_scan_1 2 Int32 Int32(size = 0)
  Expression: <expr after aggregation> header: CAST(count()_collator , Int64_String)_collator_0  0 Int64 Int64(size = 0), CAST(sum(table_scan_0)_collator_0 , Nullable(Decimal(41,6))_String)_collator_0  0 Nullable(Decimal(41,6)) Nullable(size = 0, Decimal256(size = 0), UInt8(size = 0)), table_scan_1 2 Int32 Int32(size = 0)
   Aggregating header: table_scan_1 2 Int32 Int32(size = 0), count()_collator  0 UInt64 UInt64(size = 0), sum(table_scan_0)_collator_0  0 Decimal(37,2) Decimal128(size = 0)
    Expression: <table scan schema projection> header: table_scan_0 1 Decimal(15,2) Decimal64(size = 0), table_scan_1 2 Int32 Int32(size = 0)
     UnorderedInputStream header: c1 1 Decimal(15,2) Decimal64(size = 0), c2 2 Int32 Int32(size = 0)
"] [source="MPP<gather_id:1, query_ts:1721791182686661720, local_query_id:5, server_id:578, start_ts:451357227788009473,task_id:1>"] [thread_id=157]

task2:
[2024/07/24 11:19:42.703 +08:00] [INFO] [executeQuery.cpp:424] ["Query pipeline:
Union: <for mpp> header: ExchangeSender_29_CAST(tidbDivide(sumWithOverflow(exchange_receiver_1)_collator_0 , tidb_cast(multiIf(equals(sumWithOverflow(exchange_receiver_0)_collator_0 , 0_UInt8)_collator_63 , 1_UInt8, sumWithOverflow(exchange_receiver_0)_collator_0 )_collator_63 , Nullable(Decimal(20,0))_String)_collator_0 128|20)_collator_63 , Nullable(Decimal(19,6))_String)_collator_0  0 Nullable(Decimal(19,6)) Nullable(size = 0, Decimal128(size = 0), UInt8(size = 0))
 ExchangeSender x 10 header: ExchangeSender_29_CAST(tidbDivide(sumWithOverflow(exchange_receiver_1)_collator_0 , tidb_cast(multiIf(equals(sumWithOverflow(exchange_receiver_0)_collator_0 , 0_UInt8)_collator_63 , 1_UInt8, sumWithOverflow(exchange_receiver_0)_collator_0 )_collator_63 , Nullable(Decimal(20,0))_String)_collator_0 128|20)_collator_63 , Nullable(Decimal(19,6))_String)_collator_0  0 Nullable(Decimal(19,6)) Nullable(size = 0, Decimal128(size = 0), UInt8(size = 0))
  Expression: <final projection> header: ExchangeSender_29_CAST(tidbDivide(sumWithOverflow(exchange_receiver_1)_collator_0 , tidb_cast(multiIf(equals(sumWithOverflow(exchange_receiver_0)_collator_0 , 0_UInt8)_collator_63 , 1_UInt8, sumWithOverflow(exchange_receiver_0)_collator_0 )_collator_63 , Nullable(Decimal(20,0))_String)_collator_0 128|20)_collator_63 , Nullable(Decimal(19,6))_String)_collator_0  0 Nullable(Decimal(19,6)) Nullable(size = 0, Decimal128(size = 0), UInt8(size = 0))
   Expression: <projection> header: CAST(tidbDivide(sumWithOverflow(exchange_receiver_1)_collator_0 , tidb_cast(multiIf(equals(sumWithOverflow(exchange_receiver_0)_collator_0 , 0_UInt8)_collator_63 , 1_UInt8, sumWithOverflow(exchange_receiver_0)_collator_0 )_collator_63 , Nullable(Decimal(20,0))_String)_collator_0 128|20)_collator_63 , Nullable(Decimal(19,6))_String)_collator_0  0 Nullable(Decimal(19,6)) Nullable(size = 0, Decimal128(size = 0), UInt8(size = 0))
    Expression: <expr after aggregation> header: sumWithOverflow(exchange_receiver_0)_collator_0  0 Int64 Int64(size = 0), sumWithOverflow(exchange_receiver_1)_collator_0  0 Nullable(Decimal(41,6)) Nullable(size = 0, Decimal256(size = 0), UInt8(size = 0)), exchange_receiver_2 0 Int32 Int32(size = 0)
     Aggregating: <enable fine grained shuffle> header: exchange_receiver_2 0 Int32 Int32(size = 0), sumWithOverflow(exchange_receiver_0)_collator_0  0 Int64 Int64(size = 0), sumWithOverflow(exchange_receiver_1)_collator_0  0 Nullable(Decimal(41,6)) Nullable(size = 0, Decimal256(size = 0), UInt8(size = 0))
      TiRemote(ExchangeReceiver): <squashing after exchange receiver, enable fine grained shuffle> header: exchange_receiver_0 0 Int64 Int64(size = 0), exchange_receiver_1 0 Nullable(Decimal(41,6)) Nullable(size = 0, Decimal256(size = 0), UInt8(size = 0)), exchange_receiver_2 0 Int32 Int32(size = 0): schema: {<exchange_receiver_0, Int64>, <exchange_receiver_1, Nullable(Decimal(41,6))>, <exchange_receiver_2, Int32>}
"] [source="MPP<gather_id:1, query_ts:1721791182686661720, local_query_id:5, server_id:578, start_ts:451357227788009473,task_id:2>"] [thread_id=158]

after: no cast is needed

task1:
[2025/02/24 16:50:08.250 +08:00] [DEBUG] [executeQuery.cpp:424] ["Query pipeline:
ExchangeSender: <enable fine grained shuffle>, header: ExchangeSender_27_CAST(count()_collator , Int64_StringV2)_collator_0  0 Int64 Int64(size = 0), ExchangeSender_27_CAST(sum(table_scan_0)_collator_0 , Nullable(Decimal(37,2))_StringV2)_collator_0  0 Nullable(Decimal(37,2)) Nullable(size = 0, Decimal128(size = 0), UInt8(size = 0)), ExchangeSender_27_table_scan_1 2 Int32 Int32(size = 0)
 Expression: <final projection>, header: ExchangeSender_27_CAST(count()_collator , Int64_StringV2)_collator_0  0 Int64 Int64(size = 0), ExchangeSender_27_CAST(sum(table_scan_0)_collator_0 , Nullable(Decimal(37,2))_StringV2)_collator_0  0 Nullable(Decimal(37,2)) Nullable(size = 0, Decimal128(size = 0), UInt8(size = 0)), ExchangeSender_27_table_scan_1 2 Int32 Int32(size = 0)
  Expression: <expr after aggregation>, header: CAST(count()_collator , Int64_StringV2)_collator_0  0 Int64 Int64(size = 0), sum(table_scan_0)_collator_0  0 Decimal(37,2) Decimal128(size = 0), table_scan_1 2 Int32 Int32(size = 0)
   Aggregating, header: table_scan_1 2 Int32 Int32(size = 0), count()_collator  0 UInt64 UInt64(size = 0), sum(table_scan_0)_collator_0  0 Decimal(37,2) Decimal128(size = 0)
    Expression: <table scan schema projection>, header: table_scan_0 1 Decimal(15,2) Decimal64(size = 0), table_scan_1 2 Int32 Int32(size = 0)
     UnorderedInputStream, header: c1 1 Decimal(15,2) Decimal64(size = 0), c2 2 Int32 Int32(size = 0)
"] [source="MPP<gather_id:1, query_ts:1740387008208933368, local_query_id:3, server_id:1492, start_ts:456232011865522178,task_id:1>"] [thread_id=388]

task2:
[2025/02/24 16:50:08.264 +08:00] [DEBUG] [executeQuery.cpp:424] ["Query pipeline:
Union: <for mpp>, header: ExchangeSender_29_CAST(tidbDivide(sumWithOverflow(exchange_receiver_1)_collator_0 , tidb_cast(multiIf(equals(sumWithOverflow(exchange_receiver_0)_collator_0 , 0_UInt8)_collator_63 , 1_UInt8, sumWithOverflow(exchange_receiver_0)_collator_0 )_collator_63 , Nullable(Decimal(20,0))_StringV2)_collator_0 128|20)_collator_63 , Nullable(Decimal(19,6))_StringV2)_collator_0  0 Nullable(Decimal(19,6)) Nullable(size = 0, Decimal128(size = 0), UInt8(size = 0))
 ExchangeSender x 20, header: ExchangeSender_29_CAST(tidbDivide(sumWithOverflow(exchange_receiver_1)_collator_0 , tidb_cast(multiIf(equals(sumWithOverflow(exchange_receiver_0)_collator_0 , 0_UInt8)_collator_63 , 1_UInt8, sumWithOverflow(exchange_receiver_0)_collator_0 )_collator_63 , Nullable(Decimal(20,0))_StringV2)_collator_0 128|20)_collator_63 , Nullable(Decimal(19,6))_StringV2)_collator_0  0 Nullable(Decimal(19,6)) Nullable(size = 0, Decimal128(size = 0), UInt8(size = 0))
  Expression: <final projection>, header: ExchangeSender_29_CAST(tidbDivide(sumWithOverflow(exchange_receiver_1)_collator_0 , tidb_cast(multiIf(equals(sumWithOverflow(exchange_receiver_0)_collator_0 , 0_UInt8)_collator_63 , 1_UInt8, sumWithOverflow(exchange_receiver_0)_collator_0 )_collator_63 , Nullable(Decimal(20,0))_StringV2)_collator_0 128|20)_collator_63 , Nullable(Decimal(19,6))_StringV2)_collator_0  0 Nullable(Decimal(19,6)) Nullable(size = 0, Decimal128(size = 0), UInt8(size = 0))
   Expression: <projection>, header: CAST(tidbDivide(sumWithOverflow(exchange_receiver_1)_collator_0 , tidb_cast(multiIf(equals(sumWithOverflow(exchange_receiver_0)_collator_0 , 0_UInt8)_collator_63 , 1_UInt8, sumWithOverflow(exchange_receiver_0)_collator_0 )_collator_63 , Nullable(Decimal(20,0))_StringV2)_collator_0 128|20)_collator_63 , Nullable(Decimal(19,6))_StringV2)_collator_0  0 Nullable(Decimal(19,6)) Nullable(size = 0, Decimal128(size = 0), UInt8(size = 0))
    Expression: <expr after aggregation>, header: sumWithOverflow(exchange_receiver_0)_collator_0  0 Int64 Int64(size = 0), sumWithOverflow(exchange_receiver_1)_collator_0  0 Nullable(Decimal(37,2)) Nullable(size = 0, Decimal128(size = 0), UInt8(size = 0)), exchange_receiver_2 0 Int32 Int32(size = 0)
     Aggregating: <enable fine grained shuffle>, header: exchange_receiver_2 0 Int32 Int32(size = 0), sumWithOverflow(exchange_receiver_0)_collator_0  0 Int64 Int64(size = 0), sumWithOverflow(exchange_receiver_1)_collator_0  0 Nullable(Decimal(37,2)) Nullable(size = 0, Decimal128(size = 0), UInt8(size = 0))
      TiRemote(ExchangeReceiver): <squashing after exchange receiver, enable fine grained shuffle>, header: exchange_receiver_0 0 Int64 Int64(size = 0), exchange_receiver_1 0 Nullable(Decimal(37,2)) Nullable(size = 0, Decimal128(size = 0), UInt8(size = 0)), exchange_receiver_2 0 Int32 Int32(size = 0): schema: {<exchange_receiver_0, Int64>, <exchange_receiver_1, Nullable(Decimal(37,2))>, <exchange_receiver_2, Int32>}
"] [source="MPP<gather_id:1, query_ts:1740387008208933368, local_query_id:3, server_id:1492, start_ts:456232011865522178,task_id:2>"] [thread_id=389]

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: guo-shaoge <[email protected]>

pkg/planner/core/task.go

codecov · 2025-02-24T08:42:55Z

Codecov Report

Attention: Patch coverage is 57.69231% with 11 lines in your changes missing coverage. Please review.

Project coverage is 72.9424%. Comparing base (aa21818) to head (965890a).
Report is 272 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #59709        +/-   ##
================================================
- Coverage   73.1282%   72.9424%   -0.1858%     
================================================
  Files          1680       1709        +29     
  Lines        464584     499085     +34501     
================================================
+ Hits         339742     364045     +24303     
- Misses       103978     113181      +9203     
- Partials      20864      21859       +995

Flag	Coverage Δ
integration	`42.7604% <38.4615%> (?)`
unit	`72.3645% <57.6923%> (+0.0518%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.6910% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`43.0255% <ø> (-2.3881%)`	⬇️

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge · 2025-02-26T08:12:23Z

/retest

Signed-off-by: guo-shaoge <[email protected]>

pkg/expression/aggregation/base_func.go

windtalker

lgtm

XuHuaiyu

Can we add an unit test case for TypeInfer4AvgSum ?

Signed-off-by: guo-shaoge <[email protected]>

ti-chi-bot · 2025-03-10T09:08:43Z

[LGTM Timeline notifier]

Timeline:

2025-03-10 04:51:12.126896243 +0000 UTC m=+158627.732425155: ☑️ agreed by windtalker.
2025-03-10 09:08:41.710646647 +0000 UTC m=+174077.316175574: ☑️ agreed by XuHuaiyu.

Signed-off-by: guo-shaoge <[email protected]>

ti-chi-bot · 2025-03-11T01:42:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hawkingrei, windtalker, XuHuaiyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/expression/OWNERS~~ [XuHuaiyu,windtalker]
~~pkg/planner/OWNERS~~ [hawkingrei]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

guo-shaoge · 2025-03-11T02:19:56Z

/retest

guo-shaoge · 2025-03-11T02:41:21Z

/retest

tiprow · 2025-03-11T02:52:02Z

@guo-shaoge: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
tidb_parser_test	`965890a`	link	true	`/test tidb_parser_test`
fast_test_tiprow	`965890a`	link	true	`/test fast_test_tiprow`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

close pingcap#59708

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/needs-tests-checked size/M Denotes a PR that changes 30-99 lines, ignoring generated files. sig/planner SIG: Planner labels Feb 24, 2025

guo-shaoge force-pushed the opt_tiflash_avg_sum_type branch from b8a2bd8 to 43335ac Compare February 24, 2025 08:11

guo-shaoge changed the title ~~expression: optimize return type of partial sum for tiflash~~ planner: optimize return type of partial sum for tiflash Feb 24, 2025

expression: optimize return type of partial sum for tiflash

97251aa

Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge force-pushed the opt_tiflash_avg_sum_type branch from 43335ac to 97251aa Compare February 24, 2025 08:21

ti-chi-bot bot removed the do-not-merge/needs-tests-checked label Feb 24, 2025

hawkingrei reviewed Feb 24, 2025

View reviewed changes

pkg/planner/core/task.go Outdated Show resolved Hide resolved

guo-shaoge added 2 commits February 24, 2025 16:46

fix

e9bc73c

Signed-off-by: guo-shaoge <[email protected]>

fix

8eaf40b

Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge requested a review from windtalker February 24, 2025 10:35

fix

bc0f414

Signed-off-by: guo-shaoge <[email protected]>

ti-chi-bot bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 25, 2025

guo-shaoge added 3 commits February 25, 2025 15:02

refine

954e871

Signed-off-by: guo-shaoge <[email protected]>

fix

3298f6d

Signed-off-by: guo-shaoge <[email protected]>

fix

c1b8f87

Signed-off-by: guo-shaoge <[email protected]>

ti-chi-bot bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Feb 26, 2025

comment

d44dbd5

Signed-off-by: guo-shaoge <[email protected]>

windtalker reviewed Mar 3, 2025

View reviewed changes

pkg/expression/aggregation/base_func.go Show resolved Hide resolved

guo-shaoge requested a review from windtalker March 10, 2025 04:47

windtalker approved these changes Mar 10, 2025

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Mar 10, 2025

XuHuaiyu reviewed Mar 10, 2025

View reviewed changes

add case

60d011e

Signed-off-by: guo-shaoge <[email protected]>

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 10, 2025

guo-shaoge requested a review from XuHuaiyu March 10, 2025 08:50

XuHuaiyu approved these changes Mar 10, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Mar 10, 2025

bazel

965890a

Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge requested a review from hawkingrei March 10, 2025 12:08

hawkingrei approved these changes Mar 11, 2025

View reviewed changes

ti-chi-bot bot added the approved label Mar 11, 2025

ti-chi-bot bot merged commit d884a4d into pingcap:master Mar 11, 2025
22 of 24 checks passed

guo-shaoge deleted the opt_tiflash_avg_sum_type branch March 11, 2025 03:52

zeminzhou pushed a commit to zeminzhou/tidb that referenced this pull request May 6, 2025

planner: optimize return type of partial sum for tiflash (pingcap#59709)

e10ec85

close pingcap#59708

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

planner: optimize return type of partial sum for tiflash #59709

planner: optimize return type of partial sum for tiflash #59709

Uh oh!

guo-shaoge commented Feb 24, 2025 •

edited

Loading

Uh oh!

Uh oh!

codecov bot commented Feb 24, 2025 •

edited

Loading

Uh oh!

guo-shaoge commented Feb 26, 2025

Uh oh!

Uh oh!

windtalker left a comment

Uh oh!

XuHuaiyu left a comment

Uh oh!

ti-chi-bot bot commented Mar 10, 2025

Uh oh!

ti-chi-bot bot commented Mar 11, 2025

Uh oh!

guo-shaoge commented Mar 11, 2025

Uh oh!

guo-shaoge commented Mar 11, 2025

Uh oh!

tiprow bot commented Mar 11, 2025

Uh oh!

Uh oh!

Uh oh!

planner: optimize return type of partial sum for tiflash #59709

planner: optimize return type of partial sum for tiflash #59709

Uh oh!

Conversation

guo-shaoge commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

Uh oh!

codecov bot commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

guo-shaoge commented Feb 26, 2025

Uh oh!

Uh oh!

windtalker left a comment

Choose a reason for hiding this comment

Uh oh!

XuHuaiyu left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Mar 10, 2025

[LGTM Timeline notifier]

Uh oh!

ti-chi-bot bot commented Mar 11, 2025

Uh oh!

guo-shaoge commented Mar 11, 2025

Uh oh!

guo-shaoge commented Mar 11, 2025

Uh oh!

tiprow bot commented Mar 11, 2025

Uh oh!

Uh oh!

Uh oh!

guo-shaoge commented Feb 24, 2025 •

edited

Loading

codecov bot commented Feb 24, 2025 •

edited

Loading