[feature](function) add approx_top_sum aggregation function #43643

zzzxl1993 · 2024-11-11T15:38:13Z

What problem does this PR solve?

Problem Summary:

The function approx_top_sum has been implemented. Here is an example of its usage: select approx_top_sum(c1, c2, c3, 10, 300) from tbl.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-11T15:38:18Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

github-actions

clang-tidy made some suggestions

github-actions · 2024-11-11T15:43:18Z

be/src/vec/aggregate_functions/aggregate_function_approx_top_sum.h

+
+#pragma once
+
+#include <rapidjson/encodings.h>


warning: 'rapidjson/encodings.h' file not found [clang-diagnostic-error]

#include <rapidjson/encodings.h> ^

zzzxl1993 · 2024-11-12T02:13:02Z

run buildall

zzzxl1993 · 2024-11-12T05:57:30Z

run buildall

zzzxl1993 · 2024-11-12T11:28:42Z

run buildall

superdiaodiao · 2024-11-12T12:06:14Z

regression-test/suites/inverted_index_p0/test_index_approx_top_k.groovy

-        qt_sql """ select approx_top_k(clientip) from ${tableName}; """
-        qt_sql """ select approx_top_k(clientip, 10) from ${tableName}; """


May I ask why deleting these tests in a feature PR?

Currently, it is difficult to implement default values for the variable arguments in the function framework. It is hoped that users will directly specify the last two parameters.

zzzxl1993 · 2024-11-12T13:34:48Z

run buildall

xiaokang · 2024-11-13T23:21:46Z

be/src/vec/aggregate_functions/aggregate_function.h

@@ -43,7 +43,7 @@ class IDataType;

 struct AggregateFunctionAttr {
    bool enable_decimal256 {false};
-    std::vector<std::pair<std::string, bool>> column_infos;
+    std::vector<std::string> column_names;


Why change this basic data structure?

Optimize the redundant data structures added in approx_top_k.

xiaokang · 2024-11-13T23:35:31Z

be/src/vec/aggregate_functions/aggregate_function_approx_top_sum.h

+                sub_writer.Key(_column_names[i].data(), _column_names[i].size());
+                sub_writer.String(row_str.data(), row_str.size());
+            }
+            sub_writer.Key("count");


xiaokang · 2024-11-13T23:36:56Z

...-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/ApproxTopK.java

@@ -55,9 +55,19 @@ public ApproxTopK(boolean distinct, boolean alwaysNullable, Expression... varArg

    @Override
    public void checkLegalityBeforeTypeCoercion() {
-        if (arity() < 1) {
+        if (arity() < 3) {


Why change arguments num from 1 to 3?

The existing function framework code cannot implement approx_top_k and approx_top_sum with default values.

zzzxl1993 · 2024-11-14T15:39:51Z

run buildall

doris-robot · 2024-11-14T17:23:27Z

TeamCity be ut coverage result:
Function Coverage: 37.96% (9904/26090)
Line Coverage: 29.16% (82764/283794)
Region Coverage: 28.29% (42514/150280)
Branch Coverage: 24.86% (21556/86694)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cf48a1bd8a8007fe40606b7a4d907a182403e85c_cf48a1bd8a8007fe40606b7a4d907a182403e85c/report/index.html

zzzxl1993 · 2024-11-15T03:34:45Z

run buildall

doris-robot · 2024-11-15T04:15:31Z

TeamCity be ut coverage result:
Function Coverage: 37.96% (9903/26090)
Line Coverage: 29.16% (82762/283795)
Region Coverage: 28.29% (42519/150280)
Branch Coverage: 24.86% (21554/86694)
Coverage Report: http://coverage.selectdb-in.cc/coverage/adf71e19982471db714eaf0aa221f815962c102d_adf71e19982471db714eaf0aa221f815962c102d/report/index.html

xiaokang

LGTM

github-actions · 2024-11-17T01:17:24Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-17T01:17:26Z

PR approved by anyone and no changes requested.

qidaye

LGTM

…3643) Problem Summary: 1. The function approx_top_sum has been implemented. Here is an example of its usage: select approx_top_sum(c1, c2, c3, 10, 300) from tbl. ### Release note Add new function `approx_top_sum`.

…3643) Problem Summary: 1. The function approx_top_sum has been implemented. Here is an example of its usage: select approx_top_sum(c1, c2, c3, 10, 300) from tbl. Add new function `approx_top_sum`.

pick #40813 pick #43643

github-actions bot reviewed Nov 11, 2024

View reviewed changes

zzzxl1993 force-pushed the 202411111526 branch from ca8d282 to 1ea36ee Compare November 12, 2024 05:57

zzzxl1993 force-pushed the 202411111526 branch from 1ea36ee to 17ecc13 Compare November 12, 2024 11:28

superdiaodiao reviewed Nov 12, 2024

View reviewed changes

zzzxl1993 force-pushed the 202411111526 branch from 17ecc13 to 23d7adc Compare November 12, 2024 13:34

xiaokang reviewed Nov 13, 2024

View reviewed changes

zzzxl1993 force-pushed the 202411111526 branch from 23d7adc to cf48a1b Compare November 14, 2024 15:39

[feature](function) add approx_top_sum aggregation function

adf71e1

zzzxl1993 force-pushed the 202411111526 branch from cf48a1b to adf71e1 Compare November 15, 2024 03:34

xiaokang approved these changes Nov 17, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 17, 2024

github-actions bot added the reviewed label Nov 17, 2024

qidaye approved these changes Nov 17, 2024

View reviewed changes

qidaye merged commit 13ced20 into apache:master Nov 17, 2024
25 of 28 checks passed

zzzxl1993 mentioned this pull request Nov 17, 2024

[feature](function) add approx_top_k aggregation function #44082

Merged

qidaye pushed a commit that referenced this pull request Nov 18, 2024

[feature](function) add approx_top_k aggregation function (#44082)

15e26de

pick #40813 pick #43643

		qt_sql """ select approx_top_k(clientip) from ${tableName}; """
		qt_sql """ select approx_top_k(clientip, 10) from ${tableName}; """

[feature](function) add approx_top_sum aggregation function #43643

[feature](function) add approx_top_sum aggregation function #43643

Uh oh!

Conversation

zzzxl1993 commented Nov 11, 2024 • edited by qidaye Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Nov 11, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Nov 11, 2024

Choose a reason for hiding this comment

Uh oh!

zzzxl1993 commented Nov 12, 2024

Uh oh!

zzzxl1993 commented Nov 12, 2024

Uh oh!

zzzxl1993 commented Nov 12, 2024

Uh oh!

superdiaodiao Nov 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zzzxl1993 Nov 12, 2024

Choose a reason for hiding this comment

Uh oh!

zzzxl1993 commented Nov 12, 2024

Uh oh!

xiaokang Nov 13, 2024

Choose a reason for hiding this comment

Uh oh!

zzzxl1993 Nov 15, 2024

Choose a reason for hiding this comment

Uh oh!

xiaokang Nov 13, 2024

Choose a reason for hiding this comment

Uh oh!

zzzxl1993 Nov 15, 2024

Choose a reason for hiding this comment

Uh oh!

xiaokang Nov 13, 2024

Choose a reason for hiding this comment

Uh oh!

zzzxl1993 Nov 15, 2024

Choose a reason for hiding this comment

Uh oh!

zzzxl1993 commented Nov 14, 2024

Uh oh!

doris-robot commented Nov 14, 2024

Uh oh!

zzzxl1993 commented Nov 15, 2024

Uh oh!

doris-robot commented Nov 15, 2024

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 17, 2024

Uh oh!

github-actions bot commented Nov 17, 2024

Uh oh!

qidaye left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zzzxl1993 commented Nov 11, 2024 •

edited by qidaye

Loading

superdiaodiao Nov 12, 2024 •

edited

Loading