Fix column level lineage miss when use unnest #26559
Open
+44
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@Praveen2112 Could you help review it? thanks |
hqbhoho
force-pushed
the
feature/add_unnest_lineage
branch
from
e3311e1 to
f6b8286
Compare
chenjian2664
reviewed
Sep 6, 2025
testing/trino-tests/src/test/java/io/trino/execution/TestEventListenerBasic.java
Outdated
Show resolved
Hide resolved
hqbhoho
force-pushed
the
feature/add_unnest_lineage
branch
2 times, most recently
from
24a364a to
c01ecc0
Compare
hqbhoho
force-pushed
the
feature/add_unnest_lineage
branch
from
c01ecc0 to
34a0097
Compare
hqbhoho
force-pushed
the
feature/add_unnest_lineage
branch
from
34a0097 to
34b5b3c
Compare
|
I think overall is good, but I am not familiar and confident in lineage part. still needs @Praveen2112 to take a look |
Praveen2112
reviewed
Sep 22, 2025
| new OutputColumnMetadata("test_varchar", VARCHAR_TYPE, ImmutableSet.of(new ColumnDetail("mock", "default", "tests_table_unnest", "test_varchar_array"))), | ||
| new OutputColumnMetadata("test_bigint", BIGINT_TYPE, ImmutableSet.of(new ColumnDetail("mock", "default", "tests_table_unnest", "test_bigint")))); | ||
| assertLineage( | ||
| "SELECT test_varchar_unnest AS test_varchar, test_bigint_unnest AS test_bigint FROM mock.default.tests_table_unnest CROSS JOIN UNNEST(test_varchar_array) WITH ORDINALITY AS t(test_varchar_unnest, test_bigint_unnest)", |
There was a problem hiding this comment.
What if we if provide t(test_varchar_unnest, test_bigint_unnest, row_number) and try to assert the lineage of the row number ?
There was a problem hiding this comment.
If use row number column, this column lineage is empty.
Comment on lines
1672
to
1680
| for (Field field : outputFields) { | ||
| for (Map.Entry<NodeRef<Expression>, List<Field>> entry : mappings.entrySet()) { | ||
| Expression expression = entry.getKey().getNode(); | ||
| List<Field> fields = entry.getValue(); | ||
| if (fields.contains(field)) { | ||
| analysis.addSourceColumns(field, analysis.getExpressionSourceColumns(expression)); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Any specific reason for this for loop - instead we could add them as process each expression right ? Like
expressionOutputs.forEach(field -> analysis.addSourceColumns(field, analysis.getExpressionSourceColumns(expression)));
And similarly for cardinality as well.
There was a problem hiding this comment.
You are right, only add them as process each expression. Thank you for your feedback!
hqbhoho
force-pushed
the
feature/add_unnest_lineage
branch
from
34b5b3c to
b6013e0
Compare
hqbhoho
force-pushed
the
feature/add_unnest_lineage
branch
from
b6013e0 to
36f52e1
Compare
|
@Praveen2112 PTAL, thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
ReOpening #24620 since it went stale.
Fixes #16946
Additional context and related issues
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text: