Skip to content

[Fix](load) Restrict the import of VARCHAR(0) data to avoid coredump #40940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 19, 2024
Merged

[Fix](load) Restrict the import of VARCHAR(0) data to avoid coredump #40940

merged 1 commit into from
Sep 19, 2024

Conversation

liaoxin01
Copy link
Contributor

The VARCHAR(0) created has an actual length of 0, and BE didn't restrict it, leading to a BE core dump. #38427 changed VARCHAR(0) to have a length of 65533. This PR restricts data import when the length is 0 to avoid a core dump.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@liaoxin01
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41091 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0bb7001f3c53100074ed4b6e7bf0c09461d5ed1, data reload: false

------ Round 1 ----------------------------------
q1	18001	7378	7267	7267
q2	2429	168	178	168
q3	11116	1121	1140	1121
q4	10227	772	661	661
q5	7707	3078	3027	3027
q6	237	148	148	148
q7	1000	612	615	612
q8	9420	2015	2025	2015
q9	6877	6403	6372	6372
q10	7013	2287	2313	2287
q11	439	248	253	248
q12	408	211	209	209
q13	17778	2955	3005	2955
q14	238	226	226	226
q15	582	530	510	510
q16	685	609	616	609
q17	967	843	780	780
q18	7268	6712	6547	6547
q19	1408	1044	1108	1044
q20	598	311	284	284
q21	4093	3161	2993	2993
q22	1089	1008	1018	1008
Total cold run time: 109580 ms
Total hot run time: 41091 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7267	7210	7210	7210
q2	333	226	230	226
q3	2902	2766	2751	2751
q4	1920	1682	1714	1682
q5	5372	5394	5371	5371
q6	230	143	145	143
q7	2126	1716	1728	1716
q8	3196	3319	3264	3264
q9	8389	8460	8433	8433
q10	3367	3367	3348	3348
q11	579	468	474	468
q12	769	607	567	567
q13	6544	3014	3060	3014
q14	285	253	254	253
q15	558	506	515	506
q16	697	690	674	674
q17	1792	1543	1543	1543
q18	7708	7501	7322	7322
q19	1647	1611	1451	1451
q20	2035	1797	1833	1797
q21	5342	5177	5124	5124
q22	1115	1013	1024	1013
Total cold run time: 64173 ms
Total hot run time: 57876 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.33% (9583/25672)
Line Coverage: 28.72% (79217/275867)
Region Coverage: 28.19% (41021/145509)
Branch Coverage: 24.82% (20908/84246)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b0bb7001f3c53100074ed4b6e7bf0c09461d5ed1_b0bb7001f3c53100074ed4b6e7bf0c09461d5ed1/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 193561 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b0bb7001f3c53100074ed4b6e7bf0c09461d5ed1, data reload: false

query1	944	389	358	358
query2	6511	2108	1996	1996
query3	6705	211	219	211
query4	34364	23405	23325	23325
query5	4396	478	431	431
query6	256	163	160	160
query7	4620	292	302	292
query8	271	221	225	221
query9	9661	2669	2639	2639
query10	482	304	268	268
query11	18035	15133	15251	15133
query12	157	95	99	95
query13	1624	418	397	397
query14	10282	7263	7129	7129
query15	253	170	178	170
query16	7610	439	464	439
query17	1670	570	553	553
query18	1944	304	300	300
query19	347	142	149	142
query20	119	115	111	111
query21	212	106	105	105
query22	4585	4177	4259	4177
query23	34742	34037	33785	33785
query24	11174	2867	2895	2867
query25	642	389	384	384
query26	1348	162	163	162
query27	2705	285	286	285
query28	7966	2473	2442	2442
query29	875	428	420	420
query30	315	161	151	151
query31	1020	787	782	782
query32	101	57	61	57
query33	784	290	285	285
query34	985	493	489	489
query35	889	728	724	724
query36	1080	919	941	919
query37	150	96	84	84
query38	4064	3903	3922	3903
query39	1459	1397	1419	1397
query40	294	112	98	98
query41	52	50	52	50
query42	118	99	96	96
query43	526	496	498	496
query44	1260	822	786	786
query45	195	173	168	168
query46	1133	764	737	737
query47	1881	1801	1805	1801
query48	449	361	370	361
query49	1148	421	415	415
query50	807	394	411	394
query51	7081	7012	6988	6988
query52	98	87	85	85
query53	256	181	183	181
query54	1211	466	477	466
query55	78	75	79	75
query56	298	267	264	264
query57	1218	1092	1069	1069
query58	248	235	233	233
query59	3192	3004	2919	2919
query60	310	279	283	279
query61	125	125	98	98
query62	889	651	687	651
query63	217	182	181	181
query64	5238	644	631	631
query65	3254	3249	3178	3178
query66	1424	302	302	302
query67	16018	15506	15418	15418
query68	3491	554	553	553
query69	421	294	284	284
query70	1193	1144	1123	1123
query71	330	275	266	266
query72	6091	3942	3979	3942
query73	751	330	327	327
query74	9906	9004	8951	8951
query75	3390	2668	2672	2668
query76	2546	909	839	839
query77	425	286	292	286
query78	10262	9568	9330	9330
query79	1035	882	869	869
query80	861	576	560	560
query81	522	252	259	252
query82	832	235	229	229
query83	209	155	161	155
query84	246	104	104	104
query85	746	382	346	346
query86	359	321	316	316
query87	4438	4309	4235	4235
query88	4501	4074	4046	4046
query89	377	357	355	355
query90	1838	322	309	309
query91	157	160	165	160
query92	75	72	75	72
query93	913	906	889	889
query94	783	381	352	352
query95	448	404	398	398
query96	486	486	486	486
query97	3129	3150	3119	3119
query98	226	230	222	222
query99	1433	1312	1310	1310
Total cold run time: 296785 ms
Total hot run time: 193561 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.13 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b0bb7001f3c53100074ed4b6e7bf0c09461d5ed1, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.02
query3	0.22	0.06	0.06
query4	1.65	0.09	0.10
query5	0.52	0.52	0.50
query6	1.13	0.74	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.58	0.50	0.49
query10	0.55	0.56	0.53
query11	0.13	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	2.98	3.04	3.11
query15	0.90	0.83	0.81
query16	0.38	0.40	0.40
query17	1.02	1.06	1.02
query18	0.22	0.21	0.20
query19	1.92	1.82	1.94
query20	0.01	0.01	0.01
query21	15.38	0.58	0.57
query22	2.37	1.82	1.75
query23	17.34	0.83	0.64
query24	3.56	1.57	1.82
query25	0.32	0.21	0.12
query26	0.44	0.14	0.14
query27	0.04	0.05	0.03
query28	9.01	1.11	1.08
query29	12.55	3.28	3.25
query30	0.25	0.06	0.06
query31	2.87	0.39	0.37
query32	3.27	0.45	0.46
query33	3.00	3.03	3.05
query34	16.70	4.38	4.35
query35	4.46	4.36	4.41
query36	0.70	0.47	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.84 s
Total hot run time: 33.13 s

dataroaring
dataroaring approved these changes Sep 19, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

sollhui
sollhui approved these changes Sep 19, 2024
View reviewed changes
Copy link
Contributor

@sollhui sollhui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit adf7fb1 into apache:master Sep 19, 2024
24 of 29 checks passed
@liaoxin01 liaoxin01 deleted the fix_varchar0 branch September 19, 2024 03:47
@liaoxin01 liaoxin01 mentioned this pull request Sep 19, 2024
Merged
liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Sep 19, 2024
@liaoxin01
[Fix](load) Restrict the import of VARCHAR(0) data to avoid coredump (a…
319d412 
…pache#40940)

The VARCHAR(0) created has an actual length of 0, and BE didn't restrict
it, leading to a BE core dump.
#[38427](apache#38427) changed VARCHAR(0)
to have a length of 65533. This PR restricts data import when the length
is 0 to avoid a core dump.
liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Sep 19, 2024
@liaoxin01
[Fix](load) Restrict the import of VARCHAR(0) data to avoid coredump (a…
a2ce6f3 
…pache#40940)

The VARCHAR(0) created has an actual length of 0, and BE didn't restrict
it, leading to a BE core dump.
to have a length of 65533. This PR restricts data import when the length
is 0 to avoid a core dump.
@liaoxin01 liaoxin01 mentioned this pull request Sep 19, 2024
Merged
liaoxin01 added a commit that referenced this pull request Sep 19, 2024
@liaoxin01
[Fix](load) Restrict the import of VARCHAR(0) data to avoid coredump #…
f2b93d5 
…40940 (#40974)

cherry pick from #40940
dataroaring pushed a commit that referenced this pull request Sep 19, 2024
@liaoxin01
[Fix](load) Restrict the import of VARCHAR(0) data to avoid coredump (#…
a6be436 
…40940) (#40976)

cherry pick from #40940
dataroaring pushed a commit that referenced this pull request Sep 26, 2024
@liaoxin01 @dataroaring
[Fix](load) Restrict the import of VARCHAR(0) data to avoid coredump (#…
845302d 
…40940)

The VARCHAR(0) created has an actual length of 0, and BE didn't restrict
it, leading to a BE core dump.
#[38427](#38427) changed VARCHAR(0)
to have a length of 65533. This PR restricts data import when the length
is 0 to avoid a core dump.
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

+1 more reviewer

@sollhui sollhui sollhui approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.7-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@liaoxin01 @doris-robot @sollhui @dataroaring