Skip to content

[improvement](jdbc catalog) Modify the maximum number of connections in the connection pool to 30 by default #36720

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jun 28, 2024
Merged

[improvement](jdbc catalog) Modify the maximum number of connections in the connection pool to 30 by default #36720

merged 2 commits into from
Jun 28, 2024

Conversation

zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Jun 24, 2024

In many cases, we found that users would use JDBC Catalog to perform a large number of queries, which resulted in the maximum of 10 connections being insufficient, so I adjusted it to 30, which covered most needs.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zy-kkk
Copy link
Member Author

zy-kkk commented Jun 24, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40039 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dff36215c908103a56574b5981bda9bf1e57ad2e, data reload: false

------ Round 1 ----------------------------------
q1	17620	4928	4276	4276
q2	2014	193	194	193
q3	10446	1157	1113	1113
q4	10182	771	770	770
q5	7456	2722	2686	2686
q6	221	134	130	130
q7	953	593	611	593
q8	9234	2070	2082	2070
q9	8943	6586	6484	6484
q10	9056	3739	3747	3739
q11	452	233	234	233
q12	473	225	228	225
q13	17771	2964	2967	2964
q14	272	221	218	218
q15	520	468	476	468
q16	512	369	376	369
q17	962	624	688	624
q18	8141	7577	7472	7472
q19	6200	1554	1551	1551
q20	643	322	325	322
q21	4952	3212	3864	3212
q22	399	327	332	327
Total cold run time: 117422 ms
Total hot run time: 40039 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4398	4279	4284	4279
q2	383	260	263	260
q3	3001	2961	2952	2952
q4	2031	1682	1719	1682
q5	5492	5580	5516	5516
q6	216	127	127	127
q7	2292	1875	1890	1875
q8	3295	3486	3442	3442
q9	8661	8840	8828	8828
q10	4133	3839	3768	3768
q11	604	506	503	503
q12	785	649	641	641
q13	16111	3179	3208	3179
q14	310	267	266	266
q15	537	473	496	473
q16	487	421	417	417
q17	1829	1532	1479	1479
q18	8091	7933	7779	7779
q19	1859	1673	1497	1497
q20	3111	1912	1855	1855
q21	5234	5030	4722	4722
q22	714	559	521	521
Total cold run time: 73574 ms
Total hot run time: 56061 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173739 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dff36215c908103a56574b5981bda9bf1e57ad2e, data reload: false

query1	908	398	387	387
query2	6416	2415	2378	2378
query3	6632	212	210	210
query4	20301	17235	17342	17235
query5	3555	496	457	457
query6	235	164	174	164
query7	4580	292	284	284
query8	299	268	274	268
query9	8608	2370	2354	2354
query10	575	303	276	276
query11	10484	10015	10049	10015
query12	112	80	83	80
query13	1633	391	366	366
query14	9170	6774	7027	6774
query15	253	198	191	191
query16	7757	280	267	267
query17	1410	554	522	522
query18	2058	270	272	270
query19	199	149	157	149
query20	88	87	81	81
query21	215	133	124	124
query22	4728	4111	4259	4111
query23	34112	33608	33637	33608
query24	11074	2873	2949	2873
query25	621	382	368	368
query26	867	160	153	153
query27	2312	324	330	324
query28	6832	2109	2122	2109
query29	882	633	635	633
query30	266	159	162	159
query31	939	785	761	761
query32	98	54	54	54
query33	765	282	287	282
query34	1065	479	491	479
query35	750	656	622	622
query36	1140	984	985	984
query37	154	75	69	69
query38	2999	2870	2868	2868
query39	888	838	857	838
query40	218	132	133	132
query41	53	53	53	53
query42	113	106	104	104
query43	578	533	549	533
query44	1260	741	751	741
query45	202	160	159	159
query46	1086	727	734	727
query47	1942	1781	1791	1781
query48	376	312	293	293
query49	832	408	409	408
query50	794	397	401	397
query51	6906	6843	6790	6790
query52	99	93	91	91
query53	361	294	284	284
query54	897	439	435	435
query55	73	74	75	74
query56	281	254	264	254
query57	1119	1048	1054	1048
query58	246	232	239	232
query59	3345	3097	3165	3097
query60	290	265	281	265
query61	95	92	91	91
query62	604	434	444	434
query63	325	286	288	286
query64	8760	2287	1750	1750
query65	3159	3147	3107	3107
query66	826	323	342	323
query67	15573	15232	14917	14917
query68	4479	539	538	538
query69	440	296	302	296
query70	1173	1126	1167	1126
query71	403	274	272	272
query72	7124	5927	5509	5509
query73	742	330	327	327
query74	6001	5549	5452	5452
query75	3430	2703	2644	2644
query76	2200	944	915	915
query77	451	297	291	291
query78	10421	9900	9679	9679
query79	1794	505	515	505
query80	2471	462	471	462
query81	597	223	222	222
query82	1074	105	104	104
query83	305	172	169	169
query84	260	87	87	87
query85	1212	280	266	266
query86	458	334	327	327
query87	3312	3130	3124	3124
query88	3279	2466	2446	2446
query89	474	387	372	372
query90	1656	182	189	182
query91	124	96	97	96
query92	60	49	50	49
query93	2001	504	498	498
query94	1125	189	187	187
query95	403	307	313	307
query96	600	270	269	269
query97	3181	3037	3073	3037
query98	218	204	188	188
query99	1200	856	840	840
Total cold run time: 269061 ms
Total hot run time: 173739 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dff36215c908103a56574b5981bda9bf1e57ad2e, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.03	0.04
query3	0.21	0.05	0.05
query4	1.68	0.06	0.06
query5	0.53	0.49	0.50
query6	1.14	0.72	0.72
query7	0.01	0.01	0.02
query8	0.05	0.03	0.04
query9	0.54	0.48	0.49
query10	0.54	0.54	0.56
query11	0.16	0.12	0.11
query12	0.14	0.11	0.12
query13	0.59	0.59	0.58
query14	0.76	0.77	0.78
query15	0.83	0.82	0.80
query16	0.35	0.37	0.37
query17	1.00	1.01	1.03
query18	0.23	0.24	0.24
query19	1.90	1.82	1.79
query20	0.01	0.01	0.01
query21	15.40	0.67	0.65
query22	4.23	7.30	1.94
query23	18.27	1.36	1.27
query24	2.24	0.22	0.22
query25	0.16	0.08	0.08
query26	0.27	0.18	0.17
query27	0.08	0.08	0.08
query28	13.20	1.01	1.01
query29	12.57	3.36	3.30
query30	0.26	0.07	0.05
query31	2.87	0.38	0.38
query32	3.27	0.46	0.46
query33	2.90	2.88	2.91
query34	17.31	4.42	4.47
query35	4.55	4.54	4.49
query36	0.66	0.47	0.47
query37	0.18	0.16	0.16
query38	0.17	0.15	0.15
query39	0.05	0.03	0.04
query40	0.17	0.14	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.8 s
Total hot run time: 30.72 s

@zy-kkk
Copy link
Member Author

zy-kkk commented Jun 24, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40355 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85969729de9e801a86dcc0f89404112b2a36644f, data reload: false

------ Round 1 ----------------------------------
q1	17599	4372	4302	4302
q2	2008	188	197	188
q3	10452	1141	1073	1073
q4	10195	770	742	742
q5	7478	2652	2643	2643
q6	217	132	137	132
q7	937	604	604	604
q8	9218	2077	2023	2023
q9	8847	6498	6431	6431
q10	8992	3766	3734	3734
q11	442	242	247	242
q12	515	230	229	229
q13	17766	3003	2945	2945
q14	268	219	208	208
q15	499	491	474	474
q16	502	379	385	379
q17	956	714	705	705
q18	7926	7525	7514	7514
q19	7851	1327	1548	1327
q20	659	322	321	321
q21	4914	3800	3859	3800
q22	389	342	339	339
Total cold run time: 118630 ms
Total hot run time: 40355 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4418	4269	4218	4218
q2	376	274	263	263
q3	3051	2865	2912	2865
q4	1946	1683	1725	1683
q5	5470	5522	5434	5434
q6	216	131	135	131
q7	2292	1800	1802	1800
q8	3258	3393	3427	3393
q9	8735	8714	8751	8714
q10	4134	3751	3817	3751
q11	617	491	501	491
q12	820	650	659	650
q13	17212	3136	3168	3136
q14	335	267	288	267
q15	520	495	493	493
q16	471	434	446	434
q17	1807	1504	1512	1504
q18	8045	7939	7781	7781
q19	1798	1532	1666	1532
q20	2057	1911	1876	1876
q21	5087	4997	4638	4638
q22	652	546	532	532
Total cold run time: 73317 ms
Total hot run time: 55586 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170646 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85969729de9e801a86dcc0f89404112b2a36644f, data reload: false

query1	934	388	377	377
query2	6448	2451	2223	2223
query3	6626	212	215	212
query4	19059	17377	17041	17041
query5	3616	465	464	464
query6	243	157	168	157
query7	4587	288	299	288
query8	323	288	285	285
query9	8493	2364	2340	2340
query10	561	307	279	279
query11	10448	9929	9934	9929
query12	109	83	88	83
query13	1651	371	363	363
query14	10111	7496	7012	7012
query15	238	188	195	188
query16	7997	276	274	274
query17	1889	567	524	524
query18	2068	285	277	277
query19	200	159	167	159
query20	90	84	86	84
query21	212	131	125	125
query22	4452	4117	4020	4020
query23	33915	33564	33554	33554
query24	11150	2914	2943	2914
query25	599	412	397	397
query26	753	164	157	157
query27	2306	332	328	328
query28	5907	2112	2108	2108
query29	903	668	636	636
query30	265	165	154	154
query31	959	773	777	773
query32	103	53	63	53
query33	840	299	293	293
query34	1052	475	476	475
query35	746	631	614	614
query36	1112	957	956	956
query37	140	67	83	67
query38	2994	2848	2801	2801
query39	908	838	835	835
query40	210	127	132	127
query41	57	51	52	51
query42	108	102	99	99
query43	676	565	534	534
query44	1195	739	739	739
query45	201	162	172	162
query46	1069	704	720	704
query47	1843	1753	1766	1753
query48	367	306	300	300
query49	832	416	423	416
query50	764	399	393	393
query51	6843	6754	6740	6740
query52	106	86	97	86
query53	354	282	283	282
query54	883	435	438	435
query55	73	75	73	73
query56	282	252	256	252
query57	1156	1050	1037	1037
query58	247	240	265	240
query59	3375	3239	3314	3239
query60	279	268	273	268
query61	89	90	93	90
query62	623	440	446	440
query63	353	285	285	285
query64	8554	2223	1712	1712
query65	3177	3099	3089	3089
query66	745	337	318	318
query67	15524	15057	14818	14818
query68	6171	568	543	543
query69	703	497	390	390
query70	1161	1158	1128	1128
query71	475	273	289	273
query72	8776	2752	2648	2648
query73	819	327	323	323
query74	5804	5546	5459	5459
query75	4299	2599	2658	2599
query76	3450	993	960	960
query77	744	303	296	296
query78	10291	9980	9853	9853
query79	4673	510	508	508
query80	993	464	468	464
query81	542	217	222	217
query82	897	107	101	101
query83	319	181	170	170
query84	267	98	88	88
query85	1211	341	286	286
query86	405	337	301	301
query87	3331	3094	3067	3067
query88	4546	2471	2452	2452
query89	506	386	373	373
query90	1868	187	187	187
query91	132	101	100	100
query92	59	49	50	49
query93	5356	501	503	501
query94	1145	194	184	184
query95	401	305	305	305
query96	610	268	268	268
query97	3247	3013	3013	3013
query98	216	200	197	197
query99	1139	842	846	842
Total cold run time: 279548 ms
Total hot run time: 170646 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.68 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 85969729de9e801a86dcc0f89404112b2a36644f, data reload: false

query1	0.04	0.04	0.02
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.66	0.09	0.09
query5	0.53	0.48	0.48
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.52	0.48	0.50
query10	0.55	0.53	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.58	0.58
query14	0.77	0.78	0.79
query15	0.82	0.83	0.80
query16	0.35	0.37	0.38
query17	0.95	1.01	1.06
query18	0.21	0.27	0.23
query19	1.78	1.77	1.76
query20	0.01	0.00	0.01
query21	15.41	0.67	0.66
query22	4.07	7.72	2.03
query23	18.28	1.39	1.26
query24	2.17	0.24	0.22
query25	0.16	0.09	0.08
query26	0.25	0.18	0.17
query27	0.08	0.08	0.08
query28	13.23	1.01	1.00
query29	12.63	3.35	3.28
query30	0.25	0.06	0.05
query31	2.87	0.39	0.39
query32	3.26	0.47	0.47
query33	2.83	2.86	2.96
query34	17.07	4.37	4.43
query35	4.48	4.47	4.46
query36	0.65	0.48	0.48
query37	0.20	0.16	0.16
query38	0.16	0.14	0.15
query39	0.05	0.03	0.04
query40	0.17	0.14	0.14
query41	0.10	0.04	0.05
query42	0.05	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 109.06 s
Total hot run time: 30.68 s

morningman
morningman approved these changes Jun 27, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 27, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

JNSimba
JNSimba approved these changes Jun 28, 2024
View reviewed changes
Copy link
Member

@JNSimba JNSimba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zy-kkk zy-kkk merged commit bafbb94 into apache:master Jun 28, 2024
@zy-kkk zy-kkk deleted the jdbc_connection_pool_default_max branch June 28, 2024 12:13
zy-kkk added a commit to zy-kkk/doris that referenced this pull request Jun 28, 2024
@zy-kkk
[improvement](jdbc catalog) Modify the maximum number of connections …
3912e37 
…in the connection pool to 30 by default (apache#36720)

In many cases, we found that users would use JDBC Catalog to perform a
large number of queries, which resulted in the maximum of 10 connections
being insufficient, so I adjusted it to 30, which covered most needs.
@zy-kkk zy-kkk mentioned this pull request Jun 28, 2024
Merged
zy-kkk added a commit to zy-kkk/doris that referenced this pull request Jun 28, 2024
@zy-kkk
[improvement](jdbc catalog) Modify the maximum number of connections …
c2c934f 
…in the connection pool to 30 by default (apache#36720)

In many cases, we found that users would use JDBC Catalog to perform a
large number of queries, which resulted in the maximum of 10 connections
being insufficient, so I adjusted it to 30, which covered most needs.
@zy-kkk zy-kkk mentioned this pull request Jun 28, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jun 30, 2024
@zy-kkk @dataroaring
[improvement](jdbc catalog) Modify the maximum number of connections …
1cbc322 
…in the connection pool to 30 by default (#36720)

In many cases, we found that users would use JDBC Catalog to perform a
large number of queries, which resulted in the maximum of 10 connections
being insufficient, so I adjusted it to 30, which covered most needs.
morningman pushed a commit that referenced this pull request Jul 1, 2024
@zy-kkk
[branch-2.1][improvement](jdbc catalog) Modify the maximum number of …
62c4451 
…connections in the connection pool to 30 by default (#37023)

pick (#36720)

In many cases, we found that users would use JDBC Catalog to perform a
large number of queries, which resulted in the maximum of 10 connections
being insufficient, so I adjusted it to 30, which covered most needs.
morningman pushed a commit that referenced this pull request Jul 1, 2024
@zy-kkk
[branch-2.0][improvement](jdbc catalog) Modify the maximum number of …
8802608 
…connections in the connection pool to 30 by default (#37024)

pick (#36720)

In many cases, we found that users would use JDBC Catalog to perform a
large number of queries, which resulted in the maximum of 10 connections
being insufficient, so I adjusted it to 30, which covered most needs.
@xiaokang xiaokang mentioned this pull request Jul 14, 2024
Closed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@zy-kkk
[branch-2.0][improvement](jdbc catalog) Modify the maximum number of …
c948c17 
…connections in the connection pool to 30 by default (apache#37024)

pick (apache#36720)

In many cases, we found that users would use JDBC Catalog to perform a
large number of queries, which resulted in the maximum of 10 connections
being insufficient, so I adjusted it to 30, which covered most needs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@JNSimba JNSimba JNSimba approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.5-merged dev/3.0.0-merged kind/behavior-changed reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@zy-kkk @doris-robot @morningman @JNSimba @dataroaring