Skip to content

[fix](s3-client) Fix s3-client max-connection default value which cause s3 to be slow and unable to handle high concurrency in many-files scenarios. #58511

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Dec 2, 2025
Merged

[fix](s3-client) Fix s3-client max-connection default value which cause s3 to be slow and unable to handle high concurrency in many-files scenarios. #58511

morningman merged 1 commit into from
Dec 2, 2025

Conversation

@kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented Nov 28, 2025 •
edited by morningman
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #51825, #58514

Problem Summary:

Release note

Fix s3-client max-connection default value introduced by #51825 which cause s3 to be slow and unable to handle high concurrency in many-files scenarios.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@kaka11chen
[fix](s3-client) Fix s3-client max-connection default value which cau…
4b64a17 
…ses s3 to be slow and unable to handle high concurrency in multi-file scenarios.
@Thearas
Copy link
Contributor

Thearas commented Nov 28, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen kaka11chen mentioned this pull request Nov 28, 2025
Merged
16 tasks
morningman pushed a commit that referenced this pull request Nov 28, 2025
@kaka11chen
[fix](s3-client) Fix s3-client max-connection default value which cau…
85af0ca 
…ses s3 to be slow and unable to handle high concurrency in multi-file scenarios. (#58512)

### What problem does this PR solve?

Problem Summary:

### Release note

Cherry-pick #58511 

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@kaka11chen kaka11chen changed the title [fix](s3-client) Fix s3-client max-connection default value which cause s3 to be slow and unable to handle high concurrency in multi-file scenarios. [fix](s3-client) Fix s3-client max-connection default value which cause s3 to be slow and unable to handle high concurrency in many-files scenarios. Nov 28, 2025
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Nov 28, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Nov 28, 2025

TPC-H: Total hot run time: 34118 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4b64a17face758bdf3661ff11629f71ae3e37112, data reload: false

------ Round 1 ----------------------------------
q1	17696	4974	4870	4870
q2	2030	307	207	207
q3	10261	1296	737	737
q4	10232	902	373	373
q5	7513	2458	2205	2205
q6	185	171	140	140
q7	922	773	645	645
q8	9338	1335	1013	1013
q9	6948	5282	5313	5282
q10	6865	2243	1811	1811
q11	518	311	287	287
q12	336	366	225	225
q13	17796	3681	3070	3070
q14	234	249	221	221
q15	591	514	526	514
q16	898	857	809	809
q17	583	712	530	530
q18	7420	7109	7026	7026
q19	1157	943	569	569
q20	353	340	240	240
q21	2939	3203	2407	2407
q22	1024	988	937	937
Total cold run time: 105839 ms
Total hot run time: 34118 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4955	4919	4910	4910
q2	326	417	333	333
q3	2118	2681	2302	2302
q4	1329	1722	1258	1258
q5	4212	4183	4489	4183
q6	226	171	132	132
q7	2063	1944	1860	1860
q8	2577	2515	2495	2495
q9	7482	7416	7396	7396
q10	3067	3258	2819	2819
q11	584	523	511	511
q12	711	732	649	649
q13	3566	3912	3332	3332
q14	309	309	294	294
q15	580	513	501	501
q16	890	947	880	880
q17	1139	1390	1437	1390
q18	7800	7650	7691	7650
q19	765	773	777	773
q20	2026	2053	1879	1879
q21	4652	4332	4126	4126
q22	1062	1090	980	980
Total cold run time: 52439 ms
Total hot run time: 50653 ms

@doris-robot
Copy link

doris-robot commented Nov 28, 2025

TPC-DS: Total hot run time: 181197 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4b64a17face758bdf3661ff11629f71ae3e37112, data reload: false

query1	1036	407	385	385
query2	6616	1174	1171	1171
query3	6754	234	227	227
query4	25086	23352	22812	22812
query5	5038	648	513	513
query6	343	242	229	229
query7	4658	511	313	313
query8	314	244	243	243
query9	8732	2582	2600	2582
query10	558	362	322	322
query11	15254	14827	14610	14610
query12	185	120	118	118
query13	1705	596	445	445
query14	9331	5997	5966	5966
query15	209	201	185	185
query16	7634	700	532	532
query17	1259	788	652	652
query18	2043	443	347	347
query19	215	214	186	186
query20	135	129	123	123
query21	224	136	119	119
query22	3867	4049	3832	3832
query23	32863	31971	31809	31809
query24	8548	2442	2446	2442
query25	643	576	482	482
query26	1254	288	180	180
query27	2677	509	350	350
query28	4294	2142	2131	2131
query29	839	661	531	531
query30	314	246	215	215
query31	815	716	608	608
query32	86	84	80	80
query33	618	408	350	350
query34	847	860	562	562
query35	809	878	735	735
query36	870	929	828	828
query37	125	114	85	85
query38	3828	3886	3724	3724
query39	1481	1588	1412	1412
query40	226	128	123	123
query41	87	64	62	62
query42	127	120	114	114
query43	435	446	411	411
query44	1317	763	776	763
query45	214	202	190	190
query46	888	998	650	650
query47	1681	1726	1654	1654
query48	393	440	321	321
query49	756	510	412	412
query50	671	694	414	414
query51	3904	3956	3795	3795
query52	111	120	107	107
query53	241	273	198	198
query54	317	305	292	292
query55	98	95	97	95
query56	323	330	316	316
query57	1101	1184	1076	1076
query58	296	290	277	277
query59	2399	2396	2297	2297
query60	362	363	339	339
query61	160	166	163	163
query62	774	718	658	658
query63	231	196	198	196
query64	4555	1244	905	905
query65	4063	3952	4002	3952
query66	1133	451	342	342
query67	15236	14880	14670	14670
query68	8377	965	626	626
query69	547	349	307	307
query70	1108	1017	1010	1010
query71	485	342	323	323
query72	5933	4805	4847	4805
query73	711	563	351	351
query74	8844	8734	8697	8697
query75	3588	3023	2541	2541
query76	3804	1130	732	732
query77	812	415	329	329
query78	9527	9770	8980	8980
query79	2244	805	576	576
query80	642	574	492	492
query81	538	271	238	238
query82	524	146	111	111
query83	265	260	251	251
query84	270	115	95	95
query85	1000	495	458	458
query86	397	286	294	286
query87	4075	4041	4028	4028
query88	4180	2261	2266	2261
query89	401	331	300	300
query90	1914	222	217	217
query91	192	169	147	147
query92	83	69	67	67
query93	1935	981	654	654
query94	673	454	335	335
query95	496	422	413	413
query96	511	544	292	292
query97	2680	2710	2565	2565
query98	240	212	222	212
query99	1329	1397	1270	1270
Total cold run time: 272531 ms
Total hot run time: 181197 ms

@doris-robot
Copy link

doris-robot commented Nov 28, 2025

ClickBench: Total hot run time: 27.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4b64a17face758bdf3661ff11629f71ae3e37112, data reload: false

query1	0.05	0.04	0.05
query2	0.10	0.05	0.05
query3	0.25	0.09	0.09
query4	1.60	0.11	0.11
query5	0.26	0.26	0.26
query6	1.17	0.64	0.64
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.58	0.51	0.50
query10	0.56	0.56	0.57
query11	0.15	0.10	0.11
query12	0.15	0.11	0.11
query13	0.64	0.60	0.62
query14	1.00	1.00	0.99
query15	0.81	0.79	0.82
query16	0.40	0.40	0.41
query17	1.06	1.08	1.08
query18	0.23	0.21	0.21
query19	1.92	1.81	1.84
query20	0.02	0.01	0.01
query21	15.46	0.27	0.14
query22	5.00	0.05	0.05
query23	16.15	0.27	0.10
query24	0.94	0.68	0.68
query25	0.11	0.09	0.11
query26	0.13	0.14	0.14
query27	0.06	0.06	0.05
query28	4.79	1.21	1.03
query29	12.58	3.86	3.30
query30	0.28	0.15	0.12
query31	2.81	0.61	0.40
query32	3.23	0.54	0.47
query33	3.00	3.12	2.98
query34	16.72	5.20	4.48
query35	4.53	4.60	4.53
query36	0.66	0.49	0.49
query37	0.11	0.06	0.06
query38	0.06	0.04	0.04
query39	0.05	0.03	0.03
query40	0.17	0.14	0.13
query41	0.09	0.04	0.03
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 98.05 s
Total hot run time: 27.81 s

@doris-robot
Copy link

doris-robot commented Nov 28, 2025

BE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.88% (18503/34989)
Line Coverage 38.42% (169487/441144)
Region Coverage 33.22% (131597/396097)
Branch Coverage 34.17% (56660/165795)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 28, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.66% (24585/34307)
Line Coverage 58.17% (256383/440751)
Region Coverage 53.33% (213816/400909)
Branch Coverage 54.79% (91331/166692)

morningman
morningman approved these changes Dec 2, 2025
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 2, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 2, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 2, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit 681fdd3 into apache:master Dec 2, 2025
35 of 37 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 2, 2025
@kaka11chen
[fix](s3-client) Fix s3-client max-connection default value which cau…
58b2ba6 
…se s3 to be slow and unable to handle high concurrency in many-files scenarios. (#58511)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #51825, #58514

Problem Summary:

### Release note

Fix s3-client max-connection default value introduced by #51825 which
cause s3 to be slow and unable to handle high concurrency in many-files
scenarios.
@github-actions github-actions bot mentioned this pull request Dec 2, 2025
Closed
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Dec 2, 2025
@kaka11chen
[fix](s3-client) Fix s3-client max-connection default value which cau…
cd90d9c 
…se s3 to be slow and unable to handle high concurrency in many-files scenarios. (apache#58511)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: apache#51825, apache#58514

Problem Summary:

### Release note

Fix s3-client max-connection default value introduced by apache#51825 which
cause s3 to be slow and unable to handle high concurrency in many-files
scenarios.
@kaka11chen kaka11chen mentioned this pull request Dec 2, 2025
Merged
16 tasks
yiguolei pushed a commit that referenced this pull request Dec 3, 2025
@kaka11chen
[fix](s3-client) Fix s3-client max-connection default value which cau…
fc6f8ea 
…se s3 to be slow and unable to handle high concurrency in many-files scenarios. (#58511) (#58626)

### What problem does this PR solve?

Related PR: #51825, #58514

Problem Summary:

### Release note

Cherry-pick #58511 

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

+1 more reviewer

@hubgeter hubgeter hubgeter approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@kaka11chen @Thearas @doris-robot @hello-stephen @morningman @hubgeter @yiguolei