feat(arc): add ClickBench results for Arc on c6a.4xlarge #634

xe-nvdk · 2025-10-06T15:19:27Z

Hey everyone,

We’re the new folks in the neighborhood, sharing ClickBench results for Arc, our time-series warehouse that’s launching soon.
I’ve made sure everything follows the benchmark requirements, but happy to adjust if needed.

Appreciate your work on this project!
– Ignacio

CLAassistant · 2025-10-06T15:19:34Z

All committers have signed the CLA.

arc/benchmark.sh

arc/run.sh

xe-nvdk

We are going to push a new update of this PR in a few minutes. Thank you for marking the issues.

arc/run.sh

arc/benchmark.sh

…nd c6a.4xlarge in aws

arc/benchmark.sh

xe-nvdk · 2025-10-07T15:37:41Z

Just updated the files and make it public the repo. Thanks.

rschu1ze · 2025-10-07T19:13:02Z

arc/benchmark.sh

+
+# Install Python and dependencies
+echo "Installing dependencies..."
+pip3 install fastapi uvicorn duckdb pyarrow requests gunicorn


This requires running pip with --break-system-packages.

Would it be possible to create a Python venv? See e.g. chdb/benchmark.sh for an example.

Yep, we have in our start.sh in the repo, I'm adding to this script.

rschu1ze · 2025-10-07T19:13:55Z

arc/benchmark.sh

+
+# Create API token for benchmark
+python3 << EOF
+from api.auth import AuthManager, Permission


I got the next error here:

Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: cannot import name 'Permission' from 'api.auth' (/data/ClickBench/arc/arc/api/auth.py)

I checked, there is indeed no Permission class in file auth.py.

Uff, thank you for this, its old code, in our repo we have this right. Let me update it here too.

alexey-milovidov · 2025-10-11T17:04:28Z

arc/README.md

+## Prerequisites
+
+- Ubuntu/Debian Linux (or compatible)
+- Python 3.11+


There should be no prerequisites - the benchmark runs automatically on an empty AWS machine with Ubuntu AMI.

Thanks for the feedback. We’ll revisit the submission later this year. For now, we’re happy to have the benchmark numbers internally and will use them for our own reference. Once we release official binaries, we’ll try again to get included in ClickBench.

It's not a problem, let's push this PR to ClickBench. The more systems included, the better.

Hi @alexey-milovidov we just updated, we were able to run the benchmark.sh according to clickbench guidelines. Let me know if you have issues running, but shouldn't have any. Thank you.

alexey-milovidov · 2025-10-12T10:30:01Z

No success so far:

Running ClickBench queries via Arc HTTP API...
================================================
Checking if Arc is running at http://localhost:8000...
Arc is running. Using parquet file: /ClickBench/arc/hits.parquet
Running 43 queries via Arc HTTP API...
Query 1 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 2 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 3 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 4 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 5 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 6 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 7 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 8 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 9 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 10 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 11 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 12 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 13 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 14 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 15 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 16 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 17 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 18 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 19 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 20 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 21 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 22 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 23 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 24 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 25 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 26 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 27 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 28 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 29 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 30 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 31 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 32 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 33 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 34 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 35 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 36 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 37 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 38 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 39 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 40 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 41 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 42 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Query 43 failed: 401 - {"error":"Unauthorized","detail":"Invalid or missing API token"}
Benchmark complete!

alexey-milovidov · 2025-10-12T10:34:07Z

However, it did something before:

Creating API token...
Created API token: EdodzXfV99KRO-0XoWONWxxUs0AGH2HqjcNRq4c4rfg
Token created successfully

xe-nvdk · 2025-10-13T20:56:35Z

Ok, Thanks, this should be good now.

alexey-milovidov · 2025-10-14T15:05:04Z

This still does not fit the format. It prints:

Formatting results...

✓ Benchmark complete!

Results saved to: results.json
Logs saved to: log.txt

To view results:
  cat results.json

But it does not run the actual command cat results.json to put results in the log.

alexey-milovidov · 2025-10-14T15:05:52Z

This looks contradictory:

Verifying query cache configuration...
======================================================================
Query Cache Configuration Check
======================================================================
✓ arc.conf:     enabled = True
✓ .env:         QUERY_CACHE_ENABLED = false
  Environment:  QUERY_CACHE_ENABLED not set

⚠️  FINAL RESULT: Query cache is ENABLED
    TTL: 60s
    Max size: 100
======================================================================

alexey-milovidov · 2025-10-14T15:06:38Z

This is not compliant:

Dataset size:
-rw-r--r-- 1 root root 14G Jun 25  2022 hits.parquet

xe-nvdk · 2025-10-14T15:10:40Z

This still does not fit the format. It prints:
Formatting results...

✓ Benchmark complete!

Results saved to: results.json
Logs saved to: log.txt

To view results:
  cat results.json
But it does not run the actual command cat results.json to put results in the log.

Ok. I thought that you were mentioning the post results for Arc, not what was printed.

xe-nvdk · 2025-10-14T15:11:19Z

This is not compliant:

Dataset size:
-rw-r--r-- 1 root root 14G Jun 25  2022 hits.parquet

What do you mean? is what is downloaded and saved in the data folder to query through the query http endpoint.

xe-nvdk · 2025-10-14T15:14:14Z

This looks contradictory:

Verifying query cache configuration...
======================================================================
Query Cache Configuration Check
======================================================================
✓ arc.conf:     enabled = True
✓ .env:         QUERY_CACHE_ENABLED = false
  Environment:  QUERY_CACHE_ENABLED not set

⚠️  FINAL RESULT: Query cache is ENABLED
    TTL: 60s
    Max size: 100
======================================================================

But if you see final result is what is important.

Looks like that we are playing to find any specific small detail to not keep moving forward with this and as I said, I respect what you guys built here, but I'm not, at least for now keep chasing this. We have the numbers for our internal references, for now is what is matter, if somebody want to replicate it can use what we have in our fork and the number can easy be replicated.

Thank you.

alexey-milovidov · 2025-10-14T17:58:39Z

No need to close the PR. I can help with merging it.
We will need to remove all the AI slop, and it will be alright.

xe-nvdk · 2025-10-14T18:07:07Z

Ok, let me go through this in a few days.

Two fixes to benchmark.sh based on PR feedback: 1. Actually output results.json content to log - Changed from showing "To view results: cat results.json" - Now runs `cat results.json` directly so results appear in log - Makes CI logs and benchmark runs more useful 2. Remove contradictory checkmarks in cache configuration check - Was showing ✓ for both arc.conf (enabled=True) AND .env (enabled=false) - Now shows config sources as informational only (no checkmarks) - Only final result gets status indicator: * ✓ for cache disabled (good for benchmarks) * ✗ for cache enabled (with warning) - Clearer indication of actual runtime behavior Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude <[email protected]>

xe-nvdk · 2025-10-16T12:01:36Z

I think that we have it @alexey-milovidov, Can you check and let me know? now its print everything and print the results on screen. Thank you!

Also, I deleted the cache enabled results, we are going to submit those in a different folder, like arc-query-cache, unless that you recommend something different.

alexey-milovidov · 2025-10-16T22:53:13Z

It works! I will run it on every machine type...

xe-nvdk · 2025-10-16T23:14:24Z

Thank you! We are going to add more systems too! Let me know any feedback that you have. Thanks again!

alexey-milovidov · 2025-10-26T03:37:34Z

@xe-nvdk, something is wrong: #658

xe-nvdk added 2 commits October 6, 2025 12:00

adding arc values

29e86a5

we missed one query, now is complete

a433b23

rschu1ze reviewed Oct 6, 2025

View reviewed changes

arc/benchmark.sh Outdated Show resolved Hide resolved

arc/benchmark.sh Show resolved Hide resolved

arc/run.sh Outdated Show resolved Hide resolved

rschu1ze self-assigned this Oct 6, 2025

xe-nvdk commented Oct 6, 2025

View reviewed changes

arc/run.sh Outdated Show resolved Hide resolved

arc/benchmark.sh Show resolved Hide resolved

arc/benchmark.sh Outdated Show resolved Hide resolved

fixing run.sh and re run, just in case both benchmark in pro m3 max a…

01b692e

…nd c6a.4xlarge in aws

This comment was marked as resolved.

Sign in to view

disabling query caching and re ran the benchmarks

b663892

rschu1ze reviewed Oct 6, 2025

View reviewed changes

arc/benchmark.sh Outdated Show resolved Hide resolved

updating repo to match the current for arc

7a40588

Merge branch 'main' into main

b934055

rschu1ze reviewed Oct 7, 2025

View reviewed changes

xe-nvdk and others added 4 commits October 9, 2025 07:11

Merge branch 'ClickHouse:main' into main

fa56ed3

Merge branch 'ClickHouse:main' into main

7c0ccab

adding updated values for m3 max

1db5924

Merge branch 'main' of github.com:Basekick-Labs/ClickBench

08fe758

alexey-milovidov reviewed Oct 11, 2025

View reviewed changes

xe-nvdk and others added 9 commits October 12, 2025 19:20

updating results and scripts for arc

bde45ce

Merge branch 'ClickHouse:main' into main

7135fff

fixing benchmark to load the data

3a00ca3

Merge branch 'main' of github.com:Basekick-Labs/ClickBench

757d7fa

fixing token creation

6e70633

fixing api env passing

32c62ba

fixing db specification for api creation

56702bc

making sure that we don't have enabled query cache

82abc81

adding results for arc in clickbench

d6904f8

xe-nvdk and others added 3 commits October 13, 2025 17:55

refining format of the results

ecd0414

Merge branch 'ClickHouse:main' into main

b905b50

Merge branch 'main' of github.com:Basekick-Labs/ClickBench

ad86bf5

xe-nvdk and others added 3 commits October 13, 2025 17:58

deleting comments in the results

97da2bd

adding time-series tag

716b715

Merge branch 'ClickHouse:main' into main

705c8bf

xe-nvdk closed this Oct 14, 2025

alexey-milovidov reopened this Oct 14, 2025

xe-nvdk and others added 5 commits October 15, 2025 20:14

Some fixes for results display, and print of caching status

a49a8ef

fixing and modifying things based on clickbench team

9a0b9b1

Merge branch 'main' of github.com:Basekick-Labs/ClickBench

229e53f

Merge branch 'ClickHouse:main' into main

9f3b46d

alexey-milovidov approved these changes Oct 16, 2025

View reviewed changes

alexey-milovidov assigned alexey-milovidov and unassigned rschu1ze Oct 16, 2025

alexey-milovidov merged commit d164185 into ClickHouse:main Oct 16, 2025

feat(arc): add ClickBench results for Arc on c6a.4xlarge #634

feat(arc): add ClickBench results for Arc on c6a.4xlarge #634

Uh oh!

Conversation

xe-nvdk commented Oct 6, 2025

Uh oh!

CLAassistant commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xe-nvdk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

xe-nvdk commented Oct 7, 2025

Uh oh!

rschu1ze Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

xe-nvdk Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

rschu1ze Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

xe-nvdk Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Oct 11, 2025

Choose a reason for hiding this comment

Uh oh!

xe-nvdk Oct 11, 2025

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Oct 12, 2025

Choose a reason for hiding this comment

Uh oh!

xe-nvdk Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov commented Oct 12, 2025

Uh oh!

alexey-milovidov commented Oct 12, 2025

Uh oh!

xe-nvdk commented Oct 13, 2025

Uh oh!

alexey-milovidov commented Oct 14, 2025

Uh oh!

alexey-milovidov commented Oct 14, 2025

Uh oh!

alexey-milovidov commented Oct 14, 2025

Uh oh!

xe-nvdk commented Oct 14, 2025

Uh oh!

xe-nvdk commented Oct 14, 2025

Uh oh!

xe-nvdk commented Oct 14, 2025

Uh oh!

alexey-milovidov commented Oct 14, 2025

Uh oh!

xe-nvdk commented Oct 14, 2025

Uh oh!

xe-nvdk commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexey-milovidov commented Oct 16, 2025

Uh oh!

xe-nvdk commented Oct 16, 2025

Uh oh!

alexey-milovidov commented Oct 26, 2025

Uh oh!

Reviewers

Assignees

CLAassistant commented Oct 6, 2025 •

edited

Loading

xe-nvdk Oct 7, 2025 •

edited

Loading

xe-nvdk commented Oct 16, 2025 •

edited

Loading