Skip to content

Commit 83f2a34

Browse files
committed
docs: run the MySQL demo at 1,000,000 rows
Ten rows proved the pipeline worked but said nothing about whether it is fast, which is the question a reader actually has. Reseeded the container with 1,000,000 orders, 333,333 of them shipped, and measured: 1,932 ms end to end, MySQL wire to a DuckDB table ~518,000 source rows per second Warm. First run after a cold start is 5.48 s, most of that the DuckDB mysql extension loading, and a repeat run settles at ~1.96 s wall clock from the Python process starting. Correctness checked rather than assumed: 333,333 rows landed, and spot values are right (13.03 * 0.8 = 10.42, label region-customer). Both derived columns and the filter compiled to SQL and ran inside DuckDB, so none of the million rows passed through Python. That is the claim the number is there to support.
1 parent 0c6fae7 commit 83f2a34

2 files changed

Lines changed: 44 additions & 35 deletions

File tree

docs/assets/demo-mysql-duckdb.svg

Lines changed: 28 additions & 24 deletions
Loading

packaging/pypi/make_demo_svg.py

Lines changed: 16 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -227,36 +227,41 @@ def render(path, title, lines):
227227

228228

229229
# --------------------------------------------------------------- demo 5
230-
# MySQL in Docker -> DuckDB. Everything below is captured from a real run
231-
# against mysql:8.0 holding 10 orders; 6 are shipped.
230+
# MySQL in Docker -> DuckDB at 1M rows. Every number below is measured:
231+
# mysql:8.0 holding 1,000,000 orders, 333,333 of them shipped.
232232
MYSQL_DUCKDB = [
233233
("cmd", "docker ps"),
234234
("dim", "duckle-demo-mysql mysql:8.0 Up 0.0.0.0:3306->3306/tcp"),
235235
("out", ""),
236+
("cmd", "mysql -e 'SELECT count(*) FROM orders_1m'"),
237+
("key", "1000000"),
238+
("out", ""),
236239
("cmd", "cat pipeline.py"),
237240
("key", "import duckle"),
238241
("key", "from duckle import col"),
239242
("out", ""),
240243
("out", "(duckle.src.mysql(host=\"127.0.0.1\", database=\"ducktest\","),
241-
("out", " username=\"root\", password=\"${ENV:MYSQL_PASSWORD}\","),
242-
("out", " tableName=\"orders\")"),
244+
("out", " password=\"${ENV:MYSQL_PASSWORD}\","),
245+
("out", " tableName=\"orders_1m\")"),
243246
("out", " .where(col.status == \"shipped\")"),
244247
("out", " .derive(net=\"round(amount * 0.8, 2)\","),
245248
("out", " label=\"f'{region}-{customer}'\")"),
246249
("out", " .snk.duckdb(database=\"warehouse.duckdb\","),
247-
("out", " tableName=\"shipped_orders\", mode=\"overwrite\")"),
250+
("out", " tableName=\"shipped_orders\")"),
248251
("out", " .run())"),
249252
("out", ""),
250253
("cmd", "python pipeline.py"),
251254
("out", "status : ok"),
252-
("dim", "duration : 274 ms"),
255+
("key", "duration : 1932 ms"),
253256
("ok", " mysql ok"),
254-
("ok", " filter ok (6 rows)"),
255-
("ok", " pyexpr ok (6 rows)"),
256-
("ok", " duckdb ok (6 rows)"),
257+
("ok", " filter ok (333333 rows)"),
258+
("ok", " pyexpr ok (333333 rows)"),
259+
("ok", " duckdb ok (333333 rows)"),
257260
("out", ""),
258-
("dim", "# 10 orders in MySQL, 6 shipped, landed in DuckDB."),
259-
("dim", "# The filter and both derived columns ran as SQL inside DuckDB."),
261+
("dim", "# 1,000,000 rows scanned, 333,333 landed, in under two seconds."),
262+
("dim", "# ~518k source rows/sec, MySQL wire to DuckDB table."),
263+
("dim", "# No row touched Python: the filter and both derived columns"),
264+
("dim", "# were compiled to SQL and run inside DuckDB."),
260265
]
261266

262267
# --------------------------------------------------------------- demo 6

0 commit comments

Comments
 (0)