Hardcharger™ is fast for two reasons. DuckDB 2.0 runs the query, and on the queries derived from TPC-H1 it beats DuckDB 1.5 in-process before any network is involved. Then the result travels through an asynchronous pipeline in which no stage waits for another unless a queue is full.
flowchart LR D[(DuckDB 2.0<br/>parallel execution)] --> P[Produce<br/>Arrow batches,<br/>122,880 rows each] P --> Z[Compress<br/>zstd worker pool,<br/>when asked for] Z --> T[Encrypt and send<br/>TLS in 1 MiB writes,<br/>overlapped with the next] T --> C1[connection 1] T --> C2[connection 2] T --> CN[connection N]
🔁 Overlapped stages
DuckDB keeps producing into a buffer bounded by bytes (32 MiB) while earlier batches are encoded, compressed and encrypted. The socket never waits for the next batch to be built. With TLS on, one stream runs as fast as plain HTTP.
🧵 Parallel compression
When a client asks for zstd, a pool of threads compresses several batches at once, and the batches still leave in order.
🔐 TLS without a tax
The TLS stream encrypts a whole gathered write (up to 1 MiB) at once and sends it while it encrypts the next one, instead of one 16 KB record per socket write.
🔀 Partitioned results
A client can read a large result as several partitions over separate connections at once, past the limits of one TCP flow and one client core. Order is kept when the query asks for one. The drivers and the CLI ask for 8 by default; small results come as one stream.
📦 Big batches
Results go out in batches of up to 122,880 rows (one DuckDB row group), so per-batch overhead is negligible. A small query is still one batch and one round trip.
📥 Parallel ingest
Uploads are read from the socket in 1 MiB pieces and inserted on all of DuckDB’s threads, in one transaction. A failed load leaves nothing behind.
📤 Partitioned uploads
A large load can go up as several parts over separate connections at once, inserted from all of them in parallel and committed as one: all or nothing.
j of the result in partition j mod k, so a client rebuilds
the exact order; the benchmarks found ordered and unordered reads equally fast on a LAN.| You want | Do this |
|---|---|
| A large result faster over a fast network | Partitions: adbc.warp.statement.result_partitions (ADBC), resultPartitions (JDBC), --result-partitions (the CLI); each is 8 by default. On a fast LAN, 32 was fastest in the benchmarks; the server caps it at max_result_partitions (32). |
| A large load faster | Partitioned uploads, when the rows needn’t keep the upload’s order. |
| A large result faster over a slow or long link | zstd (adbc.warp.ipc_compression=zstd, JDBC ipcCompression=zstd) cuts the bytes to about a quarter. On a LAN it is slower than none, because the client decompresses on one core. Over high latency, prefer up to 8 partitions, in any order. |
| Long queries behind a proxy | Submit asynchronously (adbc.warp.statement.async, JDBC async=true) so no request outlives the proxy’s timeout. |
| Bounded memory | memory_limit, and max_concurrent_statements to queue rather than overload. |
| Smaller first batches | result_batch_rows: a full scan’s first batch takes about 21 ms at 122,880 rows and 2 ms at 2,048, but throughput drops with small batches. |
A tested example of reading partitions is in Warp SQL in one page.
Derived from the TPC-H benchmark; not an official TPC-H result and not comparable to published TPC-H results, as these results do not comply with the TPC-H specification. TPC-H™ is a trademark of the Transaction Processing Performance Council (TPC). See the benchmarks. ↩
Built by GizmoData™