Prometheus metrics

With --enable-metrics, Hardcharger™ collects runtime metrics and serves them in the Prometheus text format at GET /metrics on --metrics-port (9091), a separate plaintext port.

$ curl -s localhost:9091/metrics | grep -E '^hardcharger_(build_info|up_seconds|sessions\{)' | head -3
hardcharger_build_info{…} 1
hardcharger_up_seconds …
hardcharger_sessions{state="…"} …

Keep the port on a private network: --metrics-bind-address (default 0.0.0.0) picks the interface, and --metrics-port 0 turns the HTTP endpoint off while keeping the SQL view.

In SQL

The same numbers are in hardcharger_metrics(), for administrators and dashboards that speak SQL:

$ HARDCHARGER_PASSWORD='change-me' hardcharger --host localhost --tls=false --username hardcharger --csv --command \
    "SELECT name, labels['state'] AS state FROM hardcharger_metrics() WHERE name = 'hardcharger_queue_depth' ORDER BY state"
name,state
hardcharger_queue_depth,active
hardcharger_queue_depth,queued

Scrape it

scrape_configs:
  - job_name: hardcharger
    scrape_interval: 15s
    static_configs:
      - targets: ["hardcharger-0.hardcharger:9091", "hardcharger-1.hardcharger:9091"]

In Kubernetes with the Prometheus Operator, a PodMonitor on the metrics port does the same (Docker and Kubernetes).

What to alert on

Alert when	Metric
The server restarted uncleanly	hardcharger_last_exit_unclean == 1, or increase(hardcharger_unclean_exits_total[1h]) > 0
Health checks fail	hardcharger_health_check_status == 0
Statements are being turned away	rate(hardcharger_queue_rejected_total[5m]) > 0
Statements wait long for a slot	histogram_quantile(0.95, rate(hardcharger_queue_wait_seconds_bucket[5m]))
Errors climb	rate(hardcharger_statements_total{status="error"}[5m])
Memory nears DuckDB’s limit	hardcharger_duckdb_memory_usage_bytes / hardcharger_duckdb_memory_limit_bytes > 0.9
Sessions near the cap	sum(hardcharger_sessions) / hardcharger_session_limit > 0.9 (when max_sessions is set)
Metrics themselves stopped	hardcharger_metrics_collection_success == 0, or the scrape is missing (up == 0)

Every metric

Generated from a running server’s /metrics:

Metric	Type	Labels	Meaning
hardcharger_build_info	gauge	duckdb_version, server, version	Server build: version, DuckDB version.
hardcharger_concurrency_limit	gauge		max_concurrent_statements (0: no queue).
hardcharger_draining	gauge		1 while the server shuts down.
hardcharger_duckdb_memory_limit_bytes	gauge		DuckDB memory_limit in bytes.
hardcharger_duckdb_memory_usage_bytes	gauge		Memory DuckDB's buffer manager holds.
hardcharger_duckdb_spill_bytes	gauge		Size of DuckDB's temporary (spill) files now.
hardcharger_duckdb_spill_files	gauge		DuckDB's temporary (spill) files now.
hardcharger_duckdb_temp_storage_bytes	gauge		Temporary storage DuckDB holds for evicted buffers, from duckdb_memory().
hardcharger_duckdb_threads	gauge		DuckDB threads setting.
hardcharger_health_check_duration_seconds	gauge		How long the last health check took.
hardcharger_health_check_status	gauge		1 when the health check passes.
hardcharger_host_cpus	gauge		Logical CPUs of the host.
hardcharger_host_memory_bytes	gauge		Physical memory of the host.
hardcharger_last_exit_unclean	gauge		1 if the last run on this database did not stop cleanly (NaN: unknown).
hardcharger_metrics_collection_duration_seconds	gauge		How long the last collection took.
hardcharger_metrics_collection_success	gauge		1 when the last collection succeeded.
hardcharger_metrics_last_collection_success_seconds	gauge		Unix time of the last successful collection.
hardcharger_queue_admin_bypass_total	counter		Statements that skipped the queue.
hardcharger_queue_depth	gauge	state	Statements holding or waiting for a queue slot.
hardcharger_queue_rejected_total	counter		Statements the queue refused or timed out.
hardcharger_queue_wait_seconds	histogram		Time statements waited in the queue.
hardcharger_session_limit	gauge		max_sessions (0: unlimited).
hardcharger_sessions	gauge	state	Open client sessions, by state.
hardcharger_sessions_opened_total	counter		Client sessions opened.
hardcharger_sessions_reaped_total	counter		Client sessions ended for being idle or killed.
hardcharger_statement_duration_seconds	histogram	kind	Client statement run time, by kind.
hardcharger_statements_active	gauge		Statements running now.
hardcharger_statements_total	counter	status	Client statements finished, by status.
hardcharger_unclean_exits_total	counter		Runs on this database that did not stop cleanly.
hardcharger_up_seconds	gauge		Seconds since the server started.
process_open_fds	gauge		Open file descriptors of the server process.
process_resident_memory_bytes	gauge		Resident memory of the server process.
process_start_time_seconds	gauge		Unix time the server process started.
process_threads	gauge		Operating-system threads of the server process.