Parquet/CSV logs on disk
Data / Knowledge library
DuckDB
in-process sql analytics over tables and files
Why learn DuckDB?
Useful for local analytics, embedded reporting, and exploring columnar files. It is not a drop-in replacement for every transactional server workload.
Where it came from
DuckDB is an in-process analytical database designed for SQL workloads. It can query data from files and application data structures without requiring a separate database server.
Build your understanding
Work through these exercises in order. Try the task before opening a reference.
- Query a CSV directly with SQL.
- Aggregate a Parquet dataset.
- Inspect the query execution plan.
- Compare a file scan with a persisted table.
- Measure memory and runtime for a realistic analytical query.
Documentation and sources
- Official documentation (duckdb.org)
Catalog research notes
- Research snapshot: September 9, 2026. This entry summarizes the linked primary documentation; it does not assert a latest release version.
Connect the ideas
Built on
Explore next
Real World Example
In-Process Financial Log Analytics
Financial analysts performing ad-hoc risk assessments on high-volume transaction logs.
The business problem
Traditional client-server databases introduce significant latency due to network overhead and serialization costs when analysts need to query multi-gigabyte transaction logs stored locally.
How this topic solves it
DuckDB provides an in-process analytical engine that embeds directly into the analyst's Python environment, allowing SQL queries to run against local files without the need for a separate database server or data movement.
- Install the DuckDB Python package within the local analysis environment.
- Point the engine to the directory containing raw transaction logs in Parquet or CSV format.
- Execute SQL queries directly against the files using the vectorized execution engine.
- Leverage the in-process architecture to perform joins and aggregations without exporting data to an external server.
Shows how DuckDB bypasses external servers to access local data.
Vectorized query processor
Python/R analysis script
User interface
- Data Files → Direct read → DuckDB Engine
- DuckDB Engine → Embedded execution → Host Process
- Host Process → Returns results → Analyst
Implementation example
This demonstrates the in-process capability where the engine reads files directly from the local filesystem into the Python environment.
import duckdb
# Query local files directly without importing data
results = duckdb.query("SELECT category, SUM(amount) FROM 'logs/*.parquet' GROUP BY category").df()Expected business outcome
Analysts achieve near-instantaneous query results on local datasets, eliminating the infrastructure maintenance and network bottlenecks associated with traditional database servers.
Limitations and checks
Because DuckDB runs embedded within the host process, it shares memory and CPU resources with the host application; heavy analytical queries may impact the performance of other tasks running in the same process, which can be monitored by observing system resource utilization during query execution.