Measured
datakumi is built for master and reference data: products, suppliers, cost centres, code lists and mappings. These typically run from hundreds to a few hundred thousand records per dataset. Several operations read the whole dataset, so this page says what they cost as a dataset grows. The numbers are measured, not estimated.
Measured with python -m scripts.benchmark_scale on Docker Desktop on a Windows laptop (8 CPUs, 8 GB for Docker), with PostgreSQL and MinIO in containers on the same machine. Each dataset has 10 fields (text, numbers, a date), a list rule, required fields and one field that doesn't allow duplicates.
| Operation | 10,000 records | 100,000 | 500,000 |
|---|---|---|---|
| Create the dataset from a file (validate and write) | 0.3 s | 0.7 s | 2.2 s |
| Open a grid page, sorted and filtered | 0.05 s | 0.08 s | 0.24 s |
| Publish 100 changed records (validate, merge) | 0.4 s | 0.7 s | 1.5 s |
| Issues and health (every rule, every record) | 0.05 s | 0.09 s | 0.27 s |
| Each record's validity history, for "as of" (two versions) | 0.06 s | 0.2 s | 1.0 s |
| A Parquet or CSV copy (published copies, dbt seeds) | under 0.1 s | under 0.1 s | under 0.1 s |
| Record history, prepared once after the dataset is created | 0.5 s | 3.3 s | 18 s |
| Record history of each later version (100 changed records) | 0.2 s | 0.2 s | 0.5 s |
| The dataset in memory | 1 MB | 7 MB | 37 MB |