Run it with Docker

Running datakumi with Docker, where its data lives, storage, and production operations.

Needs Docker with Compose. Everything (the app, MinIO, and a readable copy of the lake) starts with one command:

cp .env.example .env          # then set MDM_SEED_DEMO=true for the demo, or MDM_ADMIN_EMAIL/PASSWORD for a clean start
docker compose up -d --build  # app at http://localhost:8000 (API docs at /docs once signed in), MinIO console at :9001

On the very first start the app either loads the demo (MDM_SEED_DEMO=true) or creates the first admin from MDM_ADMIN_EMAIL / MDM_ADMIN_PASSWORD, who is asked to choose a new password at first sign-in. After that these settings are ignored. Demo accounts (admin, owner, editor, dk-editor, viewer @example.com) share the dev password in scripts/seed.py, which also gives each one an API token for the CLI.

Where the data is kept

The metadata (people, access, rules, dataset details, drafts, change sets, comments, audit) is in PostgreSQL, in the Docker volume mdm_postgres. Containers can be rebuilt or removed without losing it, but docker compose down -v deletes volumes, so don't use -v. Back it up with docker compose exec postgres pg_dump -U mdm mdm > mdm.sql, together with minio/.

Everything else lives on the host under MDM_DATA_DIR (set in .env, default ./data):

Lake layout

The bucket mirrors the catalog: each dataset is a Delta table at <top folder>/<sub folder>/.../<dataset>, with folder names turned into path names (Product master -> product_master); datasets outside any folder sit at the root.

lake/
  commercial/
    product_master/
      products/            _delta_log/ + *.parquet
      global_products/
  reference_data/
    currencies/
    markets/

Folder paths use each folder's technical name (Datasets > Folders in the sidebar), set from the display name when the folder is created; renaming the display name later doesn't move anything. Moving a dataset to another folder, changing a folder's technical name, or moving or deleting a folder, moves the affected Delta directories (all versions are kept, so history and time travel still work) and is recorded in the audit log. Changes that would put one dataset inside another's directory, or give two sibling folders the same path name, are refused. Each dataset's path is shown on its Settings tab. On start-up the app moves any table that isn't where its folder says it should be, e.g. tables from older versions stored under tables/<name>.

Storage: MinIO, Amazon S3, Azure or Google Cloud Storage

The lake starts in the bundled MinIO, or wherever MDM_LAKE_URI / MDM_S3_* point. Admins can connect it to other storage under Storage in the profile menu:

Each connection can keep the lake in a sub-folder (prefix) of the bucket or container. Setting up a connection works like this:

1. Test connection writes, reads back and deletes a small file, and counts how many of the catalog's datasets are already there.
2. Save and switch then offers two options:
   • Copy and switch: copies each missing dataset with its full history, checks that each copy reads back at the same version, and only then switches.
   • Switch only: for storage that already holds the same datasets.

Editing pauses while datasets are copied. If anything fails, nothing is switched. A dataset that already exists at the target with different data is refused rather than overwritten. Nothing is ever deleted: the old copies stay where they were.

How a saved connection is kept:

• Precedence: it overrides the environment variables, including after restarts, and the switch is recorded in the audit log.
• Encryption: credentials are encrypted in the metadata database and never sent back to the browser.
• Encryption key: set it with MDM_SECRET_KEY (any passphrase). Otherwise it is generated as mdm-secret.key in <MDM_DATA_DIR>/app/. Back the key up with the database. Without it, saved credentials can't be read: the app falls back to the environment settings and asks an admin to set the connection up again.

Two limits apply:

• Lake mirror: the readable lake/ folder only mirrors the bundled MinIO.
• Single writer: the app must be the only writer to the lake. It serialises writes per dataset in one process, which is what makes S3 commits safe without a separate lock service.

Published copies: Delta, Parquet or CSV for other platforms

The app's own tables stay Delta. Platforms that don't read Delta can get a copy of each dataset as Delta, Parquet or CSV. The copies go under a folder in the lake (_published by default) and follow every change, move and rename. An admin turns this on under Storage, below the lake connection; each dataset shows its copy, status and an Update copy button on its Settings tab. To see which format suits a platform, what each format keeps and how the files are laid out, read published-copies.md.

Copies can also carry each record's history (every version with _valid_from and _valid_to, for "as of" questions), and keep a renamed field's old name for a notice period (30 days by default), so pipelines have time to switch. Both are under Admin › Storage › Published copies.

Catalog registration and dbt seeds

Under Admin → Integrations, admins can keep two of the data team's systems in step with the datasets. Both are updated by a background job within about 15 seconds of each change. Failures are retried and admins are told. Tokens are encrypted like the lake connection.

• Catalog registration: each dataset becomes a table in Databricks Unity Catalog (<catalog>.<schema>.<dataset>, an external Delta table, through a SQL warehouse) or in the AWS Glue Data Catalog (<database>.<dataset>, read by Athena, EMR and Redshift Spectrum). The name is the dataset's technical name, so it doesn't change when the dataset moves to another folder (the registration follows the new path). The table carries the description, column descriptions, owner and tags. Register the dataset's own Delta table, or its Delta published copy (without the fields left out of copies). Databricks needs read access to the lake (an external location); Glue needs the lake in Amazon S3. In the catalog, its own grants decide who reads the table: datakumi's row and field limits apply in datakumi.
• dbt seeds: each dataset becomes <project>/<seeds folder>/<dataset>.csv with a .yml of column types (for Databricks, Snowflake, BigQuery, Postgres, Redshift or DuckDB), descriptions and tests from its rules (the key unique and not null, required fields not null, accepted values, relationships to other seeds and, for a parent field, to the dataset itself), and for a hierarchy level columns described with its level names. They're committed and pushed to the branch you choose, one commit per update. dbt builds them with dbt seed or dbt build. Datasets over the record limit (10,000 by default) are left out: seeds suit small reference tables (see scale.md). The token is sent per git command and is never stored in the working copy.

Moves need approval

Field changes (renaming a field, changing its type, deleting it) break the queries that read it, so on datasets that need approval (any approval mode but "publish directly") they're requests too. An admin or an owner of the dataset other than the requester approves them, unless the dataset lets owners approve their own changes. The request shows what the change affects and who reads the dataset (its consumers). It appears on the dataset's Approvals tab and under Approvals. A change that can't be made (values that don't convert, a field something depends on) is refused when it's asked for, not when it's approved.

A dataset's lake path is what downstream pipelines read. Any change to it is therefore a move request, which must be approved before any files move. An admin or an owner of every affected dataset can approve it. Owners may approve their own requests: the move still goes through the request, the path preview and the audit log. These changes are requests:

• moving a dataset to another folder (drag it in the sidebar, or change its folder on the Settings tab);
• moving a folder under another folder;
• changing a folder's or a dataset's technical name.

A request works like this:

• Before sending: the request dialog lists every path that changes (from → to) and takes a note for the approver.
• While waiting:
  • the request appears under Approvals → Waiting for me for everyone who can approve it, the requester included if they own what moves, and the others are notified;
  • the moving item shows a clock in the sidebar;
  • the requester can withdraw it.
• On approval: the layout is checked again and the move is applied all or nothing. Every dataset is copied and checked first, then the catalog switches, then the old copies are removed. If anything fails, nothing changes and the request is marked failed with the reason.
• Exceptions: changes that move no files, such as reordering or moving an empty folder, apply at once. A folder can only be deleted once it holds no datasets.
• Audit log: requests and decisions are recorded there.

Renaming

Owners change technical names on a dataset's Settings tab (the dataset's in Overview, a field's in its panel); admins change folder technical names from the folder settings icon next to a folder in the sidebar's dataset tree. Both are move requests, as above.

Restoring a dataset to a version from before a column was added or renamed is refused, because its rules describe today's columns.