Beyond a million records per dataset

datakumi isn't designed for transactional or event data in the millions. Expect each write and each first look at a new version to take several seconds, and memory to grow in proportion. Keep such data in the lakehouse and manage its reference data here.