The lake store (beta)
By default Dataflare keeps events in D1, which is exact, real-time and free-plan friendly. D1 databases have a size limit, though, so for high volumes Dataflare can stream events into Apache Iceberg tables in R2 and query them with Basin SQL (Cloudflare's DataFusion engine, formerly R2 SQL):
npx dataflare-cli init --lake
Status: beta. Pipelines and Basin SQL are themselves in beta, and the Basin SQL response format is not documented yet, so Dataflare parses it tolerantly (normaliseRows() in packages/worker/src/stores/lake.ts). Please report anything odd.
What it creates
- An R2 bucket
glass-datawith the data catalog enabled - Two Pipelines streams,
glass_eventsandglass_identities, with fixed schemas (packages/core/src/lake.ts) - Two sinks writing Iceberg tables
glass.eventsandglass.identities(Parquet, files rolled every 60 seconds) - Two pipelines connecting them (
INSERT INTO <sink> SELECT * FROM <stream>) - Worker bindings
EVENTS_STREAMandIDENTITIES_STREAM, and the API token as theCF_API_TOKENsecret so the Worker can query Basin SQL
D1 still holds projects, people, identity links, flags, experiments and the replay index.
Differences from D1 mode
| D1 | Lake | |
|---|---|---|
| Freshness | Immediate | About 1–2 minutes (file roll interval) |
| History | Until D1's size limit | Unbounded |
| Cost | Included in Workers plans up to limits | R2 storage, Pipelines and Basin SQL scans |
| Person properties | At event time, or current via SQL join | At event time |
| Deleting a person's events | Yes | Not yet (roadmap) |
| Other engines | Any Iceberg reader (DuckDB, Spark, Snowflake) can read the same tables |
Using a narrower token
The Worker only needs to read in lake mode. After init, you can replace its token with one that has just Workers R2 SQL › Read and Workers R2 Data Catalog › Read:
npx wrangler secret put CF_API_TOKEN --config .glass/wrangler.jsonc
Schema changes
Pipelines streams can't be altered after creation, so the stream schemas are versioned (LAKE_VERSION in packages/core/src/lake.ts). Custom properties never need a schema change: they live in the props JSON column. A new built-in column would need new streams (glass_events_v2) and queries over both tables; that migration path isn't built yet and is tracked in the roadmap.