# Dataflare documentation

> Open-source, agent-first analytics that runs in your own Cloudflare account: web analytics, product analytics (trends, funnels, retention), session replay with text transcripts, feature flags and A/B tests. Installed with one command and queried through MCP, a CLI or HTTP.

Install: `npx dataflare-cli init`

<!-- https://dataflare.dev/docs/getting-started/ -->

# Getting started

Dataflare deploys into your Cloudflare account with one command. You need a Cloudflare account and Node 20+.

## 1. Create an API token

At [dash.cloudflare.com/profile/api-tokens](https://dash.cloudflare.com/profile/api-tokens), create a custom token with:

| Permission                                                                        | Why                             |
| --------------------------------------------------------------------------------- | ------------------------------- |
| Account › Workers Scripts › Edit                                                  | Deploy the Dataflare Worker     |
| Account › D1 › Edit                                                               | Create and migrate the database |
| Account › Workers R2 Storage › Edit                                               | Store session replays           |
| Account › Account Settings › Read                                                 | Find your account               |
| Zone › Workers Routes › Edit                                                      | Only with `--route`             |
| Account › Pipelines › Edit, Workers R2 Data Catalog › Edit, Workers R2 SQL › Read | Only with `--lake`              |

The token is used by the CLI on your machine. It is not stored by Dataflare, except in lake mode, where it becomes a Worker secret so the Worker can query Basin SQL. (Use a separate read-only token there if you prefer; see [lake.md](lake.md).)

## 2. Run `dataflare init`

From your project's root:

```sh
CLOUDFLARE_API_TOKEN=… npx dataflare-cli init
```

It will:

1. Create a D1 database (`glass`) and an R2 bucket for replays (`glass-replays`). If R2 isn't enabled on your account yet, replay is skipped; enable R2 in the dashboard and run `dataflare upgrade` later.
2. Deploy the Dataflare Worker to `https://glass.<your-subdomain>.workers.dev`, apply migrations, and set its secrets (two API keys and a salt).
3. Create a project named after your `package.json` (override with `--project`).
4. Send a test event and check it arrived.
5. Print the install snippet for your framework, and write files for your coding agent: `AGENTS.md` (and `CLAUDE.md` if you have one), `.claude/skills/glass/SKILL.md` and `.mcp.json`.
6. Write `.github/workflows/glass.yml`, so later upgrades deploy from CI rather than from your machine (see [Upgrading](#upgrading)).

It also creates `.glass/` in your project. Commit it (`config.json`, `wrangler.jsonc`, `worker.js`, `migrations/`); `credentials.json` holds your keys and is git-ignored automatically.

Useful flags:

| Flag                   |                                                                                                                                              |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `--dry-run`            | Show what would be created, change nothing                                                                                                   |
| `--account <id>`       | Choose the account non-interactively                                                                                                         |
| `--project <id>`       | Project id (lowercase letters, digits, `-`, `_`)                                                                                             |
| `--eu`                 | Create the database and buckets in the EU jurisdiction                                                                                       |
| `--route acme.com`     | Also serve Dataflare from your own domain at `/_glass` (first-party, so ad blockers don't block it). Needs the domain on Cloudflare, proxied |
| `--no-route`           | Remove a route set by an earlier init                                                                                                        |
| `--retention-days <n>` | Delete events older than n days (also on `dataflare upgrade`)                                                                                |
| `--lake`               | Store events in Iceberg on R2 instead of D1 (beta; for high volume)                                                                          |
| `--no-replay`          | Skip replay storage                                                                                                                          |
| `--no-agent-files`     | Don't write AGENTS.md, SKILL.md or .mcp.json                                                                                                 |
| `--no-ci`              | Don't write `.github/workflows/glass.yml`                                                                                                    |
| `--json`               | Machine-readable output                                                                                                                      |

`dataflare init` is safe to re-run: it finds existing resources and keeps your keys.

Each deployment is named after its project (`glass-<project>`: Worker, D1 database, replay bucket), so several sites can each have their own Dataflare in one Cloudflare account. Pass `--name` to choose another.

**Sites not on Cloudflare** (Vercel, Netlify): load the snippet straight from the Worker's URL. A rewrite or proxy from your own domain makes every visitor look like the proxy's server, which breaks countries and cookieless visitor counts.

**Cookie banners:** load Dataflare with `data-persistence="cookieless"` (it stores nothing on the device, so it can run before consent) and call `glass.consent(true)` when the visitor accepts, `glass.consent(false)` when they reject. Cross-day visitor ids and session replay start after consent; add `data-replay-cookieless="true"` to record visitors who haven't chosen yet, still storing nothing on the device. Add Dataflare to your privacy page.

## 3. Add the snippet

The script tag works everywhere and needs no build step:

```html
<script defer src="https://glass.<you>.workers.dev/s.js" data-project="my-app"></script>
```

| Framework            | Where                                                                                                            |
| -------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Next.js              | `app/layout.tsx`: `<Script src="…/s.js" data-project="my-app" strategy="afterInteractive" />` from `next/script` |
| Astro                | `src/layouts/Layout.astro`, with `is:inline` on the script tag                                                   |
| SvelteKit            | `src/app.html`                                                                                                   |
| Nuxt                 | `app.head.script` in `nuxt.config.ts`                                                                            |
| Remix / React Router | `app/root.tsx`                                                                                                   |
| Vite apps            | `npm i dataflare-web`, then `glass.init({ host, project })` in `main.ts`                                         |
| Server-side          | `POST <host>/e`; see [sdk.md](sdk.md#server-side-events)                                                         |

## 4. Track what matters

Pageviews, clicks on links and buttons, JavaScript errors and web vitals are captured automatically. Add the events that describe your product:

```js
glass.capture('signup_completed', { plan: 'pro' });
glass.identify(user.id, { plan: user.plan, company_size: 12 }); // after login or signup
```

Without JavaScript:

```html
<button data-glass-event="pricing_cta_clicked" data-glass-prop-plan="pro">Start trial</button>
```

Name events `object_action` in snake_case and past tense. Put details in properties, and never send emails, names or other personal data.

Document every custom event once it is shipping — agents and teammates need to know **how and when** it fires, and **what each property means**:

```sh
npx dataflare-cli events signup_completed --describe "Fired when the user finishes signup after email verification."
```

Prefer the MCP `define_event` tool so you can set property meanings in the same call (`plan` → chosen plan id, etc.). `list_events` returns an `undocumented` list; clear it before building funnels or trends on those events.

## 5. Look at the data

- **Dashboard:** `https://glass.<you>.workers.dev/app`. Sign in with the key from `npx dataflare-cli key`. Share a view-only dashboard with `npx dataflare-cli key --read`.
- **Your agent:** with `.mcp.json` in place, ask "what were our top traffic sources this week?". See [agents.md](agents.md).
- **Terminal:** `npx dataflare-cli query web --from -7d`, `npx dataflare-cli events`, `npx dataflare-cli --help`.

## Upgrading

Init is the only deploy that has to run on your machine. After that, the Dataflare version is pinned in `.glass/config.json` and `.github/workflows/glass.yml` deploys whatever `.glass/` holds when it changes on your default branch.

One-time setup, right after init (agents should do this as part of init):

1. Add a repository secret `CLOUDFLARE_API_TOKEN` holding a Cloudflare API token with the same scopes as init: `gh secret set CLOUDFLARE_API_TOKEN`.
2. Commit `.glass/` and `.github/workflows/glass.yml` and push. (`credentials.json` stays git-ignored; the Worker's keys were set by init and survive every deploy.)

To upgrade:

```sh
npx dataflare-cli@latest upgrade   # stages the new Worker, migrations and version into .glass/; deploys nothing
git add .glass && git commit -m "Upgrade Dataflare" && git push   # or open a PR; CI migrates and deploys
```

In CI, the workflow runs `npx dataflare-cli@<pinned version> upgrade --ci`, which applies migrations and redeploys without prompts or local credentials.

- No workflow yet (installed before this existed, or with `--no-ci`)? `npx dataflare-cli ci` writes it. Options: `--branch`, `--secret`. A workflow you have edited by hand (without the generated header) is left alone.
- No GitHub Actions? Without the workflow, `dataflare upgrade` deploys directly, as before. `dataflare upgrade --local` forces a direct deploy even when the workflow exists; use it only in emergencies, since the repo then no longer matches what is live.
- `dataflare upgrade` refuses to replace a newer Worker with an older one (for example a stale `npx` cache). Use `@latest`. `--allow-downgrade` overrides this.

You'll also be told when an upgrade is available: the dashboard shows a banner after login, `dataflare status` (and other CLI commands) print a reminder, and MCP/`overview` include `upgradeAvailable` so agents can ask you to run the command above.

## Removing Dataflare

Delete the Worker, the D1 database and the R2 buckets from the Cloudflare dashboard (they're all named after `--name`, `glass` by default), then delete `.glass/` and the Dataflare section of `AGENTS.md`.


---

<!-- https://dataflare.dev/docs/sdk/ -->

# Browser SDK

Two ways to load it, with the same API:

- **Script tag** (served by your Worker, no build step): `<script defer src="https://<host>/s.js" data-project="my-app"></script>`, then use `window.glass`.
- **npm**: `npm i dataflare-web`, then `import { glass } from 'dataflare-web'` and call `glass.init({ host, project })`.

The SDK is about 4 KB gzipped and has no dependencies. Session replay loads a separate recorder from your Worker only for recorded sessions.

## `glass.init(options)`

| Option            | Default          |                                                                                                                                                                                                           |
| ----------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `host`            | required         | Your Worker URL, e.g. `https://glass.acme.workers.dev` or `/_glass`                                                                                                                                       |
| `project`         | required         | Project id                                                                                                                                                                                                |
| `persistence`     | `'localStorage'` | `'cookieless'` stores nothing on the device (see [privacy.md](privacy.md))                                                                                                                                |
| `autocapture`     | `true`           | `false`, or `{ pageviews, clicks, errors, webVitals }` to pick                                                                                                                                            |
| `replay`          | off              | `{ sampleRate: 0.1, onErrorSampleRate: 1, maskAllText: false, maskTextSelector: '[data-glass-mask]', blockSelector: '[data-glass-block]', console: ['warn', 'error'], network: true, cookieless: false }` |
| `flags`           | `true`           | Load feature flags on init                                                                                                                                                                                |
| `beforeSend`      |                  | `(event) => event                                                                                                                                                                                         | null` to edit or drop events |
| `flushIntervalMs` | `5000`           | How often queued events are sent                                                                                                                                                                          |
| `debug`           | `false`          | Log events to the console                                                                                                                                                                                 |

Script tag equivalents: `data-project`, `data-host`, `data-persistence="cookieless"`, `data-autocapture="false"`, `data-replay-sample-rate="0.1"`, `data-replay-on-error-sample-rate="1"`, `data-replay-console="false"` (or `"all"`), `data-replay-network="false"`, `data-replay-mask-all-text="true"`, `data-replay-cookieless="true"`, `data-debug="true"`.

## Events

```js
glass.capture('invite_sent', { role: 'editor', count: 3 });
```

Events are batched and sent every few seconds, when 20 are waiting, and when the page is hidden (with `sendBeacon`, so nothing is lost on navigation). Names starting with `$` are reserved.

Automatic events:

| Event         | When                                          | Properties                                   |
| ------------- | --------------------------------------------- | -------------------------------------------- |
| `$pageview`   | Load and client-side navigation (History API) |                                              |
| `$pageleave`  | Leaving a page                                | `duration_ms`                                |
| `$click`      | Clicks on links, buttons, `[role=button]`     | `tag`, `text`, `href`, `outbound`            |
| `$exception`  | Uncaught errors, unhandled rejections         | `type`, `message`, `stack`                   |
| `$web_vitals` | Page hidden                                   | `metric` (LCP, CLS, INP, FCP, TTFB), `value` |
| `$exposure`   | First `glass.flag(key)` read on a page        | `flag`, `variant`                            |

Add `data-glass-no-capture` to an element to exclude it from click capture. Add `data-glass-event="name"` (and `data-glass-prop-*` attributes) to send a custom event on click instead.

## People

```js
glass.identify(user.id, { plan: 'pro' }); // after login: links this device to the user
glass.setPersonProperties({ plan: 'team' }); // update properties
glass.setPersonProperties({ first_source: 'ad' }, true); // only if not already set
glass.reset(); // on logout: new anonymous id
```

Events sent before `identify()` on the same device count towards that user in every report, so a funnel like "visited pricing → signed up" connects across signup. Person properties are copied onto each event when it's received, so `person.plan` filters reflect the plan at the time of the event.

## Feature flags

```js
glass.onFlags(() => {
  if (glass.flag('new-checkout') === 'test') showNewCheckout();
});
glass.isEnabled('beta-editor'); // true / false
glass.flags(); // all values, without recording exposures
```

`glass.flag(key)` returns `"on"` for a simple flag, the variant key for a multivariate flag, or `null`. The first read on a page records an `$exposure` event, which experiments count, so read the flag where the variant is actually shown. Values are cached on the device, so returning visitors get them instantly. Flags are evaluated by your Worker, so targeting rules are never sent to browsers.

## Session replay

Turn it on with a sample rate. Inputs are always masked; `maskAllText` masks all text; elements matching `blockSelector` aren't recorded at all.

```js
glass.init({ host, project, replay: { sampleRate: 0.1, onErrorSampleRate: 1 } });
glass.startReplay(); // record this session regardless of sampling
glass.stopReplay();
```

`onErrorSampleRate` keeps the last minute in memory and uploads it only if an error happens, so you get the run-up to every error without recording everything.

Besides the page, recordings carry what developers need to debug:

- **Console messages**: `warn` and `error` by default. `console: true` records every level, `console: false` none. Your code's log messages can contain personal data, so only turn on routine levels if you need them.
- **Network requests**: method, URL (without the query string), status and timing of `fetch` and `XMLHttpRequest` calls. Never headers or bodies. `network: false` turns it off.
- **Client-side navigation**, so single-page apps show the right URL in the player.

Mark elements whose text should be masked with `data-glass-mask`, and elements to leave out entirely with `data-glass-block`.

Each session gets a timeline: labelled clicks (with a CSS selector you can search your code for), rage clicks, dead clicks (a click on a control after which nothing changed), clicks followed by an error, errors with stack frames, failed and slow requests, and console errors. The dashboard player shows it beside the recording, and agents read it with `summarise_replay`.

## Consent

```js
glass.init({ host, project, persistence: 'cookieless' }); // before consent
glass.consent(true); // after the user agrees: switch to a persistent id, allow replay
glass.consent(false); // delete stored ids, stop replay, go back to cookieless
```

Replay normally waits for consent. With `replay: { sampleRate: 1, cookieless: true }` (`data-replay-cookieless="true"`) it also records cookieless visitors, storing nothing on the device; the Worker files each page load under the visit's cookieless ids. `consent(false)` stops it for the rest of the page, so call it on load for visitors who rejected earlier.

## Server-side events

Send events from your backend with a plain POST:

```js
await fetch('https://glass.acme.workers.dev/e', {
  method: 'POST',
  body: JSON.stringify({
    project: 'my-app',
    events: [
      {
        id: crypto.randomUUID(),
        event: 'invoice_paid',
        ts: Date.now(),
        distinct_id: user.id,
        user_id: user.id,
        props: { amount: 49 },
      },
    ],
  }),
});
```


---

<!-- https://dataflare.dev/docs/agents/ -->

# Agents: MCP, CLI and HTTP

Everything in Dataflare is available to agents in three equivalent ways. They call the same service, so results match exactly.

## MCP

`dataflare init` adds this to `.mcp.json`, which Claude Code, Cursor and other MCP clients read:

```json
{ "mcpServers": { "glass": { "command": "npx", "args": ["-y", "dataflare-cli", "mcp"] } } }
```

`dataflare mcp` is a small stdio bridge that forwards to your Worker's `/mcp` endpoint with the key from `.glass/credentials.json`, so no secret goes into the config file. Clients that support remote servers can connect directly instead:

```sh
claude mcp add --transport http glass https://glass.<you>.workers.dev/mcp --header "Authorization: Bearer $GLASS_API_KEY"
```

Use the read-only key (`npx dataflare-cli key --read`) for agents that shouldn't change flags or experiments; write tools then return a clear error.

### Tools

| Tool                                                                               |                                                                                                                                                                                                                                                  |
| ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `overview`                                                                         | Start here: headline stats vs the previous period, top pages, referrers, errors, frequent events and the events that moved most                                                                                                                  |
| `query_web`                                                                        | The full web analytics view                                                                                                                                                                                                                      |
| `query_trends`                                                                     | Counts, unique people, sums or averages over time, with breakdowns                                                                                                                                                                               |
| `query_funnel`                                                                     | Conversion through ordered steps within a window, with breakdowns                                                                                                                                                                                |
| `query_retention`                                                                  | Cohort retention by day, week or month                                                                                                                                                                                                           |
| `sql`                                                                              | One read-only SELECT against the event store                                                                                                                                                                                                     |
| `list_events`                                                                      | Event names, counts and documentation status (`undocumented` lists custom events still missing a description)                                                                                                                                    |
| `describe_event`                                                                   | Properties sent with an event: types, coverage, examples                                                                                                                                                                                         |
| `define_event`                                                                     | Document how/when an event fires and what each property means                                                                                                                                                                                    |
| `mark_event`                                                                       | Mark an event as a `conversion` (signup, purchase, pledge) or `important`; Reports and `overview` then show its count and rate, and conversion rates per source                                                                                  |
| `query_sources`                                                                    | Visitors by channel, referrer, UTM source/medium/campaign, landing page, country or device (first touch in the range), with conversions and bounce                                                                                               |
| `site_health`                                                                      | Errors over time and the share of sessions they hit, pages where errors happen, likely 404 pages, the slowest pages by LCP, replay frustration                                                                                                   |
| `usage_and_cost`                                                                   | What Dataflare adds to the Cloudflare bill per month, with recommendations (replay sampling, lake store, retention, noisy events) and a prompt for each                                                                                          |
| `instrument`                                                                       | The exact code to track an event in a framework                                                                                                                                                                                                  |
| `search_persons`, `get_person`                                                     | Identified people plus anonymous device counts; one person's properties, devices, events and replays                                                                                                                                             |
| `property_values`, `list_person_properties`                                        | Top values of a property (with event and people counts) and the person property names, so filters use values that exist                                                                                                                          |
| `list_cohorts`, `save_cohort`, `count_cohort`, `delete_cohort`                     | Saved person filters (and optional did-event) with live membership counts. Filter any query to a cohort with `{"property":"cohort","value":"<key>"}`                                                                                             |
| `list_replays`                                                                     | Find recorded sessions: by person, signals (`errors`, `rage_clicks`, `dead_clicks`, `console_errors`, `failed_requests`), visited `url`, `did_event` / `not_event`, event or cohort `filters`, text `search`; sort by `most_frustrated` and more |
| `summarise_replay`                                                                 | A session as a transcript: pages, clicks with labels and CSS selectors, rage and dead clicks, errors with stack frames, failed requests, console errors, notes. Highlights link to each moment                                                   |
| `replay_snapshot`                                                                  | What was on screen at a moment (`at: "01:24"`), as text: headings, links, buttons (and whether disabled), masked form fields, dialogs                                                                                                            |
| `analyse_replays`                                                                  | Patterns across sessions: most rage- and dead-clicked elements, common errors, failing requests, where frustrated sessions end; each with example links                                                                                          |
| `annotate_replay`, `delete_replay_note`                                            | Pin a finding to a moment of a session; it shows on the player timeline and in the transcript                                                                                                                                                    |
| `list_flags`, `save_flag`, `delete_flag`                                           | Feature flags                                                                                                                                                                                                                                    |
| `list_experiments`, `create_experiment`, `experiment_results`, `update_experiment` | A/B tests: create, start, read, stop, ship                                                                                                                                                                                                       |
| `list_projects`, `create_project`                                                  | Projects                                                                                                                                                                                                                                         |

Every result includes a `summary` sentence, the `sql` that ran, and a `dashboard` link to the same view. Tool input schemas are generated from the same zod schemas the API validates with, so the descriptions agents see are always current.

### Recipes

- **"First look at this project."** `overview` → `list_events`. If `upgradeAvailable` is set (also on `/health` and in MCP initialize instructions), tell the user. With their go-ahead, run `npx dataflare-cli@latest upgrade`, which stages `.glass/` when `.github/workflows/glass.yml` exists, then commit and push so CI deploys it. Don't deploy Dataflare from a laptop. For every name in `undocumented`, find the capture site in code (or ask), then `define_event` with a how/when description and property meanings before building queries on those events.
- **"Why did X drop?"** `overview` → `query_funnel` with `breakdown: "browser"` (or `utm_source`, `device`) → `list_replays` with `did_event` (the step before) and `not_event` (the step after), `sort: "most_frustrated"` → `summarise_replay` → `replay_snapshot` at the moment it goes wrong → grep the selector in the code → `annotate_replay` with what you found.
- **"What's annoying users?"** `analyse_replays` for the period → open the top findings' example links with `summarise_replay`.
- **"Add tracking for Y."** `instrument` with the event name → add the code → deploy → `describe_event` to confirm it arrives → `define_event` with how/when it fires and what each property means → if it means success (a signup, purchase, pledge), `mark_event` with role `conversion`.
- **"Which campaigns work?"** `query_sources` with `dimension: "utm_campaign"` and `"channel"`. Filter anything to whole visits from a source with `{"property":"session.utm_source","value":"google"}`.
- **"Test Z."** `create_experiment` with a goal → branch on `glass.flag(key)` in code → `update_experiment` with `start` → later `experiment_results` → `update_experiment` with `ship`.

## CLI

All commands accept `--json`. In a project set up with `dataflare init` they need no configuration; elsewhere set `GLASS_HOST`, `GLASS_API_KEY` and optionally `GLASS_PROJECT`.

```sh
npx dataflare-cli query web --from -7d
npx dataflare-cli query trends --event signup_completed --math unique_persons --interval week --breakdown utm_source
npx dataflare-cli query funnel --steps '$pageview,signup_started,signup_completed' --window 1d --where 'device=mobile'
npx dataflare-cli query retention --cohort signup_completed --return '$pageview' --period week
npx dataflare-cli query funnel --spec '{"steps":[{"event":"a"},{"event":"b"}],"from":"-90d"}' --sql
npx dataflare-cli sql "SELECT path, COUNT(*) AS n FROM events WHERE event = '\$pageview' GROUP BY path ORDER BY n DESC LIMIT 10"
npx dataflare-cli events signup_completed
npx dataflare-cli replays --errors --url /checkout --not purchase
npx dataflare-cli replays summary <session-id>
npx dataflare-cli replays snapshot <session-id> --at 01:24
npx dataflare-cli replays analyse --from -7d
npx dataflare-cli flags set new-checkout --variants control:50,test:50
npx dataflare-cli experiments create checkout-v2 --goal purchase_completed
```

## HTTP

`GET /api` on your Worker lists every endpoint, `GET /api/schema` returns JSON Schemas for every query spec, and `GET /llms.txt` has these instructions with your real URLs. Authenticate with `Authorization: Bearer <key>`.

```sh
curl -s https://glass.<you>.workers.dev/api/query \
  -H "Authorization: Bearer $GLASS_API_KEY" \
  -d '{"kind":"funnel","steps":[{"event":"$pageview"},{"event":"signup_completed"}],"from":"-30d","breakdown":"utm_source"}'
```

Errors are JSON with an `error` message and, where useful, a `hint` saying how to fix it.


---

<!-- https://dataflare.dev/docs/queries/ -->

# Queries

Every query is a JSON spec. The same specs work in the dashboard (they're in its URLs), the MCP tools, the CLI (`--spec`) and `POST /api/query`.

## Time ranges

`from` (inclusive) and `to` (exclusive) accept ISO dates (`2026-10-01`), epoch milliseconds, `now`, `today`, `yesterday`, or relative offsets (`-30d`, `-12h`, `-2w`, `-3mo`, `-1y`). The default is the last 30 days. Buckets are UTC; shift them with `utcOffsetMinutes` (e.g. `60` for UTC+1).

## Properties

| Written as                                                            | Means                                        |
| --------------------------------------------------------------------- | -------------------------------------------- |
| `path`, `url`, `host`, `referrer_host`, `title`                       | Page columns                                 |
| `utm_source`, `utm_medium`, `utm_campaign`, `utm_term`, `utm_content` | Campaign tags from the page URL              |
| `country`, `region`, `city`, `device`, `browser`, `os`                | From the request                             |
| `event`, `session_id`, `user_id`, `distinct_id`                       | Identity columns                             |
| `plan` or `props.plan`                                                | The event property `plan`                    |
| `props.address.city`                                                  | A nested event property                      |
| `person.plan`                                                         | The person's `plan` at the time of the event |

## Filters

```json
{ "property": "person.plan", "op": "eq", "value": "pro" }
```

Ops: `eq` (default), `neq`, `contains`, `not_contains`, `in`, `not_in`, `gt`, `gte`, `lt`, `lte`, `is_set`, `is_not_set`. Negative ops (`neq`, `not_contains`, `not_in`) include events where the property is missing. `contains` ignores case. Numeric comparisons apply to event and person properties.

## Web

```json
{ "kind": "web", "from": "-7d", "filters": [{ "property": "utm_campaign", "value": "launch" }] }
```

Visitors are distinct devices; a session ends after 30 minutes of inactivity. Referrers and UTM tags are attributed to the session's first pageview, so a campaign visit counts once, not once per page. Bounce rate is the share of sessions with one pageview.

## Trends

```json
{
  "kind": "trends",
  "series": [
    { "event": "signup_completed", "math": "unique_persons" },
    { "event": "order_paid", "math": "sum", "property": "amount" }
  ],
  "interval": "week",
  "breakdown": "utm_source",
  "from": "-90d"
}
```

`math`: `total` (default), `unique_persons`, `unique_sessions`, `sum`, `avg`. With a breakdown, the top 10 values are kept (`breakdownLimit`); for totals and sums the rest are combined as "Other".

## Funnels

```json
{
  "kind": "funnel",
  "steps": [
    { "event": "$pageview", "filters": [{ "property": "path", "value": "/pricing" }] },
    { "event": "signup_started" },
    { "event": "signup_completed" }
  ],
  "window": "7d",
  "breakdown": "device"
}
```

A person enters at their first step-1 event in the range. Each later step must happen strictly after the previous one and within `window` of step 1. Breakdowns use the property of the step-1 event. The result has counts, conversion from the start and from the previous step, and the mean time between steps.

## Retention

```json
{
  "kind": "retention",
  "cohortEvent": { "event": "signup_completed" },
  "returnEvent": { "event": "$pageview" },
  "period": "week",
  "periods": 8,
  "from": "-90d"
}
```

A person's cohort is the period of their first `cohortEvent` in the range. Period 0 is the cohort itself; period _k_ counts people who did `returnEvent` _k_ periods later. The average weights cohorts by size and only includes cohorts old enough to have that period.

## SQL

```json
{
  "query": "SELECT browser, COUNT(DISTINCT distinct_id) AS visitors FROM events WHERE project = 'my-app' AND ts_ms >= 1767225600000 GROUP BY browser",
  "limit": 100
}
```

One read-only `SELECT` (or `WITH … SELECT`). In D1 mode you can read `events`, `identities`, `persons`, `flags`, `experiments`, `replays` and `event_definitions`; in lake mode, `<namespace>.events` and `<namespace>.identities`. `ts_ms` is epoch milliseconds; `props`, `person_props` and `flags` are JSON (`json_extract(props, '$.plan')` in D1, `json_get_str(props, 'plan')` in the lake).

Each query result's `sql` field shows exactly what ran, which is the easiest way to learn the schema.


---

<!-- https://dataflare.dev/docs/experiments/ -->

# Flags and experiments

## Flags

A flag has a rollout percentage, optional variants, and optional targeting rules:

```json
{
  "key": "new-checkout",
  "enabled": true,
  "rollout": 50,
  "variants": [{ "key": "control", "weight": 50 }, { "key": "test", "weight": 50 }],
  "rules": [
    {
      "conditions": [{ "property": "person.plan", "value": "enterprise" }],
      "rollout": 100,
      "variant": "test"
    }
  ]
}
```

- **Evaluation** happens in your Worker (`POST /f`), so rules that mention person properties never reach browsers. The SDK caches values on the device.
- **Assignment** is a hash of the flag key, an optional `salt` and the device's `distinct_id`, so it's stable without storing anything, and identical wherever it's computed. Dataflare hashes the device id rather than the user id because the device id doesn't change at login, so signing in never flips someone's variant.
- **Rules** are checked in order; the first rule whose conditions all match sets the rollout (and optionally forces a variant). Conditions use the same property syntax as query filters: `person.*`, `country`, `device`, `browser`, `os`.
- Change the `salt` to reshuffle everyone.

```sh
npx dataflare-cli flags set beta-editor --rollout 10
npx dataflare-cli flags set beta-editor --rules '[{"conditions":[{"property":"person.plan","value":"pro"}],"rollout":100}]'
```

## Experiments

An experiment is a multivariate flag plus a goal event:

```sh
npx dataflare-cli experiments create checkout-v2 --goal purchase_completed --hypothesis "A one-page checkout converts better"
```

This creates the experiment as a draft, plus a `checkout-v2` flag with `control` and `test` at 50/50, disabled until you start. In code, read the flag where the variant is shown:

```js
if (glass.flag('checkout-v2') === 'test') renderOnePageCheckout();
```

Then `npx dataflare-cli experiments start checkout-v2`, wait, and read `npx dataflare-cli experiments results checkout-v2`. Ship the winner with `npx dataflare-cli experiments ship checkout-v2 test`; everyone gets that variant from then on.

## How results are computed

- A person enters at their **first `$exposure`** to the flag. People exposed to more than one variant are excluded and counted separately; if that number is large, the flag is being read inconsistently.
- A person **converts** if they do the goal event at or after their first exposure.
- **Conversion rate** per variant comes with a 95% Wilson interval.
- **Probability to beat control** and the **lift interval** come from a Bayesian beta-binomial model with a uniform prior, estimated with 20,000 seeded Monte Carlo draws (the same data always gives the same numbers).
- With a `value` property on the goal (e.g. revenue), Dataflare also compares **means per person** with Welch's t-test.
- **Sample ratio mismatch**: a chi-square test of exposures against the variant weights. If p < 0.001, Dataflare flags the experiment as broken and won't declare a winner, because assignment or exposure tracking is faulty.
- A **winner** is declared at ≥ 95% probability to beat control. Until then, Dataflare estimates the days needed to detect a 10% relative lift at the current traffic.

The statistics live in `packages/core/src/stats/tests.ts` with tests against known values.


---

<!-- https://dataflare.dev/docs/privacy/ -->

# Privacy

Dataflare runs entirely in your Cloudflare account. There is no Dataflare service: events, people, recordings and keys live in your D1 database, your R2 buckets and your Worker's secrets. Nothing is sent to the Dataflare project or to any third party, and the dashboard and SDK load no third-party resources. (The Dataflare website, which is separate from your deployment, uses the Meta Pixel to measure its own ads.)

## What is stored

| Data         | Where                     | Notes                                                                                                                                                            |
| ------------ | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Events       | D1 `events` (or the lake) | URL without query string (except UTM tags), referrer without query string, title, country/region/city from Cloudflare, browser/OS/device family, your properties |
| IP addresses | Nowhere                   | Used only in memory, to derive cookieless ids                                                                                                                    |
| People       | D1 `persons`              | The properties you send with `identify()` / `setPersonProperties()`                                                                                              |
| Device links | D1 `identities`           | Anonymous device id → your user id                                                                                                                               |
| Replays      | R2, gzipped               | rrweb recordings; deleted after 30 days by default (`GLASS_REPLAY_RETENTION_DAYS`)                                                                               |

Set `GLASS_RETENTION_DAYS` (a Worker variable) to delete events older than that many days.

## Cookieless mode

With `persistence: 'cookieless'` (or `data-persistence="cookieless"`), the SDK stores nothing on the device. The Worker derives a visitor id from a daily-rotating secret salt, your project, the IP address and the user agent, hashes it with SHA-256, and stores only the hash. The salt changes every day, so ids can't be linked across days or reversed to an IP. Sessions are 30-minute windows of that id.

This is enough for web analytics. It is not enough for funnels across days, retention or experiments, which need a stable id; call `glass.consent(true)` once the user agrees to switch to a persistent id. Whether you need consent in your jurisdiction is a question for your own legal advice.

## Session replay

- Every input is masked (`maskAllInputs`). Set `maskAllText` to mask all text as well, or mark elements with `data-glass-mask` to mask just their text.
- Elements matching `[data-glass-block]` (configurable) aren't recorded.
- Recording is sampled; nothing is recorded unless you set a sample rate.
- In cookieless mode replay is off unless you also set `replay.cookieless` (`data-replay-cookieless="true"`). Then nothing is stored on the device either: each page load's recording is filed under the same daily-rotating visitor and session ids as its events, and the sampling decision lasts one page load. `glass.consent(false)` stops recording for the rest of the page. Recording a visitor who hasn't consented may need a legal basis such as legitimate interest where you are; describe it on your privacy page.
- Network requests are recorded as method, URL without its query string, status and timing. Headers and bodies never are. Turn it off with `network: false`.
- Console `warn` and `error` messages are recorded by default; they contain whatever your code logs. Turn it off with `console: false`.
- Notes left on a session, and its cached summary, are deleted with the session (retention, or erasing a person).

## Deleting a person

```sh
npx dataflare-cli person <user-id> --delete
```

This deletes the person, their device links, their replays and (in D1 mode) every event from their user id and devices. In lake mode, Iceberg tables are append-only to Dataflare, so events must be removed with a separate delete job against the table; this is on the [roadmap](../ROADMAP.md).

## EU data

`dataflare init --eu` creates the D1 database and R2 buckets in Cloudflare's EU jurisdiction. The Worker itself runs on Cloudflare's global network.


---

<!-- https://dataflare.dev/docs/architecture/ -->

# Architecture

Dataflare is one Worker, one D1 database and one R2 bucket in the user's Cloudflare account, plus a CLI that deploys them. The optional lake store adds Cloudflare Pipelines and Iceberg tables.

```
                    ┌──────────────────────── your Cloudflare account ─────────────────────────┐
 browser SDK ──────▶│  POST /e  ingest pipeline ──▶ EventStore ──▶ D1 events                   │
 (s.js, replay.js)  │  POST /r  replay chunks ─────────────────▶ R2  replays/<project>/<session>│
                    │  POST /f  flag evaluation ◀── D1 flags, persons, identities              │
                    │                                                                          │
 coding agents ────▶│  POST /mcp  MCP server ─┐                                                │
 glass CLI ────────▶│  /api/*     JSON API ───┼──▶ GlassService ──▶ insight compilers ──▶ SQL   │
 you ──────────────▶│  /app       dashboard ──┘                                                │
                    │                                                                          │
                    │  lake mode: EventStore ──▶ Pipelines stream ──▶ Iceberg on R2 ──▶ Basin SQL│
                    └──────────────────────────────────────────────────────────────────────────┘
```

## Request paths

- **Ingest** (`packages/worker/src/ingest.ts`): parse a batch, check the project and origin, drop bots and malformed events, apply `identify()` links and person updates, resolve each event's person and snapshot their properties, enrich (clock skew, URL and UTM parsing, referrer, geo, user agent), and write. The body is `text/plain` JSON so browsers skip CORS preflights; the SDK uses `sendBeacon` when pages are hidden.
- **Queries** (`packages/worker/src/service.ts` → `packages/core/src/insights/`): a spec is validated with zod, compiled to SQL for the store's dialect, executed, and shaped into a result with a `summary`, the `sql` that ran and a dashboard link. The HTTP API, MCP tools and dashboard all go through `GlassService`.
- **Replay**: each upload is one gzipped JSON array of rrweb events, stored as its own R2 object keyed by start time, page load and sequence. A D1 row indexes the session, so listing replays never touches R2. Two constraints shape this:
  - _A page that is going away can send about 64 KB_ (`sendBeacon` and `keepalive` share that limit), and most pages' DOM snapshots are bigger. So `replay.js` uploads each page's snapshot the moment it's taken and never lets more than a beacon's worth of events wait, which leaves only a small remainder for the beacon when the page is hidden.
  - _rrweb numbers DOM nodes per page load_, so node 42 means something different on each page. Anything that resolves node ids, such as the transcript in `packages/core/src/replay-summary.ts`, reads each page load separately (`loadReplayPages()` groups chunks by their page id).
  - _Most of a single-page app arrives as mutations_, not in the first snapshot, and rrweb sends each added node separately. `packages/core/src/replay-dom.ts` rebuilds the DOM from snapshots plus every mutation (adds, removes, text and attribute changes), which is what names clicked elements and renders `replay_snapshot`.
  - _Reading every chunk is expensive_, so an hourly cron summarises sessions about 30 minutes after they end: the timeline is cached at `summaries/<project>/<session>.json`, and its frustration counts and a searchable digest (pages, clicked elements, errors) go on the D1 row. Search and `analyse_replays` read those; a new chunk makes the cache stale.

## Identity

Every event carries `distinct_id` (the device, stable across login), `user_id` (once identified) and `person_id` (resolved at ingest). `identify()` writes a link from device to user in `identities`; the first link wins, so a shared device never moves old events to a new user.

Queries resolve people at query time:

```sql
CASE WHEN user_id <> '' THEN user_id ELSE COALESCE(identities.person_id, person_id) END AS pid
```

So events from before signup count towards the user in funnels and retention, in both stores, without rewriting old rows.

## Portable SQL

The compilers (`packages/core/src/insights/*.ts`) target two engines: SQLite (D1) and Basin SQL (DataFusion over Iceberg). To keep one implementation:

- Time is integer milliseconds (`ts_ms`), so hour, day and week buckets are integer arithmetic.
- Funnels and retention use chained CTEs and joins, not window functions (budget-gated in Basin SQL).
- Everything engine-specific (JSON access, month arithmetic, case-insensitive search) is one of the `Dialect` methods in `packages/core/src/sql/dialect.ts`.
- Values are inlined through escaping helpers and property keys are validated, because Basin SQL's HTTP API has no parameter binding.

The core tests execute every compiler on real SQLite against the production migration, which covers the lake's logic too.

## Packages

| Package                | Published                    |                                                            |
| ---------------------- | ---------------------------- | ---------------------------------------------------------- |
| `@dataflare/core`      | no (bundled)                 | Event model, SQL, stats, flags, replay summaries, snippets |
| `dataflare-web`        | npm                          | Browser SDK                                                |
| `@dataflare/worker`    | no (bundled into the CLI)    | The Worker                                                 |
| `@dataflare/dashboard` | no (bundled into the Worker) | The dashboard                                              |
| `dataflare`            | npm                          | CLI, carrying the built Worker and migrations              |

Versioning the CLI and the Worker together means `npx dataflare-cli@x upgrade` always deploys exactly the Worker that CLI was tested with.

## Why not…

- **Analytics Engine** for the web view: it samples and keeps three months of data. D1 is exact and real-time at the volumes most Dataflare users have, and the lake covers large volumes. An Analytics Engine accelerator for real-time tiles at scale is on the roadmap.
- **A Durable Object per session** for replay: R2 objects per chunk plus a D1 index do the job with fewer moving parts.
- **A web framework** in the Worker: the routing Dataflare needs is ~60 lines (`http.ts`), and fewer dependencies are easier to audit.


---

<!-- https://dataflare.dev/docs/lake/ -->

# The lake store (beta)

By default Dataflare keeps events in D1, which is exact, real-time and free-plan friendly. D1 databases have a size limit, though, so for high volumes Dataflare can stream events into Apache Iceberg tables in R2 and query them with Basin SQL (Cloudflare's DataFusion engine, formerly R2 SQL):

```sh
npx dataflare-cli init --lake
```

Status: **beta**. Pipelines and Basin SQL are themselves in beta, and the Basin SQL response format is not documented yet, so Dataflare parses it tolerantly (`normaliseRows()` in `packages/worker/src/stores/lake.ts`). Please report anything odd.

## What it creates

- An R2 bucket `glass-data` with the data catalog enabled
- Two Pipelines streams, `glass_events` and `glass_identities`, with fixed schemas (`packages/core/src/lake.ts`)
- Two sinks writing Iceberg tables `glass.events` and `glass.identities` (Parquet, files rolled every 60 seconds)
- Two pipelines connecting them (`INSERT INTO <sink> SELECT * FROM <stream>`)
- Worker bindings `EVENTS_STREAM` and `IDENTITIES_STREAM`, and the API token as the `CF_API_TOKEN` secret so the Worker can query Basin SQL

D1 still holds projects, people, identity links, flags, experiments and the replay index.

## Differences from D1 mode

|                            | D1                                     | Lake                                                                   |
| -------------------------- | -------------------------------------- | ---------------------------------------------------------------------- |
| Freshness                  | Immediate                              | About 1–2 minutes (file roll interval)                                 |
| History                    | Until D1's size limit                  | Unbounded                                                              |
| Cost                       | Included in Workers plans up to limits | R2 storage, Pipelines and Basin SQL scans                              |
| Person properties          | At event time, or current via SQL join | At event time                                                          |
| Deleting a person's events | Yes                                    | Not yet (roadmap)                                                      |
| Other engines              |                                        | Any Iceberg reader (DuckDB, Spark, Snowflake) can read the same tables |

## Using a narrower token

The Worker only needs to _read_ in lake mode. After `init`, you can replace its token with one that has just _Workers R2 SQL › Read_ and _Workers R2 Data Catalog › Read_:

```sh
npx wrangler secret put CF_API_TOKEN --config .glass/wrangler.jsonc
```

## Schema changes

Pipelines streams can't be altered after creation, so the stream schemas are versioned (`LAKE_VERSION` in `packages/core/src/lake.ts`). Custom properties never need a schema change: they live in the `props` JSON column. A new built-in column would need new streams (`glass_events_v2`) and queries over both tables; that migration path isn't built yet and is tracked in the roadmap.


---

<!-- https://dataflare.dev/docs/extending/ -->

# Extending Dataflare

Dataflare is built from small registries so the common extensions are one file each. Every example here has a test pattern next to it in the repository.

## Add a query type

1. Create `packages/core/src/insights/paths.ts`:

   ```ts
   import { z } from 'zod/v4';
   import { eventsCte, num, text } from '../sql/source.ts';
   import { iso, resolveRange } from '../time.ts';
   import type { QueryContext, ResultBase } from './types.ts';

   export const pathsSpec = z.object({
     project: z.string().optional(),
     from: z.union([z.string(), z.number()]).optional(),
     to: z.union([z.string(), z.number()]).optional(),
     start: z.string().describe('Page path the paths start from.'),
   });

   export async function runPaths(spec: z.infer<typeof pathsSpec>, ctx: QueryContext): Promise<ResultBase & { next: { path: string; people: number }[] }> {
     const range = resolveRange(spec.from, spec.to, ctx.now);
     const d = ctx.dialect;
     const sql = `WITH ${eventsCte(d, { project: ctx.project, ...range, events: ['$pageview'] })} …`;
     const rows = await ctx.execute(sql);
     return { kind: 'paths', sql: [sql], range: { from: iso(range.from), to: iso(range.to) }, summary: '…', next: rows.map((r) => ({ path: text(r.path), people: num(r.n) })) };
   }
   ```

2. Register it in `packages/core/src/insights/index.ts`:

   ```ts
   paths: define({ description: 'Where people go next after a page.', schema: pathsSpec, run: runPaths }),
   ```

You now have `POST /api/query {"kind":"paths"}`, an MCP tool `query_paths` and `dataflare query paths --spec '…'`. Write tests in `packages/core/test/` that insert events with `insert()` and execute your SQL on SQLite (see `insights.test.ts`).

Rules for portable SQL: start from `eventsCte()` (it scopes to the project and range, and resolves people as `pid`); use `ts_ms` integer arithmetic or `bucketExpr()` for time; filters via `filtersSql()`; anything engine-specific goes in `Dialect`.

## Add an MCP tool

Add a method to `GlassService` (`packages/worker/src/service.ts`) that validates its input with zod and returns a plain object with a `summary`. Then add an entry to `TOOLS` in `packages/worker/src/mcp/tools.ts`:

```ts
{
  name: 'list_cohorts',
  title: 'List cohorts',
  description: 'Saved groups of people, with their definitions and sizes.',
  input: inputs.listCohorts,
  scope: 'read',
  run: (s, a) => s.listCohorts(a),
},
```

Add an HTTP route in `packages/worker/src/api.ts` and a CLI command if people will want it from a terminal. Test through `testApp()` in `packages/worker/test/app.test.ts`, which runs the real Worker on SQLite.

## Add a storage engine

Implement two interfaces:

- `Dialect` (`packages/core/src/sql/dialect.ts`): literals, JSON access, month arithmetic and table names for your engine.
- `EventStore` (`packages/worker/src/stores/types.ts`): `write(events)`, `writeIdentity(link)`, `query(sql)`, `deletePersonEvents(project, ids)`.

Then select it in `createStore()` (`packages/worker/src/app.ts`) from an environment variable, and add a provisioning step to `dataflare init` if it needs resources. Run the core insight tests against your dialect if your engine can run locally (DuckDB and ClickHouse both can).

## Add a framework snippet

Add an entry to `FRAMEWORKS` and a detection rule in `detectFramework()` (`packages/core/src/instrument.ts`). `dataflare init` and the `instrument` tool pick it up automatically.

## Enrich events at ingest

`toRows()` in `packages/worker/src/ingest.ts` is the enrichment step. Add a column through a new migration (and `EVENT_COLUMNS`, and the lake schema), or add derived values to `props`. Keep ingest fast: it runs on every request.

## Change the dashboard

Pages are functions from a route to DOM nodes (`packages/dashboard/src/pages/`), built with `h()` from `dom.ts`. Add a page to `PAGES` in `main.ts`. Wrap anything that shows numbers in `panel(…, { sql })` so "Show SQL" works.