Data Modeling & Best Practices

MinusOneDB stores schema-defined, queryable records that can also carry arbitrary nested JSON. You get the most out of it by modeling for its query engine rather than against it. This page collects the patterns that matter most.

Denormalize at ingest — there is no cross-document join

MinusOneDB has no cross-document JOIN. Every "join" becomes a client-side fetch-and-aggregate: many round trips, large payloads, slow first paint, and a cache layer to maintain. The facet engine is the natural shape for analytics — one faceted query per chart.

Rule: every field a query needs lives directly on the record being queried. Storage is cheap; repeating a small string (vertical, region, owner, client name) on every fact record costs little and turns each chart into a single facet query.

When you design a schema, ask:

1. Will any single view query this record type and another? → Denormalize the small side onto the large side at ingest.
2. Will users filter or group by this dimension? → Put it on the fact record.
3. Is it slowly-changing reference data? → Denormalize, and re-ingest the affected records when it changes.

Genuinely separate record types are still right for independent entities (users, sessions, audit logs) and for arbitrary-shape JSON that varies per record. The rule is about analytical dimensions, not every relationship.

A faceted analytics query looks like this:

{
  query: '*',
  facet: {
    by_vertical: {
      type: 'terms', field: 'vertical', limit: 20,
      facet: { revenue: 'sum(revenue)' }
    }
  }
}

Faceting: always set missing: true

A terms facet silently excludes records that have no value for the faceted field. The bucket sum becomes a strict subset of the dataset, so per-dimension chart totals never reconcile with the headline total — and the gap is invisible. This bites hardest when blank rates are non-trivial and the blank rows skew toward a few high-value records.

Set missing: true on every terms facet that drives a chart, and read the rollup from the sibling facet.missing key (it is not in the buckets array):

by_advertiser: {
  type: 'terms', field: 'advertiser_name', limit: 20, missing: true,
  facet: { rev: 'sum(revenue)' }
}

function bucketsWithMissing(facet, label) {
  if (!facet) return [];
  const buckets = (facet.buckets || []).slice();
  if (facet.missing && (facet.missing.count || 0) > 0) {
    buckets.push({ val: label || 'Not Reported', count: facet.missing.count, rev: facet.missing.rev || 0 });
  }
  return buckets;
}

Use a clear sentinel label ("Not Reported" for raw dimensions, "Unmapped" for taxonomy fields).

Nested-facet caveat: when a terms facet is nested inside another, the inner missing count comes back as 0 even when blank rows exist in the parent bucket. Reconstruct it client-side as parent.count - sum(child counts).

Per-tenant views: scope from the fact store

For multi-tenant dashboards (per-client, per-partner, per-seat):

• Build each tenant's view by filtering the fact store on the tenant key (e.g. AND cid:N). Do not derive it from a shared side "catalog" record keyed by an id that is common across tenants — a single catalog record per shared id carries one tenant's labels to everyone who shares that id. Consult a catalog only for genuinely id-intrinsic, non-tenant data (e.g. pricing).
• A denormalized field derived from another field must be re-derived whenever its source changes, or it goes stale and silently mis-attributes. Run a consistency pass on ingest that re-derives the dependent field wherever it disagrees with its source.

Updating data: use /modify

• The data lake is the canonical store and the target of all CRUD operations.
• /modify is the right call for any change — it does insert, update, and delete in one call. On update, unspecified fields are preserved, so a patch can be minimal.
• /write / /publish are for archival and bulk load, not for editing live data. Reaching for them to change existing records is the wrong tool.
• Prefer update over delete. Delete-and-reload produces phantom/duplicate state; update in place instead. Reserve /delete for genuinely removing a record.

POST /modify with form params:

• e — JSON array of entities. Each entity needs _record_type, plus _m1key (integer) for updates — without it the entity is treated as an insert. Set only the fields you want to change.
• deleted — optional JSON array of _m1key values to remove.
• publish — "true" (default) for a synchronous index publish.

Returns {firstInserted, updates, deletes}.

# Change one field on one row, leave everything else untouched
POST /modify
  publish=true
  e=[{"_m1key": 1549520001, "_record_type": "fact_table", "owner": "Acme"}]

→ {"firstInserted": null, "updates": 1, "deletes": 0}

OR-chunk pattern for imports keyed by an external id (when you don't have _m1key yet): chunk the external ids (~1000 per query), OR-query the fact store (q=external_id:(id1 OR id2 OR ...)) to get each row's _m1key, build a {_m1key, _record_type, new_field} patch per row (skip rows already correct), and POST /modify in batches.

Bulk delete: commit once, at the end

When deleting many records, commit the index a single time after the last batch. Every committed write is a heavy index-segment flush; committing per batch turns one delete into dozens of flushes. Collect all target _m1keys in one query (fl=_m1key), then delete carrying every id with one final commit.

Browser clients using multipart FormData hit a hard cap of 9,999 parts per request (≥10,000 returns HTTP 413), so chunk under that — but still set commit=false on every chunk except the last.

(Per the doctrine above, still prefer /modify for anything that's really an update; this is for genuine bulk removal.)

Schema field types: choose carefully — they are effectively permanent

• string is a Solr StrField with a hard ~32 KB cap per value; a write over it returns HTTP 400 (field … which is too large). Use it for ids, names, enums, and short strings you'll facet on.
• text is a Solr TextField and holds large payloads — session recordings, HTML, long JSON, base64. Use it for anything that might exceed a few KB. (The large: true flag is not always honoured; type: text is the reliable way to get a big-text field.)
• Numeric, date, and boolean fields should use the matching typed field — range queries and cost math depend on it.

Type choices are nearly permanent. Re-adding an existing field name returns HTTP 500 (duplicate key), and there is no /schema/remove endpoint. If you created a field with the wrong type, don't try to fix it in place: add a new field with a new name and the correct type, route writes there, and have readers fall back to whichever field is populated (fl: new_field,old_field). The orphaned field is harmless.

Choose up front by asking "could this value ever exceed ~32 KB?" — yes → text; no and you'll facet on it → string; numeric/date/boolean → the matching typed field.