Loading Data

Loading a fresh environment has a few sharp edges that can cause silent data loss — publishes that report success while storing nothing useful. This page covers how to load correctly and how to verify you actually did.

publish-permissive controls whether unknown columns are accepted

A freshly provisioned environment starts with publish-permissive: false. In that state /publish rejects any column that doesn't already have a schema field, returning HTTP 400 (No such property: <col>). Turning the setting on lets those columns through — but, as the section below explains, that does not make them queryable on its own; you still declare the fields you intend to query.

# Check it
curl -s "https://<env>.minusonedb.com/system" -H "m1-auth-token: $TOK" | jq '.["publish-permissive"]'

# Turn it on
curl -s -X POST "https://<env>.minusonedb.com/system" \
  -H "m1-auth-token: $TOK" \
  --data-urlencode "publish-permissive=true"

Cluster propagation can lag by a moment — re-check with the GET if it still reads false immediately after.

/publish returns HTTP 200 with the real status in the body

/publish always returns HTTP 200 at the transport layer. The actual status is inside the JSON body:

{ "path": "/publish", "status": 400, "message": "No such property: parent_name", "error": "Bad Request" }

A loader that checks only the HTTP code will classify these as success, fill its state with bogus "ok" records, and leave you with numFound: 0 and no signal that anything went wrong. Always check the body status (and the presence of error/message), not just the HTTP code.

Permissive accepts unknown fields — but drops their values

This is the most important trap. publish-permissive: true accepts a publish containing unknown fields, but it does not add them to the schema as queryable typed properties. The unknown field names are recorded in the multi-string _m1.unknownProperties field on each doc; the field values are silently dropped.

The publish looks successful — HTTP 200, climbing "Saved N total" counts — but a GET /query?q=*&rows=1 reveals docs containing only _m1key, _m1rand, _m1.source, and _m1.unknownProperties: metadata with no data.

When loading data with columns you intend to query, mirror the schema first. A fresh environment has only a handful of fields; a long-lived one may have a hundred or more, and auto-discovery does not bridge the gap.

# Pull a working env's schema, strip the universal keys, apply to the destination
curl -s "https://<source>.minusonedb.com/schema?store=<store>" -H "m1-auth-token: $SRC_TOK" | jq . > schema.json
jq 'to_entries | map(select(.key | test("^_m1key$|^_m1rand$") | not)
    | .value | {name, type, multi: (.multi // false)})' schema.json > properties.json
curl -s -X POST "https://<dest>.minusonedb.com/schema/add" \
  -H "m1-auth-token: $DST_TOK" \
  --data-urlencode "store=<store>" \
  --data-urlencode "[email protected]"

Declare the _m1.* system fields

MinusOneDB stamps a cluster of system fields on every published record. They must be present in the schema, or the first chunked publish fails with HTTP 400 (unknown field '_m1.partial'). A long-lived environment usually has them; a freshly initialized one does not — add them as a group:

Field	Type	Multi	Purpose
_m1.partial	boolean	—	set on records split across a chunked publish
_m1.source	string	—	archive URI of the publish that produced the record
_m1.receivedUTC	date	—	server-side receive timestamp
_m1.ip	string	—	source IP that submitted the write
_m1.errorProperties	string	yes	property names that failed validation
_m1.unknownProperties	string	yes	property names not in the schema (values dropped under permissive)

Add _m1.geo.* fields only when the geo feature is enabled and a geo database is loaded.

Pre-load checklist

Before launching a sustained load into a fresh environment:

1. Mirror the source schema first (pull /schema, strip _m1key/_m1rand, apply via /schema/add). Non-negotiable if your data has columns you want to query — without it, see the silent-drop trap above.
2. GET /system → confirm publish-permissive: true.
3. GET /store/list → confirm the target store exists.
4. GET /schema?store=<store> → confirm the _m1.* system fields and your user fields exist; diff against the source schema.
5. GET /query?q=*&rows=0&store=<store> → confirm the expected baseline (usually numFound: 0).
6. Send a single small /publish, then fetch one doc back (/query?q=*&rows=1&fl=*) and confirm it has your user-defined fields populated — not just _m1key and _m1.unknownProperties. This read-back is the only reliable way to catch a silent schema miss; body-status alone shows success even when every value was dropped.

Only after that read-back returns real data in real fields should you start the full load.