FIMViz.js

6 · Storage & records

Two independent ideas that pair well. Dataset.toRecord() serializes a Dataset as its source plus its op recipe — small, because the immutable pipeline is the value. Storage is a generic key-value store over IndexedDB that knows nothing about Datasets, files or FIMViz: you name the database, the tables and the keys, and it stores your value verbatim.

This page has no map. It writes to a real IndexedDB database in your browser called fimviz-notebook — the last section deletes it. Serve the repo root; IndexedDB does not work from file://.

1toRecord() — the recipe is the value

A Dataset is a lazy chain, so the honest way to persist one is to persist the chain: where the bytes came from, and which operations were applied. That is what toRecord() gives you by default, and it is why storing a five-op derivation costs about as much as storing the original. { storeMaterialized: true } embeds the decoded grid as well — call await ds.load() first, or it throws.

Dataset.fromRecord() rehydrates either form, replaying the op chain node by node. It is required: a structured clone drops prototypes, so what comes back out of IndexedDB is inert data until this rebuilds it. 

const ds      = await FimViz.parseFile(file);
const derived = ds.clip(bbox).reclassify([{ min: 2, max: Infinity, value: 1 }]);

const recipe = derived.toRecord();                          // source + ops
await derived.load();
const baked  = derived.toRecord({ storeMaterialized: true });  // …plus the decoded grid

const back = Dataset.fromRecord(recipe);   // the chain rebuilt, still lazy
await back.grid();                          // re-runs it from the stored bytes

// A callback-form reclassify does NOT survive: functions cannot be structured-cloned.
// Use range rules for a chain you intend to persist.

Output

…

2Values are stored verbatim

Keys are out-of-line — the store never reaches into your value to find its key — and that is precisely what lets the value be stored by structured clone rather than as JSON. An ArrayBuffer, a TypedArray, a Blob, a Map, a Set, a Date: all round-trip exactly.

The alternative silently destroys data, which is the whole argument: JSON.stringify(arrayBuffer) is "{}". Not an error, not a warning — an empty object where your raster used to be. The button below demonstrates it on real bytes. 

const db = new Storage({ name: "fimviz-notebook", version: 1, tables: ["files", "notes"] });

await db.put("files", "depth.tif", arrayBuffer);   // stored byte for byte
(await db.get("files", "depth.tif")).byteLength;   // unchanged

JSON.stringify(arrayBuffer);   // "{}"  ← the thing this design exists to avoid

Output

…

3Discovery — ask before you open

IndexedDB cannot downgrade: opening at a version lower than the one on disk throws a hard VersionError. And the on-disk version is not something a host can track for itself, because createTable/dropTable bump it as a side effect — so the number you passed to the constructor goes stale on its own. Those two facts are why discovery exists.

The safest open is to omit version entirely, which attaches to whatever is already there. And Storage.databases() throws on engines without indexedDB.databases() rather than reporting an empty list — "no databases" and "cannot tell" must not look alike when the next step might be destroying data. 

await Storage.databases();          // [{ name, version }, …] for this origin
await Storage.exists("fimviz-notebook");

await db.version();      // the CURRENT on-disk version — never creates, never upgrades
db.declaredVersion;      // what THIS instance was constructed with

new Storage({ name: "fimviz-notebook", tables: ["files"] });   // no version → no downgrade risk

Output

…

4Rows

db.table(name) binds the table so a call reads as a verb on the table itself. It is pure sugar — every method delegates to the db-level call with table bound, so the two forms are fully interchangeable and there is nothing to dispose.

Two distinctions the API keeps that a thinner wrapper usually loses. has() distinguishes a stored undefined from a missing key. And a broken read rejects rather than resolving undefined, so a failure cannot be mistaken for an absence. Every write really commits before its promise resolves — it awaits the transaction, not just the request. 

const files = db.table("files");

await files.put(key, value);   // → the key
await files.get(key);
await files.has(key);          // distinguishes a stored `undefined` from absent
await files.delete(key);
await files.list({ keys: true });   // cheap — no values deserialized (a filename picker)
await files.list();                // [{ key, value }] in key order
await files.clear();               // this table only

Output

…

5Two kinds of change, two different tools

Data migration needs no version bump at all: keys are out-of-line and values are verbatim, so a record's shape was never part of the schema. map() walks a table in key order inside one transaction; return a new value to update the row, Storage.DELETE to remove it, or undefined to leave it alone. That last default matters — a callback that forgets to return does nothing, rather than wiping the table.

Structural change — which stores exist — is the other tool, because IndexedDB only permits it inside an upgrade transaction. createTable/dropTable bump the version for you; versionMigrate is the hook for anything more (create a store, copy rows between stores) and runs inside that transaction. 

// DATA — no version bump.
await db.table("notes").map((key, value) => {
  if (value.stale) return Storage.DELETE;      // a Symbol — cannot collide with your data
  if (value.v === 1) return { ...value, v: 2 };   // rewrite it
  // return undefined → untouched
});   // → how many rows were updated or deleted

await db.clearAll();   // every row in every table, one transaction, schema kept

// STRUCTURAL — bumps the version.
await db.createTable("thumbnails");   // false if it already existed
await db.dropTable("thumbnails");

new Storage({ name, version: 3, versionMigrate: ({ db, transaction, oldVersion }) => { … } });

Output

…

6The round trip that motivates all of it

Parse a file, derive from it, store the recipe, close the tab, come back, rehydrate, and force — the grid is rebuilt from bytes that never left the browser. This is the whole offline-dataset path, and the record it writes is small enough that a handful of them is unremarkable.

One thing to know when the chain includes a multi-dimensional format: rehydrating skips the parser, so a selector-backed Dataset needs its decoders registered before fromRecord. 

const derived = ds.clip(bbox).reclassify(rules);

await db.table("files").put("scenario-a", derived.toRecord());
// … later, or after a reload …
const record  = await db.table("files").get("scenario-a");
const rebuilt = Dataset.fromRecord(record);

await rebuilt.grid();      // same pixels, chain replayed
await fim.addLayer(rebuilt);   // or straight onto a map

Output

…

7Scope, and the end of the line

"Not a database library" is a decision, not a gap. There is no query language, no index management, and no transaction API beyond the calls above. If you need more, the intended answer is to build an adapter around a real database rather than to grow this class — the surface is kept small enough that such an adapter could slot in later.

The engine also names no database and no table, anywhere: a host owns the schema, which is why fim.storage throws until you pass storage to mount(). And the standing policy is that a schema change bumps the version and accepts a reset — this is storage a host should treat as disposable. 

await db.close();     // drop the connection; data persists, the next call reopens
await db.destroy();   // delete the whole database

// Shared across every map on an app, and only if the host asked for it:
const fim = await mount("#map", { provider: "leaflet",
  storage: { name: "my-store", version: 1, tables: ["userFiles"] } });
fim.storage;   // throws unless that option was given

Output

…