FIMViz.js

Decisions, Tradeoffs & Incomplete Items

This is the consolidated history doc for the composable-core rewrite — it replaces four docs that had grown overlapping (COMPOSABLE_API.md, COMPOSABLE_PLAN.md, FLOODEXTENT_DECOMPOSITION.md, DATASET_LAYER_ADT.md, all removed — their content lives only here now). It answers three questions in one place: what did we decide and why (§1), what did we consider and reject, and what tradeoffs are still live (§2), and what's built vs. still open (§3). For the object model as it exists today (class shapes, method signatures), see CLASS_DIAGRAM.md and usage/USAGE.md — this doc is the why, those are the what.

Contents

1. Architecture decisions · 2. Tradeoffs, rejections, and open questions · 3. Implementation status · 4. Known gaps · 5. Incomplete / deferred work · 6. Cross-references


1. Architecture decisions (resolved)

1.1 The object model's shape

1.2 Instance-scoped DOM, event inversion, and the engine→host boundary

1.3 The floodExtent decomposition

layers/floodExtent.js (1561 lines) was the central dispatcher, not a single subsystem, so it could not migrate onto the Layer model by the same mechanical rename that worked for velocity/ensemble/depth/ damage. It tangled three concerns in one module, resolved to three homes:

Tangled concern What it looked like Resolved home
User-file layer registry (toolboxLayer) filename→rendered-file map, heterogeneous values (a USGSOverlay, a {setMap} vector proxy, or a delegating proxy for damage/ensemble/depth) fim.layers, keyed by filename (FimMap#layersByName + registerNamedLayer/getLayerByName)
Extent-scenario registry + slider (toolboxLayerExtent, _sliderSeq, showSlider/handleSlider*) model/stage/discharge/annualChance selection picking an extent raster a Scenario over axis-Datasets (§1.1's selection-axis fold-in)
Dispatcher/session context (arr_INUN, customExtentInputJson, ctaLayer2, a_val, current_map_id, …) active-scenario state, several exported as getters consumed cross-module (damage.js reads getAVal()) a per-FimMap session object (floodExtentSession.js), §1.2's WeakMap pattern

Why "the slider stays in the app" (a resolved design call): the package only validates and models a scenario and resolves a selection → a URL; a host keeps the actual slider UI. Scenario (an app-tier parseFimScenario adapter) is metadata + u + 1 generic engine Datasets (one stage-axis series per model + one time-axis series); resolveStage/resolveTime return a render-ready descriptor the existing fim.addDataset → addLayer path consumes unmodified.

The slider migration's replacement strategy: prefer the new path, self-heal on divergence. Rather than a flag-day cutover, each arr_INUN positional read was replaced one at a time with a call that prefers the Scenario read and falls back to arr_INUN, logging on any divergence — so a file that trips an edge case self-heals instead of breaking, and the log signals it happened. Three traps made this non-trivial (each would have been a silent, browser-only regression if the migration had skipped the divergence guard):

  1. Sort orderScenario sorts axis entries by stage; the legacy arr_INUN[2] read is in raw file order.
  2. null vs. ""parseFimScenario normalizes empty fields to null; arr_INUN kept "", and some downstream paths did string/number operations that treat these differently.
  3. Single-model slotarr_INUN[2][0] holds only the currently selected model, so the slider's a index is not a model index (an easy misread when porting the logic).

For the two real sample files (single-model, ascending), these traps were empirically inert, verified with byte-identical node tests against both the arr_INUN mapping and the Scenario read, then browser-confirmed at each step. The migration reached every reachable read in handleSliderChange/handleSliderInput/ showSlider, including the real-time nearest-stage scan and the metadata unit strings. What's left of arr_INUN is confirmed-dead code — an #achance checkbox + a dual-slider branch with no counterpart in the running app ("Annual Chance" there is a read-only text field, not an interactive slider); migrating unreachable code isn't testable and wasn't worth the risk. (annual_chance itself is sparse but genuinely present in some real files — it should not be treated as always-empty in parity/lookup logic elsewhere.)

A pre-existing bug fixed along the way (not caused by the migration): the Toggle Vector/Raster button stuck on vector because it carried two click handlers simultaneously — ui/ui.js's stateful .onclick (alternates raster↔vector) and a delegated data-action="toggleVectorRaster" hardcoded to 'vector'. Both fired; since the slider handler is async and guards itself with a monotonic _sliderSeq, the always-vector delegated handler happened to win the stale-load race. Fixed by dropping the redundant data-action — the lesson (see §4) is not to double-bind a delegated action and a programmatic .onclick on the same element.

2. Tradeoffs, rejections, and open questions

2.1 Map-provider abstraction — per-tier tradeoffs

mapProvider.js's seam (registerMapProvider) is landed for three of six tiers; the other three stay deferred. The tradeoffs recorded here recur for each remaining tier, so they're kept even though the specific tiers they were first decided for are done:

Tier Google Leaflet Status
Map + basemap google.maps.Map L.map + L.tileLayer
Vector (GeoJSON) google.maps.Data L.geoJSON
Static raster image google.maps.GroundOverlay L.imageOverlay
Live raster repaint remove + recreate GroundOverlay L.ImageOverlay#setUrl
Map hover/mousemove google.maps.event map.on('mousemove')
Velocity animated canvas custom OverlayView + rAF (none) 🔲 not ported
Comparison draw-mask tool DOM + getProjection() + Polygon/Marker (none) 🔲 not ported
HAZUS markers AdvancedMarkerElement (none) 🔲 not ported
Flood-depth tiles vendored arcgislink.js (none) 🔲 not ported

Decisions + rejected alternatives, per tier already built:

Why the remaining four tiers aren't a mechanical swap: Velocity's animated canvas is a full-viewport canvas repainted every frame by its own rAF loop, driven by five Google-specific listeners (resize/center_changed/dragstart/zoom_changed/idle) — the particle math is already provider-agnostic, but re-deriving "when does the field need rebuilding" from Leaflet's moveend/ zoomend/resize (different granularity/timing than Google's idle) is real design work, not a find-replace. The comparison draw-mask tool is a bespoke drawing interaction (raw DOM mouse events + getProjection().fromLatLngToPoint() + Math.pow(2, zoom) math + a Polygon/Marker snap indicator) — Leaflet's containerPointToLatLng is simpler, but it's a parallel micro-library rewrite, not a raster concern. Markers need a whole new marker tier (L.marker/L.divIcon), a clean-slate addition, not an extension of an existing one. Flood-depth's arcgislink.js is a vendored Google-only library with no portable internals — a Leaflet path means L.tileLayer at the ArcGIS REST export endpoint or Esri-Leaflet, a fresh integration each way.

Mapbox stays out of scope, deliberately: a canvas image-overlay fits Google/Leaflet's DOM-pane model, but Mapbox's WebGL sources/layers are a different rendering model entirely — a larger separate phase, not a third provider dropped into the same seam.

2.2 The interplay model — three axes, two kinds of combination

Three orthogonal axes to a rendered layer: data (Dataset), style (ColorScale), placement (Layer + provider). Two kinds of combination that must not be conflated:

Layer↔Dataset is the landed seam between them: layer.sources: Dataset[] are inputs; since Datasets are immutable, "the data changed" always means point the layer at a new Dataset (setSources/deriveSources), never mutate one in place. deriveSources exists specifically as the "hot-modify" path — derive from the current source so memoized ancestors are reused and only the changed tail of the op-chain recomputes, vs. a cold setSources swap that re-decodes from scratch. The render update mode ('auto'|'in-place'|'recreate') resolves from provider capability + layer type when 'auto'; a raster on a capable provider goes in-place (no flicker, keeps z-order/identity), everything else recreates. Correctness note that bit this exactly once: a stale async force must be dropped by the load-sequence guard (loadSeq/_sliderSeq) before it swaps the handle, or a slow load can overwrite a newer image on a reused in-place handle — 'recreate' is naturally immune (fresh handle each time), 'in-place' trades that immunity for smoothness, so the guard is load-bearing there specifically.

Dataset lifecycle is ref-counted by the owning FimMap (_acquireDataset/_releaseDataset) — a Dataset's memoized decode is evicted only when the last Layer referencing it drops it; Layers themselves never call ds.release() directly. Exclusive display claim (Layer.exclusive — true for velocity/ensemble): activating an exclusive layer tears down whatever previously claimed the slot (FimMap._claimExclusive, formalizing what ui/activeDisplayLayer.js already did ad hoc).

2.3 Storage's scope boundary — a deliberate non-feature

"Not a database library" is a scope decision, not a gap: no query DSL, no index management, no transaction API beyond put/get/list/map/clearAll. If a consumer needs more than that, the intended answer is build a StorageAdapter around a real database, not extend this class — the seam is kept small enough on purpose that such an adapter could slot in later without Storage itself growing indefinitely. Two kinds of version-management primitive are offered and deliberately not merged into one: map()/ clearAll() are DATA-migration tools needing no schema/version bump (out-of-line keys + verbatim values mean a record's shape was never part of the schema); the versionMigrate constructor hook is the STRUCTURAL counterpart (create/drop a store, copy rows between stores) — IndexedDB only allows that kind of change inside an onupgradeneeded transaction, so it can't be a plain method call.

2.4 Rejected: iframe isolation for multi-instance

An iframeMount.js ("Path A") was built, then deleted once in-DOM { isolated: true } (a separate in-page FimViz, Path B's DOM-scoping) covered the multi-instance need without the overhead and constraints of iframe boundaries (cross-frame API surface, styling isolation costs, no shared map SDK instance). There is no iframe option today, and none is planned — Path B's instance-scoped DOM (package/dom.js) made it unnecessary.

2.5 Still-open questions

3. Implementation status

3.1 Build status (what's built vs. planned)

Area Scope Status
Core FimViz ambient default + FimMap + create()/mount() + scoped $
Data Dataset, Storage, headless parseFile/addDataset (geojson+HAZUS, kml/kmz, shp, geotiff→WGS84)
Read-models ColorScale, Legend, Stats + the Filter family
Layers Layer objects with per-instance state (velocity, ensemble, depth, damage, comparison, floodExtent)
Headlessness Instance-scoped DOM + zero engine window.* calls (the engine emits; a host subscribes)
Native multi-instance mount() composed purely on the primitives + fully per-instance subsystems

The headless boundary depends on Layer state already being per-instance: state must live on Layer objects before the handlers that mutate it are rewired, or the DOM/event wiring gets done twice.

3.2 The engine→host boundary — how headlessness is enforced

Two independent properties make the "headless" claim real, and each is enforced, not merely asserted:

3.3 Storage — a generic KV store, disposable by policy

Storage is a small generic db/table/row KV mechanism over IndexedDB, not a database library: no query DSL, no index management, no transaction API beyond put/get/has/delete/list/clear/map. A host that needs more builds a StorageAdapter around a real database rather than extending this class. The engine names no database or table — a host names its own (§1.1, §1.2).

Standing policy: a schema change bumps the version and accepts a reset, not a migration framework — this is storage a host should treat as disposable. Storage still exposes the raw hook a host needs if it does want to migrate (versionMigrate runs inside the version-bumping open, with the upgrade transaction), but the engine itself ships no migration machinery. Read/error semantics are deliberately simple: a failed read and a missing key both surface as undefined — a host that needs to distinguish them wraps Storage rather than the engine growing richer error channels.

3.4 Ambient pointers — config retired; the DOM scope is the remaining blocker

The engine keeps no ambient config pointer. Config is read per-instance via fim.config (FimMap.config → the owning app's config object). The one setting that genuinely cannot be per-instance — gdalPath — has a narrow module-level holder (setGdalPath/getGdalPath), honest because gdal3.js compiles a single Emscripten module per page (GDAL is a per-page singleton), so the first mounted app's path wins by nature of the WASM runtime. The resolveUrl URL-resolution seam is threaded, not ambient: fim.addDataset binds the instance's config.resolveUrl into the parse options, and a lazy Dataset.fromURL root carries a resolver applied at force time — so two isolated apps never share a URL-rewriting rule. A test guards that no engine module references getConfig/setConfig/setActiveConfig. Relatedly, FimMap.get map() genuinely owns its map (mount.js calls _adoptMap() once boot resolves, instead of a module-level let map that made every FimMap on a page report the same map).

What still blocks true multi-instance is dom.js's _active scope, read by ~475 domId/domQs/domQsa calls (mostly a host-tier concern). That DOM scope needs threading per-instance; the config/map work above does not incidentally solve it.

4. Known gaps

5. Incomplete / deferred work

5.1 Phase 5 (not started)

Turnkey embedding of the full widget (packaging the UI tier as a separate library) is not planned — a bare mount() from the fimviz barrel alone gives a real, empty map but not the full widget chrome, since the bundled barrel self-registers no runtime; a host that wants the full widget registers its own runtime from its own composition root.

5.2 Dataset/Layer ADT — designed for, not built

5.3 Browser verification — what a real-Chrome pass found

Recorded because it's easy to lose track of what's been confirmed by hand vs. only by npm test, which does not cover google.maps rendering, Leaflet, canvas, or the GDAL WASM warp.

Note on the page names below. examples/ was rebuilt as six explanatory notebooks (01-quickstart06-storage-and-records), replacing the fourteen ad-hoc pages and the guided verify.html checklist this section was written against. The findings stand; the page names are historical. 05-ui-toolkit.html now covers what ui-tools.html and verify.html covered between them, and 04-temporal.html replaces temporal-netcdf.html — but the Pass/Fail capture and the Markdown report are gone, so a release pass is now "open the six pages and look", with no recorded verdicts.

Discharged on Leaflet by a headless-Chrome pass over every page in examples/ — all 14 load with a clean console, and ui-tools.html + temporal-netcdf.html were driven end to end (file load → tools panel → legend/stats → hover → dispatch → region draw → scoped stats; and scan → scrub → reduce across NetCDF4, Zarr and GRIB2). Four defects that npm test structurally could not see:

Fixed — the selection tools drew nothing, and the brush panned the map. The first browser pass over the four-mode selection tier found two defects that every one of the 52 headless tests had been blind to, for the same underlying reason: both live in the seam between the tool and the map, and the tool is headless by design.

Open — region draw drops its first vertex. Driving the selection step of the (now removed) verify.html with four clicks records only three, and the recorded ring is the last three corners: the first click after createRegionDraw().start() never reaches the capture handler. A user clicking the minimum three points therefore gets two and is told "need ≥3 points", which reads as the tool being broken. A second anomaly in the same trace is unexplained: the recorded vertices span twice the expected lat/lng range for their pixel positions — an exact 2× scale error, as if the click→LatLng conversion used a zoom one level below the displayed one. Both are visible in the page's own [verify] console trace; the vertex entries now carry the map's zoom and bounds at the moment of each click, which is the next thing to read. Not yet diagnosed to engine vs. example vs. Leaflet animation timing (#rg-draw calls rasterLayer.fit() immediately before start(), so an in-flight zoom animation is a live suspect for both symptoms).

Fixed (interim) — raster overlays were plate carrée content drawn into a Mercator viewport. RasterLayer._draw colorizes a grid to a data URL and hands it to addRasterImage, which is L.imageOverlay(url, [[s,w],[n,e]]) on Leaflet and a GroundOverlay on Google. Both stretch that image linearly in Web Mercator screen space, while our grid rows are evenly spaced in latitude. The two agree only near the equator, and the error grows with the extent's height:

dataset extent worst latitude error
idalia-nldas2.nc (regional) 25.06 – 36.94 °N 0.19° ≈ 21 km
CMIP tos (global ocean) −80 – +90 ° 25.67° ≈ 2850 km

This went unnoticed because every raster the library had rendered was a regional flood map ~12° tall, where the error is a couple of screen pixels — it took a global NetCDF3 to make it obvious. A second problem rides along: latitude 90 is infinite in Mercator (y = 37.3, against 3.14 at the conventional ±85.05° cutoff), so the top of such a file has nowhere to be drawn and the provider simply clamps.

The existing reprojection machinery does not address it, and it is worth recording why, because both plausible-looking routes are dead ends. resampleGrid maps destination pixels linearly in lat/lng (lat = bn - (dy+0.5)/h * (bn-bs)), so it is a plate-carrée→plate-carrée resampler and can never produce Mercator-spaced rows. And warping to EPSG:3857 is refused before it draws: both providers declare acceptsCRS as the WGS84 family only, so Layer._checkProviderCRS throws — with a message advising ds.reproject('EPSG:4326'), which walks the caller straight back into the bug. Even with that guard lifted, addRasterImage takes lat/lng bounds while a 3857 grid's are metres.

Landed: the row remap (geo/mercator.js, wired into RasterLayer._draw). Rows are resampled onto Mercator-even spacing before colorizing; columns are untouched because longitude is linear in Mercator. Only the image is reprojected — the source grid is left alone, so rasterData/meta, hover, Stats and the filters keep reading real values at real coordinates, which is what kept the change contained.

Still open: tiles, and the decision point now exists ahead of the backend. rasterRenderPlan returns mode: 'tiles' when the ideal image exceeds a pixel budget (16 Mpx) or a side limit (8192 px), both overridable per layer along with tileSize, strategy (auto/image/tiles), mercator and resample. Until a tile backend exists, that verdict means the image is drawn capped — correctly placed, aspect preserved, downsampled — plus a console.warn and a layer:raster-oversized host event. A baked image is fixed in resolution: zoom past what it was sized for and you are magnifying pixels, which no static heuristic can detect. Tiles (L.GridLayer#createTile / ImageMapType) resample per viewport and are the real answer; the plan is to build them alongside ArcGIS server data viewing, where the same tile plumbing is needed anyway. Reusable when that happens: resample.js's nearestAt/bilinearAt already take (pixels, meta, lng, lat, noData) — exactly the per-pixel query a tile needs — though they are module-private today.

Rejected: warping to EPSG:3857 through the existing reproject(). Architecturally the general answer, and the one to reach for if genuinely projected grids (polar stereographic, rotated pole) ever need to reach the map. But it is refused before it draws (acceptsCRS is WGS84-only on both providers), addRasterImage takes lat/lng bounds while a 3857 grid's are metres, and it would make every global raster depend on a ~38 MB wasm download to do four lines of trigonometry.

Still owed: everything Google-side — vector rendering and neutral-style translation on google.maps.Data (including the new point symbol), and raster overlay colorize/opacity/hit-test. Also the GDAL WASM reproject forced at a real terminal, and the http://[::1]:PORT loopback fix (confirm data actually fetches over IPv6 loopback, not just localhost).

The parts that need a human eye — legibility, tooltip tracking, gradient rendering, dispatch order, modal capture — are exercised by examples/05-ui-toolkit.html, which mounts every fimviz/ui widget on one map. There is no longer a guided checklist: verify.html, which numbered those checks and captured Pass/Fail into a Markdown report, was removed with the example rebuild. Its checks now have to be remembered rather than read, which is a real regression in release discipline and worth restoring if browser passes become routine. See examples/README.md.

5.4 Bigger, further-out additions

Event-filtering/interaction policy, more Dataset operations, additional/proprietary formats, new import/export transports — scoped separately in PACKAGE_ROADMAP.md, not repeated here.

6. Cross-references

What is public, and the one test for it (resolved)

The usage docs used to name thirteen functions the barrel didn't export — readable as promises the package wasn't making. Resolved by asking one question per name, not by exporting them all: does a consumer have a use for this that the public API doesn't already serve?

Only one passed. Dataset.formats() is now barrel-exported: "which formats can I decode right now" has no other answer, and a host needs it to build a file picker's accept list or to validate an upload before parsing. Layer.types() was added — a new name, not one of the thirteen — replacing the proposed hasLayerType export, because type is overloaded in this codebase (a registry key for addLayer, and separately layer.type's subtype discriminator, with 'depth'/'ensemble' meaning different things in each); a list named after the registry can't be misread the way a hasLayerType(type) predicate could.

Everything else stayed internal and the docs now say so:

The standing rule: a name reaches the barrel when it answers a question the public API can't, not because it happens to be exported from its module. Everything remains reachable through fimviz/src/* for anyone who really needs it — with no types and no stability promise, which is the honest signal that they are off the supported path.

A corollary, learned from parseSciwrid. The multi-dimensional formats (NetCDF/GRIB2/Zarr) spent their first iteration reachable only through fimviz/src/io/sciwrid.js, and the reason given was bundle payload: the reader carries a ~193 KB wasm that must not land in every consumer's initial download. But "off the barrel" was never what enforced that — a dynamic import() was, and it still is. Making the caller type the adapter's name bought nothing and cost the obvious thing: opening a .nc looked like a different kind of act from opening a .tif, and the vendor's name leaked into user code and error messages for no reason a user could act on. io/parse.js now routes those extensions itself, behind import("./sciwrid.js"), and dist/fimviz.js contains zero occurrences of "sciwrid" — the payload rule intact, the API surface honest. parseSciwrid stays internal, in exactly the sense parseSource is. The general lesson: when an internal name is the only way to do something ordinary, the boundary is in the wrong place — the fix is to serve the need publicly, not to promote the internal.