Dataset operations — usage reference

Dataset (barrel-exported) is a lazy, immutable node in an op-chain: a ROOT wraps a source (inline bytes/GeoJSON, or a URL); a DERIVED node is a parent + one op. Nothing decodes/warps/transforms until a terminal forces it. Every op returns a new Dataset — the original is never mutated, so one Dataset safely backs many Layers. kind is 'raster' or 'vector' — most ops are gated to one kind and throw immediately (not at force time) if called on the wrong kind.

Contents

Construction · Kind-agnostic · Terminals · Raster — unary · Raster — binary / N-ary · Vector · Notes · Standalone grid functions · Standalone warp() · Vendored primitives · GDAL escape hatch (callGdal)

Construction

new Dataset({ id?, name?, kind?, format?, crs?, bounds?, meta?, data?, url?, axis?, axes? })
// id: auto ("ds_...") · name: defaults to id · kind: 'raster'|'vector'|null
// format: 'geotiff'|'geojson'|'kml'|'kmz'|'shp'|'hazus'|null · crs: native CRS string|null
// bounds: {north,south,east,west}|null · data: ArrayBuffer|Object (inline payload) · url: a URI root
// axis: one DatasetAxis (sugar for axes:[axis]) · axes: DatasetAxis[] (selection-axis series)

Dataset.fromURL(url, { format?, name?, kind?, crs?, bounds?, meta? })
// A lazy URL root — fetches + decodes only on force. format/kind inferred from the URL extension
// when omitted; crs defaults to EPSG:4326 for vector formats, null (unknown) for raster.

Dataset.fromRecord(record)   // rehydrate a toRecord() snapshot (recipe or materialized)

Dataset.fromGrid(value, { name?, format?, meta? })
// A Dataset around an ALREADY-DECODED RasterGrid/VectorFeatures — one you got from ds.grid(),
// computed with the standalone grid functions, or built by hand. Pre-materialized: no fetch, no
// decode, no materializer needed, and load()/grid() return the value you passed in. `kind`/`crs`/
// `bounds` come from the value; `format` stays null, since there are no encoded bytes to claim one.
// This is the way BACK into the op chain, so `ds.grid()` isn't a one-way door.

Normally you don't construct directly — fim.addDataset(file) / FimViz.parseFile(source) parse a raw File/Blob/ArrayBuffer/URL into a Dataset for you.

Kind-agnostic

Axis entries: one file per entry, or one file many entries

An axis entry's ref says how to GET that entry's payload, and takes three forms. select() and reduce() behave identically across all three — the axis model doesn't care which you built:

// 1. A URL — one file per entry (the FIM Scenario shape: each stage/timestep its own raster).
{ coord: 19.5, ref: 'stage_19p5.tif' }                       // → a URL-rooted child

// 2. Named URL variants — pick one with select(coord, { variant }); ds.variantsAt(coord) lists them.
{ coord: 19.5, ref: { raster: 'a.tif', vector: 'a.kmz' } }    // → select(19.5, { variant: 'raster' })

// 3. An IN-FILE selector — the entry is a slice of the SAME source, not a separate download.
{ coord: 6, ref: { select: { variable: 'TMP', t: 1 } } }      // → a child sharing this file's bytes

Form 2 is not an axis, on purpose. A variant switches the Dataset's kind — .tif gives a raster in an unknown CRS, .kmz a vector in EPSG:4326 — whereas every axis preserves kind, CRS and bounds. It is a choice of encoding of the same datum, not a coordinate in the data. Use ds.variantsAt(coord) to enumerate what an entry offers (['raster', 'vector'], or null) rather than catching the error select() throws when you omit a required one.

Form 3 is what lets one multi-dimensional file (NetCDF/GRIB2/Zarr, every timestep inside it) back a whole temporal axis. The child shares the parent's bytes or URL — no second fetch — carries the same format, and the selection reaches the decoder as root.select:

const child = ds.select(6);      // ds.format 'netcdf4', ds.data the file's bytes
child.selector;                   // → { variable: 'TMP', t: 1 }
child.axes;                       // → null — a child is ONE payload, so it forces like any Dataset
await child.grid();               // materializer receives { kind:'inline', data, select: {…} }
await ds.reduce('mean').grid();   // every entry resolved + reduced — no extra machinery

A series is not forceable — load()/grid() on one throws, naming select()/reduce() as the way forward. That holds whether or not it carries the file's bytes, which an in-file series does:

const storm = ds.selectRange(t0, t1);   // → a NARROWER series, still lazy, still not forceable
await storm.reduce('max').grid();        // the peak within that window
storm.select(coord);                      // …or one step out of it

select's base applies to URL refs only. A selector ref may also carry name/crs/bounds to override what the child would otherwise inherit from its parent. Selecting on a Dataset with no source of its own (no data, no url) throws — there is nothing to select from. Selector children round-trip through toRecord()/fromRecord() like any root.

Writing a materializer that understands selectors is one if: root.select is simply absent for ordinary sources, so existing decoders are unaffected. See APP_STARTUP_ADVANCED.md → Materialize / decode extension seam.

Reading NetCDF / GRIB2 / Zarr

The concrete producer of selector axes — multi-dimensional scientific formats. They open through the ordinary fim.addDataset(file) / FimViz.parseFile(file), on extension (.nc, .nc4, .cdf, .grib, .grib2, .grb2, .zarr); the file's own header decides which of the four it is.

Under it is SciWrid Toolkit, reached through a dynamic import and left external to the bundle, so its ~193 KB wasm is downloaded only by a page that actually opens one of these files, and is never bundled into dist/fimviz.js. The consequence for a consumer is that sciwrid-toolkit must be resolvable — an npm dependency under a bundler, an import-map entry in a raw browser page (examples/04-temporal.html shows one). Only these four formats depend on it, and the thrown error says so.

One call covers every format — the differences live inside the adapter (GRIB2 reports nx/ny instead of a shape; Zarr reports shape as an array rather than a string). What differs for a caller is what scan() can tell us about a file, which is not uniform:

NetCDF3 gets its extent and its timestamps from a different place than the other three, and that is worth knowing about. scan() supplies neither — SciWrid populates a bbox only on its netcdf4/zarr/parquet paths, and its NetCDF3 time-units accessor is still outstanding upstream. But the files themselves carry lon/lat/time coordinate variables with CF units right there in the header, so io/netcdf3.js reads them. It is the one container format FIMViz parses itself, it reads the header only (never pixels — point it at a data variable and it refuses), and it declines any bytes that are not NetCDF-3 classic, so it cannot affect the other formats. meta.axisSource records where the axis came from: 'scan', 'header', 'caller', or 'index'. Pass header: false to switch the supplement off.

One consequence worth stating: this works from inline bytes only. A NetCDF3 opened from a URL is never fetched just to read its header — that would undo the laziness the URL path exists for — so it still needs grid.bbox.

Calendars that are not Gregorian keep the file's own numbers. A 360_day or noleap calendar has no real instants to convert to, so rather than fabricating dates or falling back to bare positions, the axis carries the raw offsets with the file's units as axis.unit — e.g. 'days since 2001-1-1' with coords 15, 45, 75…. Ordered, exact, and selectable; just not calendar dates.

Dimensions beyond (lat, lon) + time

T(time, level, lat, lon) is ordinary — ERA5, GFS, CMIP. The decoder exposes no way to pick a level, member or band, so a 4-D variable throws rather than silently handing you whichever slice the reader chose:

await fim.addDataset(file, { allowExtraDims: true });   // accept it; recorded on meta.extraDims

This is arithmetic on the declared shape, not on what the reader admits to — which is why a NetCDF3 file trips it even though scan() reports no time axis: its time dimension is invisible but still there, and was being collapsed unannounced.

const ds = await fim.addDataset(file);            // header only — nothing decoded
ds.format;                                         // 'netcdf4' — from the file, not the extension
ds.axis.entries.length;                            // 120 timesteps
ds.meta.grid;                                      // { width, height, bbox, bounds } — the file's NATIVE grid
ds.meta.unit;                                      // 'kg m-2'

const t = ds.select(Date.parse('2023-08-28T06:00:00Z'));   // → one grid, lazily, off the same bytes
await fim.addLayer(t);                                       // renders like any raster (already EPSG:4326)

await ds.reduce('mean').grid();    // temporal mean over the whole axis
await ds.reduce('max').grid();     // …or the storm peak

Which axis is the series axis (and why you cannot choose)

The series axis is always the file's outermost non-spatial dimension. That is not a simplification we chose — scan() reports a variable's shape as bare numbers ('24x170x180') with no dimension names, and the decoder exposes exactly one index knob. You can say how long that axis is, what its coordinates mean, and what to call it; there is nothing that could act on "use dimension 2 instead". A file whose layout genuinely differs needs dims.order (for the spatial pair) or a transposed copy.

Longitude conventions are applied by us, not by the reader

Asking the decoder for a [-180, …, 180] window on a 0..360 file returns the file's own pixels with the requested bbox echoed back — same data, new label, Pacific drawn where the Atlantic belongs. So lon is implemented as a column roll on the decoded grid:

• a global extent (span 360°) is genuinely rolled, half the grid width for the usual case;
• a regional extent a whole turn away (270..285 → -90..-75) is relabelled with no pixel work;
• anything else — a regional extent that would cross the antimeridian in the target window — throws, because expressing it would need the grid split and re-joined, and bounds cannot hold east < west.

await fim.addDataset(file, { grid: { bbox: [0, -80, 360, 90] }, lon: '-180..180' });
// bounds → { west: -180, east: 180, … }, and every decoded timestep is rolled to match

lon fixes where the grid claims to be. How it is drawn is a separate concern handled by geo/mercator.js: raster overlays are images placed in a lat/lng box, which Leaflet and Google both stretch linearly in Web Mercator, so the rows are resampled onto Mercator spacing before colorizing. That is automatic — a global field renders correctly with no options — and latitude beyond ±85.0511° is clipped away (it is not representable in Mercator) with the overlay box shrinking to match. Only the image is reprojected; hover, Stats and the filters keep reading the source grid.

The knobs, on addLayer(ds, { raster }) or layer.render({ raster }):

Tiles are not implemented yet. When the plan says tiles, the image is still drawn — correctly placed and aspect-preserved, but downsampled to fit the budget — with a console.warn and a layer:raster-oversized host event. A baked image is also fixed in resolution, so zooming far past what it was sized for magnifies pixels. See DECISIONS → raster overlays.

Axis coordinates are epoch milliseconds, not ISO strings — select()'s nearest-match is numeric-only, and that is what a time slider needs. The ISO timestamp is on each entry's meta.time, so ds.selectAxisEntry(Date.parse(iso)).meta.time reads it back.

No GDAL is involved: extractGrid resamples onto a geographic bbox, so these arrive as EPSG:4326 and render directly. Missing values arrive as NaN, which colorize and Stats already treat as absent. Rehydrating a stored selector Dataset skips the parser, so call registerSciwridFormats() once at boot before Dataset.fromRecord.

"has no usable geographic extent"

A common failure on real-world files, and a recoverable one. It happens for three different reasons, and the message says which:

• The format never reports one. scan() derives a bbox only on its netcdf4/zarr/parquet paths — so for GRIB2 and NetCDF3 it is always absent, whatever the file's grid looks like. Every file of those two formats needs an extent supplied. (This is a property of the reader, not of your file: don't go looking for curvilinear coordinates.)
• No bbox was derived (netcdf4/zarr). There scan() reads 1-D coordinate variables, so a curvilinear grid (2-D lat(j,i)/lon(j,i) — ocean tos products, NEMO, tripolar and rotated grids) or a Zarr store with no CF coordinates leaves nothing to place the data with.
• A bbox was derived and rejected. scan()'s matcher accepts variables named x/y, so a projected file (HRRR, RAP, NAM, WRF) reports its extent in metres — e.g. [-2699020, -1588806, 2697980, 1588806]. That is checked as degrees (|lat| ≤ 90, |lon| ≤ 360) and refused, because placing it would be silently wrong rather than loudly broken.

Not caught: a rotated-pole grid (CORDEX/COSMO, rlat/rlon) reports plausible small degree-like numbers that are not geographic. Detecting it needs the grid_mapping attribute, which scan() does not expose — so such a file will render in the wrong place without complaint. See PACKAGE_ROADMAP.md §8.1.

The pixels are perfectly readable; only the extent is unknown, so nothing can place them on a map. FIMViz will not guess one — a wrong extent silently puts every pixel in the wrong place, which is the same failure the CRS precondition exists to prevent. Supply it instead:

await fim.addDataset(file, { grid: { bbox: [-180, -90, 180, 90] } });   // width/height stay native

The thrown error names the variable, its shape, and the variables present, so you can tell which case you are in. examples/04-temporal.html has an "extent override" box that does exactly this.

Terminals (force the chain — nothing exists until one of these runs)

Reading the features out of VectorFeatures

A decoder may hand back a FeatureCollection, a lone Feature, or a bare array, so .features is whatever the format produced. Iterate the wrapper instead and the shape stops mattering:

const vf = await ds.features();

for (const feature of vf) console.log(feature.properties);   // iterable
vf.toArray();      // Feature[] in document order — normalized from all three payload shapes
vf.count;          // how many features
vf.features;       // the raw payload, if you specifically want the GeoJSON object as decoded
vf.bounds; vf.crs; // the frame it was decoded in

Raster — unary

All throw "<op>: raster-only op" if called on a vector Dataset.

Three kinds of reduction — which verb collapses what

Easy to conflate, so stated once. All three shrink something; they differ in what and by what:

groupBy is what "one variable as a series against another" means concretely. It is a distinct verb rather than an overload of the other two because its grouping key comes from data — not from the axis model, and not from geometry.

await depth.groupBy(landuse);                        // one row per distinct class
await rain.groupBy(dem, { bins: 10 });                // ten equal-width elevation bands
await rain.groupBy(dem, { bins: [0, 100, 500, 2000] });  // explicit edges
// -> [{ class, range?, count, sum, min, max, mean, area }, ...]

Note what is not here: producing a new grid from two variables is combine/difference (band math). groupBy is the table-producing half of the same question.

Raster — binary / N-ary

ds128.combine(ds177, { op: 'difference' });        // ds128 − ds177
ds128.combine([ds177, ds3], { op: 'mean' });        // N-ary
ds128.difference(ds177);                            // same as combine(..., {op:'difference'})

Vector — the one kind-changing op

Notes

• Kind gating throws immediately at op-build time (not on force) — a mistake is caught the moment you call the wrong op, not several awaits later.'
• Op-chain replay: every op above also has a matching case in Dataset.fromRecord's internal replay, so a toRecord()'d chain round-trips through Storage and rebuilds identically via fromRecord().
• Sharing: a "hot-modify" (e.g. via a Layer.deriveSources) that changes only the tail op reuses the shared parent's memoized decode — cheap compared to a cold new chain.
• Across maps: a Dataset has no back-pointer to a map, so one instance can back Layers on several maps (even of different providers) at once, decoded once. The registry and the reference count live on the app (app.datasets), so a Layer removed on one map never evicts a decode another map is still rendering.

Standalone grid functions (no Dataset needed)

Every raster Dataset op above is a thin lazy wrapper over a pure, directly-callable function on an already-decoded RasterGrid — barrel-exported, so if you already have a grid (e.g. from await ds.grid(), or built by hand), you can call these without a Dataset at all. Same signatures as the Dataset methods, minus the laziness — each takes a RasterGrid first and returns a new one:

Finished with a grid and want the op chain back? Dataset.fromGrid(grid) wraps it (above).

maskGrid(grid, polygon, opts?)          // opts: { invert? }
clipGrid(grid, bbox)
reclassifyGrid(grid, rules, opts?)      // opts: { unmatched? }
combineGrids(grids, opts?)               // grids: RasterGrid[]; opts: { op?, method? }
zonalStats(grid, zones, opts?)          // → data, not a RasterGrid
slopeGrid(grid, opts?)
aspectGrid(grid)
hillshadeGrid(grid, opts?)
rasterizeFeatures(featureCollection, bounds, opts)   // opts: { width, height, field?, burnValue? }

Resample/align (geo/resample.js, also barrel-exported):

GRID_POLICY                 // { LOW, HIGH, AVERAGE } — target-resolution policy
RESAMPLE_METHODS             // ['nearest','bilinear','average', 'cubic','cubicspline','lanczos','mode','min','max','med','q1','q3']
resolveTargetGrid(metas, policy)          // metas: [{width,height,bw,bs,be,bn}, ...] → the common target grid
resampleGrid(pixels, srcMeta, dstMeta, opts?)   // opts: { method?, noData? }
alignRasters(rasters, opts?)              // rasters: [{pixels,meta}, ...] → { grid, rasters: [{pixels,meta}, ...] }
Dataset.registerResampler(fn)   // the escape hatch for cubic/lanczos/etc. — nothing registered by default

Colorize (package/rasterImage.js, also barrel-exported — see COLOR_SCALE.md for the ColorScale side of this):

colorizeGrid(grid, opts?)     // opts: { colorScale?, alpha?, skipZero?, noData? } → Uint8ClampedArray RGBA
gridToDataURL(grid, opts?)    // colorizeGrid + canvas encode → a PNG data URL

Standalone warp() (the EAGER twin of Dataset.reproject())

import { warp } from 'fimviz';   // geo/warp.js — NOT the same as ds.reproject(toCrs)

const warped = await warp(ds, toCrs);   // async — warps immediately, returns a NEW Dataset

Dataset.reproject(toCrs) (documented above) is lazy — it builds an op node, and the warp only runs when something forces the chain. This standalone warp(ds, toCrs) is the eager equivalent — it warps immediately and returns a new, already-reprojected Dataset (still rasters only; same-CRS request returns ds unchanged, no copy). Both dispatch to the same GDAL warp underneath; pick the lazy method for a chain you're building up before rendering, or this function when you want the result right away.

Vendored primitives

Re-exported so the engine stays the single owner of these third-party dependencies — a host never imports geotiff/@tmcw/togeojson/shpjs/@googlemaps/js-api-loader directly:

fromArrayBuffer   // geotiff's buffer → GeoTIFF image accessor (pixel-level access beyond parseFile)
kml               // @tmcw/togeojson's KML XML DOM → GeoJSON
shp               // shpjs's zipped shapefile buffer → GeoJSON
Loader            // @googlemaps/js-api-loader's script loader class

GDAL escape hatch (callGdal)

Dataset.reproject/resampleTo (above) cover the one GDAL utility (gdalwarp) most callers need. For anything else gdal3.js offers — gdal_translate, gdal_rasterize, ogr2ogr, gdalinfo, ogrinfo, gdaltransform, or gdalwarp options those two don't expose — callGdal(method, ...params) dispatches to any method on the gdal3.js instance by name, passing your params through verbatim. No fimviz-specific wrapper exists (or is needed) per GDAL utility:

import { callGdal } from 'fimviz';

const file = new File([arrayBuffer], 'in.tif', { type: 'image/tiff' });
const { datasets } = await callGdal('open', file);
const outPath = await callGdal('gdal_translate', datasets[0],
  ['-of', 'GTiff', '-outsize', '50%', '50%'], 'out.tif');
await callGdal('close', datasets[0]);
const bytes = await callGdal('getFileBytes', outPath);   // Uint8Array

gdal3.js's methods fall into a few parameter shapes (see its own index.d.ts for exact signatures): lifecycle — open(fileOrFiles, options?, VFSHandlers?) → {datasets, errors}, close(dataset), getFileBytes(path), getOutputFiles(), getInfo(dataset); dataset-based utilities, output written to gdal3.js's virtual FS — gdalwarp/gdal_translate/ gdal_rasterize/ogr2ogr(dataset, options?, outputName?); info-only, no output file — gdalinfo/ogrinfo(dataset, options?); no dataset at all — gdaltransform(coords, options). An unknown method name throws immediately, listing every real method available.

callGdal dynamically imports geo/gdal.js (→ gdal3.js, ~38 MB wasm) on first call — never just from importing the fimviz barrel, so a consumer who never calls it pays nothing for GDAL, same as the reproject seam. Browser-only (gdal3.js's Emscripten loader fails in Node), so — like reproject and the warp path generally — this isn't covered by npm test; verify it by exercising the site or an example page.

callGdal is deliberately raw — it dispatches by method name but leaves the whole open/run/close/ read-bytes lifecycle to the caller, spelled out in full above. For gdalwarp specifically, warpTo and warpToGrid already hide that ceremony.