Skip to main content

MDV integration roadmap

This page explains architecture and phased context. For the branch-level shipping checklist, see MDV release checklist.

This library is intended to align with MDV and Vitessce so both projects can render SpatialData images, labels, shapes, and points from shared packages instead of relying on diverging local implementations.

The near-term target is not a full replacement for every MDV spatial feature. The target is a baseline 2D SpatialCanvas-backed MDV chart that can stand in for VivScatterComponent where appropriate, accept MDV-controlled state, compose MDV custom deck.gl layers, and avoid showing this repo's demo/editor UI inside MDV. 3D orbit/volume rendering is out of scope for this integration path. MDV is the first "use it in anger" sanity check; Vitessce compatibility remains a priority design target rather than a later afterthought.

Current state​

@spatialdata/vis exports both the full SpatialCanvas UI and a headless SpatialCanvasViewer (plus useSpatialCanvasRenderer, composeSpatialDeckLayers, and related helpers) from the public package entry. The rendering path matches MDV's existing MDVivViewer pattern:

  • image layers are rendered through Viv DetailView
  • extra deck.gl layers are composed above images
  • non-image spatial layers use the same deck.gl view model
  • shape, point, and labels renderers exist
  • shapes support table-driven fill colour and per-feature state on ShapesLayerConfig
  • @spatialdata/core exposes FeatureTableAlignment / createFeatureTableAlignment()
  • @spatialdata/layers owns shape deck rendering and buildShapeFillColorByFeatureId()

Remaining MDV blockers are packaging and polish, not the absence of a headless component:

  • publish and smoke-test the npm next prerelease in MDV
  • tooltip/pick row resolution should finish converging on the shared core resolver
  • points layer is minimal scatter only (v1.1 for MDV parity)

Viv/deck stack: this repo pins @hms-dbmi/viv@0.21.0 and deck.gl 9.2.9. MDV's matching upgrade is almost ready to merge — treat shared Viv/deck/luma versions as effectively aligned for the first @spatialdata/vis smoke test. Residual risk is extension-specific visual regression, not version skew between the two codebases.

SpatialCanvas still owns a full UI shell for demos. MDV should embed SpatialCanvasViewer and drive view state, layers, style, filtering, highlighting, and custom overlays from chart/data-store state.

Priority: headless first​

The top priority is a headless viewer path. Before MDV or Vitessce integration work gets clever, this repository should prove that the renderer can be driven by external state and external controls.

The first implementation should use a bridge path rather than a full layers-first rewrite: expose a public React SpatialCanvasViewer that reuses the existing Viv/deck rendering stack, accepts externally controlled state, and composes caller-provided deck.gl layers above SpatialData-rendered layers. This keeps MDV progress unblocked while preserving a migration path toward @spatialdata/layers.

The layers-first direction is still likely the cleaner long-term architecture, especially for MDV and Vitessce. For now, @spatialdata/layers should mature through narrow renderer slices rather than becoming the blocking path for headless mode: the current SpatialLayer package defines a schema/contract, but real image, labels, shapes, and points sublayer factories still need to be ported deliberately.

Useful in-repo validation demos (see Headless viewer guide):

  • demo/headless: local blobs.zarr fixture via SpatialCanvasViewer (no SpatialCanvas sidebars).
  • demo/headless-leva: external control panel variant of the above.
  • demo/headless-custom-layers: pass arbitrary deck.gl layers into the viewer, matching the way MDV will pass scatter, gates, contours, and ROI JSON.
  • demo/headless-controlled-view: keep view state fully controlled by an outer component, including programmatic pan/zoom/reset and saved/restored state.
  • demo/headless-tooltips: disable internal tooltip UI and route picking into an externally owned tooltip renderer.

Acceptance signal: demos should import the same public API MDV uses (SpatialCanvasViewer from @spatialdata/vis). That bar is met; the gap is example apps, not missing exports.

Phase 0: rendering stack and package sanity​

  • Confirm package names, entry points, and exports for local MDV consumption:
    • @spatialdata/core
    • @spatialdata/react
    • @spatialdata/layers
    • @spatialdata/vis
    • optional: @spatialdata/avivatorish
    • lower-level dependency: zarrextra
  • Decide whether MDV first consumes via local npm link / packed tarballs / workspace path / published prerelease.
  • Treat this repo as the place where Viv/deck/luma versions are selected. MDV should follow those versions, not constrain them.
  • Upgrade to @hms-dbmi/viv@0.21.0 / deck.gl 9.2.9 in this repo (Viv PR hms-dbmi/viv#924: uniform-buffer-backed shader props, model.shaderInputs, variable channel counts).
  • MDV Viv/deck/luma upgrade to the same stack — almost ready to merge; treat as done for integration planning.
  • Residual extension audit on both sides after MDV merge (shader-input paths, channel-count assumptions, extension prop passthrough through deckProps / SpatialCanvasViewer).
  • Run a clean pnpm build and pack the packages, then install them into ~/code/www/MDV.
  • Add one minimal MDV smoke chart that imports the package and renders a known fixture before attempting a full chart replacement.

Phase 1: headless SpatialCanvas API​

Mostly landed. @spatialdata/vis exports SpatialCanvasViewer as a separate controlled component (not a mode prop on SpatialCanvas).

Current surface:

<SpatialCanvasViewer
  spatialData={spatialData}
  coordinateSystem={coordinateSystem}
  renderStack={renderStack}
  viewState={viewState}
  onViewStateChange={setViewState}
  hostLayerResolver={resolveMdvHostLayer}
  deckProps={deckProps}
  renderTooltip={false}
  onFeatureHover={onFeatureHover}
/>

Implementation status:

  • Split viewer core (SpatialCanvasViewer / useSpatialCanvasRenderer) from SpatialCanvas UI shell.
  • Export SpatialCanvasViewer from public @spatialdata/vis entry.
  • Controlled coordinateSystem, renderStack, and viewState.
  • Host overlay descriptors plus hostLayerResolver for MDV scatter, gates, contours, ROI overlays.
  • Deprecated compatibility path for layers, layerOrder, and deckLayers.
  • showLoadingOverlay and renderTooltip={false} for external tooltip ownership.
  • onFeatureHover / onFeatureClick for shape and label picking, with onShapeHover / onShapeClick kept as compatibility callbacks.
  • Stable externally supplied Viv/deck view id or layer id suffix (getVivId compatibility).
  • Viv extension passthrough: ImageLayerConfig.vivLayerProps, vivImageExtensionResolver, vivImagePropsResolver on SpatialCanvasViewer (see Issue #56 APIs below).

See Headless viewer guide for experimentation steps. The /headless demo (packages/vis/demo) wires MDV-style ColorPaletteExtension + VivContrastExtension through vivImageExtensionResolver; brightness/contrast sliders feed vivImagePropsResolver (or saved vivLayerProps).

Issue #56 channel and extension APIs​

Tracked in GitHub issue #56. These APIs let MDV replace its local MobX↔Zustand channel bridge.

Channel config (Phase 1 ADR)​

  • Struct-of-arrays on renderStack.entries[*].props.channels: channelIds, colors, contrastLimits, channelsVisible, selections.
  • Array-of-structs deferred to vNext.
  • ChannelConfig does not carry extension-specific fields. Host apps own extension schema.

@spatialdata/avivatorish​

  • useLayerChannelState({ config, defaults, layerId, onChannelsChange }) — echo-safe layer-local channel adapter with stable channelIds, add/remove, parallel-array sync.
  • getChannelSelectionStats / getSingleSelectionStats({ includeRaster: true }) — optional 2D raster { width, height, data } for histogram UI.

@spatialdata/vis​

  • useImageLayerContext(elementKey) — per-element loader, defaults, OME channelNames (inside SpatialCanvasViewer).
  • useLayerChannelState re-exported from @spatialdata/vis for single-package MDV imports.
  • ImageChannelPanel remains internal to SpatialCanvas for now (not exported from @spatialdata/vis); MDV should use useLayerChannelState with its own UI.

Viv extension passthrough (extension-agnostic)​

Serializable on image stack entry props:

props: {
  channels: { /* core Viv channel fields only */ },
  vivLayerProps: { brightness: [0.5], contrast: [0.5] }, // host-owned keys
}

Runtime on SpatialCanvasViewer:

<SpatialCanvasViewer
  vivImageExtensionResolver={({ layerId, elementKey, channelCount, loader, channels }) => [
    new ColorPaletteExtension(),
    new VivContrastExtension(),
  ]}
  vivImagePropsResolver={({ channelCount }) => ({
    brightness: toneBrightness,
    contrast: toneContrast,
  })}
/>

Merge order into detailView.getLayers({ props }): saved vivLayerProps → vivImagePropsResolver → extensions from vivImageExtensionResolver (or global vivImageExtensions). VivSpatialViewer must not spread layer.props afterward.

Phase 2: MDV adapter chart​

Create a new MDV React chart alongside VivMdvRegionReact rather than replacing it in place.

Possible MDV chart name: SpatialDataMdvRegionReact.

Adapter responsibilities:

  • derive source / SpatialData store URL from MDV project state
  • select the active coordinate system from MDV region metadata
  • map MDV chart config to renderStack.entries
  • map MDV view state to SpatialCanvas view state
  • resolve existing MDV deck layers from host overlay descriptors:
    • scatterplot layer
    • grey/background scatter layer
    • selection layer
    • gate display layer
    • gate label layer
    • contour/field layers
    • ROI GeoJSON layer
  • preserve MDV chart linking via useViewStateLink
  • keep MDV tooltip behavior via useOuterContainerDeckTooltip
  • keep MDV keyboard/selection behavior around the viewer container

The first-pass chart can deliberately avoid replacing the channel dialog and most image editing UI. It only needs enough layer config to render an image/labels/shapes baseline and enough view-state synchronization to sanity-check against the current Viv chart.

Render stack and MDV state model​

The next MDV pass should treat @spatialdata/layers RenderStack as the canonical saved/render order:

const renderStack = {
  schemaVersion: 1,
  entries: [
    { kind: 'spatial', id: 'image', source: { elementType: 'image', elementKey: 'morphology' } },
    { kind: 'host', id: 'deck:scatter', source: { hostLayerId: 'deck:scatter' } },
    { kind: 'spatial', id: 'labels', source: { elementType: 'labels', elementKey: 'cells' } },
  ],
};

Host overlays are descriptors in saved config. MDV owns the runtime resolver that turns deck:scatter, gates, selections, contours, or ROI descriptors into deck.gl layers. This avoids storing raw deck layer instances in project state and avoids parallel stackOrder / layerOrder arrays.

Listeners follow the same split. Saved renderStack.entries[*].props should contain serializable renderer input only. MDV-owned callbacks such as onFeatureHover / onFeatureClick, tooltip portal wiring, and host overlay factories are runtime attachments supplied beside the stack. The feature events cover shapes and labels and include the same tooltip payload that built-in SpatialData.js tooltips use, so MDV does not need to decode deck.gl pick objects for normal table-backed hover/click interactions.

Constrained MobX​

MobX is acceptable for MDV-facing direct-edit control state, but it should be kept out of @spatialdata/layers and out of the default @spatialdata/vis renderer contract:

  • Use small MobX control islands for layer rows and controls.
  • Let each observer component read only the observable fields it renders.
  • Do not pass broad observable stack/config objects into MUI, dnd-kit, deck.gl, Viv, or non-observer components; pass plain values at those boundaries.
  • Sliders should patch renderer-visible props immediately and persist config intentionally, not through periodic whole-stack snapshots.
  • Avoid toJS on large stack/config objects during drag interactions.

React Compiler adoption makes this boundary more important. MobX observer components rely on observable reads rather than React's immutable-prop memoization assumptions, so MobX-heavy components should be explicit islands that can be excluded from compiler optimization or marked with "use no memo" where needed. The default SpatialData.js viewer path should remain plain-object and compiler-friendly.

Acceptance for the MDV UI path is separate from tile-fetch correctness: opacity dragging should not re-render the whole layer dialog/list, should not show a periodic stop-start GUI cadence, and should not create large stack snapshots on every granular mutation.

Tables, AnnData.js, and zarrita DataLoader​

MDV chart config ultimately drives table-backed shape colour, tooltips, and filters. The shared contract spans three packages:

ConcernOwner
region / region_key / instance_key semantics, row alignment@spatialdata/core
Column → per-feature colour maps, deck feature state@spatialdata/layers
Which column to show, loading columns for UI, passing config into the viewer@spatialdata/vis / MDV

Today: TableElement loads AnnData stores via anndata.js on top of zarrita (readZarr on the table's zarr subtree). Association helpers (loadFeatureRowIndexByFeatureIndex, loadAssociatedTableFeatureRows) use targeted obs-column loaders rather than a single high-level DataLoader API.

Near-term roadmap: expose coherent tables access through AnnData.js (and zarrita-backed DataLoader where appropriate) so integrators can read obs, var, selected X columns, obsm, and future uns / obsp surfaces without ad hoc string-column lookups or dropping to getAnnDataJS() for every chart. Richer query patterns should push upstream into anndata.js where possible; SpatialData.js should thin-wrap table elements and preserve Python association semantics.

MDV should pass resolved featureState and fillColorByColumn on the target shapes stack entry props when selection or colour mappings change — not re-load geometry. See Feature table associations.

Phase 3: layer styling, filtering, and highlighting​

The current SpatialData renderers mostly accept constant style values:

  • shapes: fillColor, strokeColor, strokeWidth
  • points: pointSize, color
  • labels: channel colors/opacities/stroke widths

MDV needs feature-aware styling:

  • color labels/shapes by MDV filter state
  • fade or hide filtered-out features
  • highlight selected/hovered/gated features
  • color by categorical or numeric table columns
  • keep image channels independent from feature-layer styling

Required changes:

  • Define a stable feature identity model for shapes and labels.
    • shapes now preserve row/feature ids in a render-oriented core payload and expose them through shared layers picking/styling helpers
    • labels need label ids mapped to associated table rows
  • Extend layer configs with feature-state props keyed by stable feature id.
  • Decide where MDV-specific filtering is evaluated:
    • MDV computes style arrays and passes them in
    • or shared package accepts filter/highlight sets and evaluates them
  • Add picking metadata that MDV can map back to datasource rows.
  • Add tests for picked shape/label id -> associated table row.
  • Add a fast path for large tables and many polygons. Avoid per-feature MobX reads in deck.gl accessors.

Suggested first design:

type FeatureStyleState = {
  visibleIds?: Set<string | number>;
  highlightedIds?: Set<string | number>;
  selectedIds?: Set<string | number>;
  colorById?: Map<string | number, [number, number, number, number]>;
};

MDV can compute this state from its datastore and filters, while @spatialdata/layers now owns the deck-facing shapes implementation, pick/tooltip resolution, and backend choice. @spatialdata/vis consumes that shared path.

For the adapter boundary, MDV should be able to own only serialized featureState, layer order/visibility, and any precomputed style/filter values keyed by stable featureId, without needing to know which shapes backend is active underneath.

Phase 4: nicer GUI in MDV​

Once the headless viewer works, MDV can selectively reuse or replace pieces of the current SpatialCanvas UI:

  • layer visibility and order
  • image channel controls
  • labels channel controls
  • tooltip-field selection
  • color-by controls for associated tables
  • shape/label filter/highlight controls

The current in-component sidebars should not appear inside MDV. Reusable panels should become standalone exports that accept controlled props and callbacks, so MDV can place them in its settings dialogs or chart menus.

Vitessce as a design target​

Vitessce is a priority consumer for the design, even though MDV is the first staging environment. The point of integrating into MDV first is to get a concrete sanity check under real application pressure, not to produce an MDV-specific architecture.

Current signals from ~/code/www/vitessce:

  • Vitessce currently uses older Viv/deck/luma versions than this repo, with Viv 0.16.x, deck.gl 8.8.x, and luma.gl 8.5.x in its workspace lockfile.
  • Its spatial views already compose Viv image layers, scale bars, expression/scatter layers, selection layers, and channel/controller state.
  • It has both current and beta spatial view paths, plus accelerated/3D paths that are likely out of scope for the first headless 2D viewer.

Likely requirements for Vitessce usefulness:

  • A headless React viewer with no MDV assumptions, no MobX dependency, and no app-specific tooltip or settings UI.
  • A plain layer/props schema that can be produced from Vitessce coordination values.
  • Compatibility with Vitessce's channel controller concepts: selections, colors, visibility, contrast/domain, colormap/rendering mode.
  • Explicit hooks for custom deck.gl layers and layer ordering.
  • A renderer API that does not require @spatialdata/react context if Vitessce wants to resolve/load data through its own loaders.
  • A separate investigation for 3D/volume paths after the 2D image/labels/shapes story is stable.

For now, the Vitessce note should be treated as an API pressure test and a success criterion: if the headless demo API can plausibly be driven by MDV chart config, Leva state, and Vitessce coordination values, it is probably the right shape. If it only works naturally in MDV, the abstraction is too narrow.

Arrow, Parquet, and deck.gl boundaries​

We should track upstream deck.gl / loaders.gl / deck.gl-community work around Arrow, GeoArrow, and GeoParquet carefully. This affects the boundary between @spatialdata/core, @spatialdata/layers, @spatialdata/vis, deck loaders, and app-specific adapters.

Current upstream read:

  • geoarrow/deck.gl-geoarrow is the renamed home for the former geoarrow/deck.gl-layers project. The published package to evaluate is @geoarrow/deck.gl-geoarrow; it targets deck.gl 9 and Apache Arrow JS.
  • The useful layer for SpatialData points is likely GeoArrowScatterplotLayer, but it expects GeoArrow point/multipoint data, not arbitrary x / y columns. SpatialData points currently store coordinate columns in Parquet, so an adapter still has to build or expose a GeoArrow point column/batch.
  • The library is most useful when we can keep Arrow chunks columnar all the way to deck.gl's binary attribute interface. It is less compelling if we first materialise every point as JS objects or as the current ndarray-ish wrapper.
  • deck.gl-community's Arrow layers are a second signal in the same direction, but the community docs explicitly warn about maintenance bandwidth. Treat that as an API pressure test rather than a dependency to bet the public contract on.

Current local state:

  • @spatialdata/core currently loads Parquet bytes/tables through parquet-wasm in VTableSource, inherited by points and shapes sources.
  • points currently return an ndarray-ish { shape, data } object with axis columns loaded from Parquet.
  • shapes currently expose a render-oriented core payload with stable feature ids, shared row-index alignment, and a mixed backend path in @spatialdata/layers.
  • labels are still their own image/tile rendering path.
  • Vitessce-derived code already has more advanced point handling in places, including tiled point loading, viewport filtering, feature-index filtering, and DataFilterExtension use.
  • VTableSource recognises points/<key>/points.parquet and points/<key>/points.parquet/part.0.parquet, but it does not yet model a multi-file Parquet dataset as multiple chunks/batches. That is the wrong shape for large point stores.

Points-specific target shape:

  • core should expose a PointsRenderData-style payload, parallel to ShapesRenderData, with stable point ids, row-index alignment, coordinate axis names, optional feature_key / instance_key columns, and the original Arrow table or record batches when available.
  • layers should own a shared points renderer. The renderer should choose between:
    • a current fallback ScatterplotLayer over typed coordinate arrays
    • a binary deck.gl attribute path for x / y / optional z
    • a GeoArrowScatterplotLayer path when data is already GeoArrow point encoded, or when the adapter can build that point column without copying too much
  • Points need the same feature-state language as shapes: hide, fade, color, radius, and filtered opacity by stable point id or row index. MDV/Vitessce filters should update feature-state or filter columns; they should not force a full Parquet reload.
  • Viewport/row-group filtering belongs behind the points data adapter, not in SpatialCanvas UI code. A multi-file Parquet directory can naturally map to progressive chunks/layers first, then later to row-group or bounding-box pruning when metadata is available.

Upstream signals to monitor:

  • deck.gl layers can load via loaders.gl and every layer supports loadOptions, but deck.gl core layers still generally consume arrays, binary attributes, flat GeoJSON, or app-supplied data rather than owning all Parquet semantics.
  • @loaders.gl/parquet / GeoParquetLoader exists but is documented as experimental/beta, with important limitations around large files and partial filtering.
  • deck.gl-community's Arrow layers accept Apache Arrow / GeoArrow tables directly and use deck.gl's binary data interface to avoid intermediate JS objects.
  • GeoParquet and GeoArrow are closely related but not identical. GeoParquet commonly stores geometry as WKB today; GeoArrow-native encodings are the direction to watch.
  • Foundational work on deck.gl v10, loaders.gl v5, and luma.gl v10 points even more strongly toward native Arrow / GeoArrow handling as a first-class direction:
    • deck.gl v10 is expected to rely more heavily on shared columnar GPU data handling rather than bespoke per-layer CPU-side mapping
    • loaders.gl v5 is introducing optimized Arrow / GeoArrow load paths across loaders
    • luma.gl v10 is exploring Arrow-like GPU-side data structures, batched GPU data, and GPU-side column transformations so full binary uploads can stay columnar even when raw Arrow blocks do not match GPU attribute/storage-buffer layouts directly
    • the practical implication for us is that representation-agnostic public APIs matter even more: we want our contracts to survive a shift from today’s fallback/bespoke handling to more native Arrow-to-GPU paths later

Boundary options:

  1. core owns Parquet and returns simple JS/typed-array structures.

    • This keeps apps and renderers decoupled from deck.gl.
    • It is easiest to test and use outside visualization.
    • It risks duplicating work that upstream deck/loaders/arrow layers may solve better.

  2. core owns data discovery and returns Arrow tables/vectors.

    • core remains deck-free but exposes columnar data close to the source format.
    • vis can choose deck.gl binary attributes, deck.gl-community Arrow layers, or fallback JS accessors.
    • This probably gives the best medium-term boundary.

  3. vis owns deck-specific loaders and consumes URLs/metadata from core.

    • This lets deck/loaders evolve naturally.
    • It risks making non-visual core APIs too thin for MDV/Vitessce table association, feature ids, and filtering.

  4. apps own their existing loaders, and vis only receives ready-to-render layer data.

    • This is useful for Vitessce compatibility and headless demos.
    • It does not solve shared SpatialData loading unless paired with another path.

Recommended direction for now:

  • Keep @spatialdata/core free of deck.gl dependencies.
  • Align @spatialdata/core's points and shapes support with Vitessce's SpatialData-derived loaders so we do not fall behind format coverage while the rendering backend evolves.
  • Move toward core exposing Arrow-ish columnar primitives for points/shapes/tables, while preserving convenience methods for simple JS arrays. For points, that means preserving Arrow batches/vectors alongside the current coordinate-array convenience path.
  • Make @spatialdata/layers responsible for choosing the rendering backend:
    • current polygon fallback for compatibility
    • current geoarrow-table runtime branch for shared columnar payloads
    • @geoarrow/deck.gl-geoarrow as the intended stronger near-term fast path when the external dependency is adopted cleanly
    • future Arrow/community-layer backends without changing the public shapes config
  • Keep feature identity, table association, coordinate transforms, and metadata interpretation in core; keep GPU filtering, tiling, layer construction, and picking/render props in layers.
  • Avoid baking parquet-wasm as the only long-term path. Treat it as the current implementation behind a replaceable interface.
  • Upgrade points before adopting GeoArrow broadly: first add stable point identity, row-index alignment, feature-state filtering/styling, and multi-part Parquet discovery; then add the GeoArrow renderer as an adapter behind the same public points config.

One more API-design note to preserve: the current table-association helpers are still obs-oriented and string-column-oriented because that is enough for the first feature-id join path. A future revision should widen the shared contract so style/filter inputs can come coherently from all of the AnnData surfaces we care about: obs, var, selected X columns for chosen var rows, obsm, potential future obsp graph/network data, and uns, without forcing integrators to encode everything as ad hoc string column lookups. In principle, many of those richer access patterns should likely be pushed upstream into anndata.js rather than duplicated forever in SpatialData.js.

  • Treat Vitessce parity tests as compatibility fixtures: when Vitessce supports a points/shapes SpatialData layout, core should either support it too or document why not.
  • Add a small compatibility matrix for points and shapes:
    • WKB GeoParquet -> JS fallback
    • WKB GeoParquet -> decoded Arrow/GeoArrow if available
    • GeoArrow-native geometry -> Arrow layer or binary deck attributes
    • tiled/row-group-aware points -> Vitessce-style path or upstream deck/loaders path

Open questions:

  • Should PointsElement.loadPoints() return Arrow vectors/tables in addition to typed arrays?
  • Should PointsElement expose a chunked/multipart API (loadPointBatches) so large points.parquet/part-*.parquet directories can progressively render without pretending they are one file?
  • Should ShapesElement.loadPolygonShapes() preserve feature ids alongside geometry in a first-class row object or columnar structure?
  • Where should row-group / viewport filtering live: core, vis, or app adapter?
  • Can deck.gl-community Arrow layers become a dependency of @spatialdata/vis, or should they be optional peer/adapter code?
  • How do we keep MDV filter/highlight state efficient if the underlying feature data is Arrow vectors rather than JS arrays?
  • How much of Vitessce's point tiling/filtering should be upstreamed into shared vis utilities versus left as a Vitessce-specific adapter until deck.gl upstream stabilizes?

Images, labels, and deck.gl-raster​

Viv is still the current image rendering foundation in this repo, MDV, and Vitessce-derived paths, but we should not assume it remains the only viable long-term raster path.

Upstream deck.gl-raster is moving quickly:

  • PR developmentseed/deck.gl-raster#467 merged an AlphaEarth Foundations GeoZarr mosaic example using ZarrLayer.
  • That example renders user-selected RGB composites from Zarr data, uploads 64-band tiles to a Texture2DArray, and performs band selection / dequantization / rescaling in shader code.
  • Recent deck.gl-raster releases also add higher-level GeoTIFF/COG and multi-band COG support.

Implications for us:

  • Track whether deck.gl-raster's ZarrLayer / raster pipeline can cover parts of our image rendering without Viv.
  • Track whether labels/segmentation rasters can be represented as a specialized raster shader pipeline rather than a Viv-derived labels layer.
  • Keep image/labels props in @spatialdata/vis abstract enough that the backend can be Viv today and deck.gl-raster/deck-native later.
  • Avoid coupling the public SpatialCanvas API to Viv-specific concepts such as DetailView, Viv selections, or Viv extension classes where a backend-neutral expression is possible.
  • Consider proposing upstream Viv support for deck.gl-raster primitives if Viv remains useful for OME/NGFF metadata, channel semantics, and viewer ergonomics.
  • Add a raster backend spike after the headless demo exists:
    • render one image-like Zarr layer through Viv
    • render the same or analogous data through deck.gl-raster
    • compare coordinate transforms, channel controls, tile loading, shader extensibility, picking/labels feasibility, and dependency weight

Open image/labels questions:

  • Is Viv primarily a metadata/channel/viewer abstraction for us, or is it also the rendering primitive we want long term?
  • Can deck.gl-raster handle OME-NGFF / SpatialData Zarr layouts directly, or do we need an adapter layer for axes, multiscales, transformations, and channel metadata?
  • Are labels better treated as raster category data, image overlays, or feature layers with table-backed identities?
  • Can one backend handle both continuous images and integer segmentation labels, or do labels need a dedicated layer either way?
  • If raster backends diverge, what is the minimum common image-layer prop schema MDV and Vitessce can both drive?

OME-TIFF, JP2K, and raster codecs​

MDV currently supports OME-TIFF, including JP2K/JPEG2000-encoded image paths, but this repo currently focuses on SpatialData/OME-Zarr-style stores. We should keep OME-TIFF in the roadmap because it is already useful in MDV and may be an important bridge format for real spatial workflows.

Current state:

  • MDV can render OME-TIFF through its Viv/Avivator-derived path.
  • This repo does not yet expose an OME-TIFF-backed SpatialCanvas image source.
  • SpatialData workflows generally point toward Zarr-backed rasters, but users may already have pyramidal OME-TIFF assets, including tiled JPEG2000-compressed images.
  • OME-TIFF has strong bioimaging metadata support and can represent tiled multi-resolution images.
  • Zarr v3 supports codec pipelines, including standard codecs such as bytes, gzip, blosc, zstd, sharding, transpose, and CRC32C. NGFF notes that future image-specific codecs may be adopted as they emerge.

Potential directions:

  • Treat OME-TIFF as an image-source backend that can participate in the same headless viewer API as SpatialData Zarr images.
  • Decide whether OME-TIFF belongs in @spatialdata/core, a sibling image-source package, or only in @spatialdata/vis.
  • Preserve a path for JP2K/HTJ2K-style image compression benefits where browser/runtime support is practical.
  • Clarify how an OME-TIFF image should attach to a SpatialData project:
    • external image reference in project metadata
    • converted/derived OME-Zarr image
    • MDV-specific region metadata
    • future community-standard image reference pattern
  • Add an image-source abstraction that can represent:
    • OME-Zarr / SpatialData Zarr image
    • OME-TIFF / pyramidal TIFF image
    • deck.gl-raster Zarr/GeoZarr image
    • labels/segmentation raster
  • Keep codec-specific behavior below the public layer config where possible. Users should configure channel/selection/rendering intent, while loaders/backends handle storage codecs.

Community questions to clarify:

  • Should SpatialData workflows explicitly support external OME-TIFF references, or should OME-TIFF primarily be converted into Zarr/OME-Zarr/SpatialData stores?
  • What is the expected story for JP2K/HTJ2K-compressed microscopy imagery in web SpatialData viewers?
  • Which Zarr raster codecs are considered acceptable or desirable for OME-NGFF / SpatialData image arrays?
  • Are image-specific codecs likely to become standard enough that we should avoid investing too much in TIFF-specific paths?
  • How should labels/segmentation arrays be encoded in Zarr when storage efficiency and random tile access both matter?
  • Should MDV's existing OME-TIFF support become a compatibility adapter around the same headless image-source API, or remain an MDV-specific path until the community guidance is clearer?

Zarr beyond SpatialData​

Future MDV should probably support useful Zarr rasters that are not SpatialData stores. GIS, Earth observation, climate, and other geospatial workflows increasingly use Zarr-family formats with their own metadata conventions. The viewer architecture should be able to support these without pretending they are SpatialData.

Relevant standards and conventions:

  • OGC has endorsed Zarr as a community standard for cloud-friendly multidimensional arrays.
  • GeoZarr is developing modular conventions for geospatial Zarr, including CRS metadata, spatial transforms, and multiscale pyramids.
  • GeoZarr aims to support both Zarr v2 and v3 and bridge scientific/geospatial conventions such as CF, STAC, OGC Tile Matrix Sets, and affine geotransforms.
  • GDAL can expose some Zarr datasets as rasters, but plain Zarr does not itself define spatial reference semantics. Metadata conventions matter.

Potential MDV use cases:

  • GIS / GeoZarr raster layers alongside biological SpatialData layers.
  • Earth-observation or environmental rasters as contextual backgrounds.
  • Non-SpatialData Zarr arrays used as image-like overlays in projects.
  • Mixed workflows where MDV displays SpatialData-derived cell/shape/label layers over geospatial rasters.

Design implications:

  • Separate the concepts of "SpatialData object" and "renderable Zarr raster source".
  • Add an image-source interface that can describe axes, CRS/coordinate transform, multiscales/pyramids, chunking, channels/bands, and storage location without requiring SpatialData-specific metadata.
  • Keep SpatialData parsing in @spatialdata/core, but consider a sibling or lower-level package for generic Zarr raster descriptors.
  • Let @spatialdata/vis consume normalized raster descriptors regardless of whether they came from SpatialData, OME-Zarr, GeoZarr, OME-TIFF, or an MDV adapter.
  • Track GeoZarr conventions for CRS, spatial transforms, and multiscales before inventing our own GIS-Zarr metadata contract.
  • Avoid overloading biological channel semantics for GIS bands. A common layer API can expose bands/channels, but app UI should label them according to source context.
  • Consider MapView / geospatial projection support separately from the current Viv-style Cartesian detail view. GIS rasters may need deck.gl MapView, basemap alignment, and longitude/latitude coordinate handling.

Open questions:

  • Should the package names remain @spatialdata/* if we add generic Zarr/GIS support, or should generic raster source parsing live outside that namespace?
  • What minimum descriptor is needed to render a non-SpatialData Zarr raster in Cartesian MDV views?
  • What additional descriptor is needed to render the same source in geospatial MapView / MapLibre-aligned views?
  • Should MDV treat GIS Zarr as contextual imagery only, or should filtering/picking/measurement workflows apply there too?
  • How should coordinate-system linking work when a project mixes SpatialData coordinate systems and geospatial CRS?

Compatibility risks​

  • Viv/deck version skew: largely resolved — both codebases target Viv 0.21.0 / deck.gl 9.2.9; MDV's upgrade PR is almost ready to merge. Remaining risk is extension-specific regression, not mismatched package versions.
  • Arrow/Parquet churn: loaders.gl Parquet and deck.gl-community Arrow layers are promising but still moving targets. We should avoid locking public core APIs to one loader implementation too early.
  • Raster backend churn: deck.gl-raster may absorb image/Zarr responsibilities that currently sit in Viv or custom labels layers. Public props should leave room for a backend swap.
  • Image format/codecs uncertainty: OME-TIFF, JP2K-compressed TIFF, OME-Zarr, SpatialData Zarr, and future Zarr image codecs may all matter. Avoid encoding one storage format too deeply into viewer props.
  • Generic Zarr scope creep: MDV may need GeoZarr/GIS/non-SpatialData Zarr support, but that should be modeled as normalized raster sources rather than by weakening the SpatialData-specific APIs.
  • View state shape: MDV's Viv chart uses Viv DetailView view states with view ids. SpatialCanvas uses a smaller { target, zoom } shape. The adapter needs careful conversion so linked charts and saved configs do not drift.
  • Layer ids: MDV relies on getVivId(...) tokens for layer filtering. VivSpatialViewer has its own generated ids. MDV embedding may need a supplied viewId / layerIdSuffix.
  • UI ownership: current SpatialCanvas resets its local store when coordinate systems change. In controlled MDV mode this could wipe chart-driven layers if not separated.
  • Tooltip ownership: both libraries have tooltip systems. MDV should own tooltip portals for now.
  • Feature ids: shapes now carry stable featureId values in render data and layer picking; labels association and tests still need to catch up to the shapes path.
  • Performance: shape/label styling from MDV filters must avoid expensive observable lookups inside render accessors.
  • Data model mismatch: MDV's regions/image metadata is not the same as a full SpatialData object. The first adapter may need a compatibility layer while MDV projects transition.
  • CSS/layout: SpatialCanvas has hard-coded dark UI styles and minimum height. Headless mode should render with parent-owned layout and no border/sidebar styles.

First-pass acceptance criteria​

  • MDV can import @spatialdata/vis from a packed or prerelease package.
  • This repo has at least one headless demo whose state is driven from an external UI rather than from SpatialCanvas's built-in panels.
  • The same headless API can be described in terms of Vitessce coordination values without requiring MDV chart classes or MobX concepts.
  • A new MDV chart can render a SpatialData-backed image layer using the same region/view size as the current Viv chart.
  • The same chart can compose at least one MDV custom deck.gl layer above the image.
  • SpatialCanvas UI controls do not appear inside MDV.
  • MDV controls view state, and existing chart linking still works for pan/zoom.
  • A labels or shapes layer renders above the image with constant styling.
  • Hover/pick returns enough information to identify a label id or shape feature.
  • The old VivMdvRegionReact path remains available during rollout.

Existing migration checklist​

  • Replace local src/react/components/avivatorish/ with @spatialdata/avivatorish where doing so does not destabilize MDV.
  • Replace viv_loader_cache.ts with createLoader / loadOmeZarrMultiscalesData from this repo. Optional shared caching can be reintroduced later; shared loader code should remain MobX-free.
  • SpatialLayer naming: MDV today uses @/webgl/SpatialLayer; this repo exports a SpatialLayer CompositeLayer from @spatialdata/layers. Converge names or use a temporary alias such as SpatialDataCompositeLayer.
  • VivScatterComponent / scatter_state: keep view-id conventions (getVivId) and metadata hooks compatible. Long term, scatter props should accept table-backed descriptions aligned with @spatialdata/core TableElement.
  • contour_state: Phase 1: extract pure functions (columnar data + view params -> contour / deck extension props) into @spatialdata/layers; leave MobX + MDV GUI in MDV. Phase 2: optional MobX-free hooks in vis.

References (upstream)​