ZipFileStore because I had a memory of it being problematic for me in earlier prototypes - and marked 'experimental' in the zarrita documentation.
At some point I thought that I was satisfied it was in fact working ok - but now I notice that there is in fact a related issues showing up, even without ZipFileStore, but perhaps more generally with '@zarrita/storage'?
error during build:
│ ../../node_modules/.pnpm/@zarrita+storage@0.1.3/node_modules/@zarrita/storage/dist/src/fs.js (1:9): "Buffer" is not exported by "__vite-browser-external", imported by "..…
│ file: /Users/ptodd/code/www/SpatialData.ts/node_modules/.pnpm/@zarrita+storage@0.1.3/node_modules/@zarrita/storage/dist/src/fs.js:1:9
│ 1: import { Buffer } from "node:buffer";
│ ^
│ 2: import * as fs from "node:fs";
│ 3: import * as path from "node:path";
....
└─ Failed in 940ms at /Users/ptodd/code/www/SpatialData.ts/packages/core
I had previously added a vite-plugin-node-polyfills to MDV which meant that it was working in the Vite dev-server... but then I realised that the actual build script was broken, and in the brief time I spent looking at it, I couldn't seem to make both dev and build work - so I attributed it to the experimental nature of ZipFileStore and moved on (we've yet to make use of zarrita in mainline MDV, but at least everything seemed ok without that being used).
I don't entirely know what I've done to cause ...@zarrita/storage/dist/src/fs.js to be included in the build - fs.js sounds like something that shouldn't be invoked in front-end code (unless I suppose it's using emscripten filesystem or something, which come to think of it could make sense). Anyway, it seems like a zarrita bug, but I'm still not entirely sure - I'll try to make a minimal repro at some point.
Now - clearly the code used in Vitessce is building ok there, and glancing at their vite config I can't see anything obviously in place that is supposed to mitigate this particular issue. I'm not sure what is so different about how our builds are configured that would cause it to be different.
Looking at the places where '@zarrita/storage' is used, I think we can refactor such that the DataSourceParams needed by things in the file currently named VZarrDataSource.ts always use the store that other SpatialData.js code is already creating. I would like to be able to support ZipFileStore, and in future we'll need more careful consideration around writeable-stores etc etc, but for now I think we can remove some of the code previously copied from vitessce and it at least builds ok.
SpatialElement would return something somewhat equivalent to the corresponding library in Python - to characterise the dependencies of that package as 'minimalist', even without the batteries-included [extras], seems disingenious... Perhaps the philosophy is not so much that these will be minimal, but they should avoid needless bloat and especially any likelihood of tricky conflicts to manage (<cough>deck.gl</cough>).
Here is an approximate summary of the ways in which these are typed in Python:
class Elements(UserDict[str, T])
...
class Images(Elements[DataArray | DataTree])
class Labels(Elements[DataArray | DataTree])
class Tables(Elements[AnnData])
class Shapes(Elements[GeoDataFrame])
class Points(Elements[DaskDataFrame])
DataArray | DataTree)It is probably reasonably fair to assume that - at least in the immediate term - the priority for these raster elements is that they will primarily be used for visualisation. For viewing images in viv/MDV, it is already possible just by passing the correct path to viv. More work will be required in order to be able to correctly use transformations, with multiple layers etc... but that doesn't necessarily entail the need in the immediate term to have any very particularly special method for loading these parts of the stores in @spatialdata/core.
It may be that we are likely to use loaders.gl for integration into other parts of the ecosystem - perhaps contributing where relevant if there exist gaps.
It would be nice to have well-formed and typed ergonomic definitions for this - we may use the OME-NGFF schema with zod to validate and provide de-facto accurate types for this data.
We should at least have simple canonical ways of querying various aspects of the metadata, shape etc sooner rather than later, along with working examples of using this to view images.
AnnData)We have added AnnData.js for this purpose. Some parts of it don't feel entirely ergonomic - but we can always make suggestions/contributions if appropriate (and that feeling may be at least partly of my not having done enough to familiarise myself with it).
The Vitessce SpatialDataTableSource is also relevant to look at particularly given that it is the parent for their implementations of Shapes and Points are based on it.
because when a table annotates points and shapes, it can be helpful to have all of the required functionality to load the table data and the parquet data.
GeoDataFrame)This is an instance where it probably makes most sense to use a similar implementation to Vitessce's SpatialDataShapesSource, with Arrow tables.
If we do that - that means that the code we need to borrow/port from vitessce leaks significantly. As of this writing, I'm in the process of copying SpatialDataShapesSource from there, and following through the imports to port whatever else is necessary... for now, I'm naming files with a V at the start, like VShapesSource corresponds to SpatialDataShapesSource - the SpatialData part seems redundant here.
It is likely that we may end up with something similar to essentially having a version of their file-types/(spatial-)zarr package(s).
If we do continue to make use of this code, we should also make sure that equivalent tests are also ported etc.
DaskDataFrame)There is some discussion about potential different ways in which point transcript data may be represented - we think that we may want to propose new evolutions of the spec in order to facilitate the kind of interactive web-based visualisation we do in Vitessce/MDV.
In terms of a direct JS-analog to Dask in Python - this is something we need to think about more. Again, the reference-point for initial implementation is Vitessce's SpatialDataPointsSource
It is a collaborative effort by the Taylor CCB Group at the Center for Human Genetics, Oxford University and the Gihlenborg Group at the Harvard Medical School - Department of Biomedical Informatics.
The approach taken to the design is to follow the structure of the original Python library, adapting it to TypeScript with the intention that it should be familiar to users of the original library and aligned with best practices in frontend development. It should be usable in a server-side environment as well, and some consideration will be given to this, but it is not the primary focus of the project and at this stage isn't something that is anticipated to be a priority.
Once it has reached a state in which a useful subset of the functionality has been implemented, it will be released in an 'alpha' state as a package on npm.
The aim is to do this soon, such that MDV can have a new type of DataLoader based on this. Actually, without getting too derailed into MDV-specifics, it may be that the first use of the library will be for accessing and rendering things like Points transcripts, in a way that sits-beside the main data at least for the time-being.
There is a lot of overlap between MDV and vitessce. It is anticipated that substantial portions of this codebase will be based on existing implementations in vitessce. We had discussed approaches to refactoring their code such that the data-loading functionality could be exposed as something to be exposed as a separate package - I wasn't quite clear how we'd adapt each-others idioms about what kind of common interface we'd use that would allow our MDV idioms to map to theirs.
The intention is that as this becomes mature, both projects will make use of it, and it will be a useful part of the ecosystem more generally.
Hopefully, it will not require a huge amount of effort to be able to get at least a useful subset of the core data-loading functionality working well.
Some functionality that will be a lower priority are things like utilities for converting between raster labels and vector shapes. As of this writing, I don't have comprehensive knowledge of the original spec; writing this is a helpful exercise in better understanding it, but it could be that there are aspects of it I haven't
The very first version will likely not include any methods for altering the data - and it may require some careful consideration of how persistence should end up working, particularly from a frontend perspective. This is a somewhat high priority: from an MDV perspective, we likely want to use zarr stores rather than HDF5 for project data in the near future.
The guiding principle is to make it such that Python code would translate in a direct way to JS/TS - so where in Python you have something like this:
import spatialdata as sd
sdata = sd.read_zarr("path/to/zarr")
You would write something like this:
import * as sd from 'spatialdata';
const sdata = await sd.readZarr("path/to/zarr");
sdata should be an object with a similar interface to the Python version (i.e. class SpatialData in both cases).
In the context of JS accessing remote data, this inevitably means a lot of async operations, and potentially complex data structures, that should use JavaScript equivalents of the packages used by spatialdata in Python.
I'm not so sure whether we want to mirror the ability to do things like
sdata["some_arbitrary_name"] = sdata["table"]
I suppose it would be possible to use a Proxy for this, and it may not be a terrible idea... but at least in a first-pass, the data will be read-only, and access will likely be via somewhat more explicit/verbose interfaces. Certainly it would be a mistake to try to introduce too much clever meta-programming before at least having a working MVP.
Looking at the Python repr returned by a SpatialData object (names changed to protect the innocent):
SpatialData object, with associated Zarr store: /path/to/spatialdata.zarr
├── Images
│ └── 'Run3-14-7-23_Output_region_1_z3': DataTree[cyx] (5, 53066, 55984), (5, 26533, 27992), (5, 13266, 13996), (5, 6633, 6998), (5, 3316, 3499)
├── Points
│ └── 'Run3-14-7-23_Output_region_1_transcripts': DataFrame with shape: (<Delayed>, 9) (2D points)
├── Shapes
│ └── 'Run3-14-7-23_Output_region_1_polygons': GeoDataFrame shape: (23961, 9) (2D shapes)
└── Tables
└── 'table': AnnData (23961, 466)
with coordinate systems:
▸ 'global', with elements:
Run3-14-7-23_Output_region_1_z3 (Images), Run3-14-7-23_Output_region_1_transcripts (Points), Run3-14-7-23_Output_region_1_polygons (Shapes)
There are elements represented by DataTree, dask.DataFrame, GeoDataFrame, and AnnData... this is not an exhaustive list.
In most cases, a combination of zarrita, AnnData.js, and some of the methods implemented in vitessce should be able to represent these various models.
The OME-NGFF specification should be able to drive validation of these, perhaps via something like zod-from-json-schema.
The core module should avoid any dependencies on the vis.gl ecosystem that both MDV and vitessce/viv use.
Re-usable visual elements are out of the scope of what we've discussed for immediate plans. It may make sense to have some @deck.gl/community-layers type package that could have some common utilities for rendering, annotating etc...
I believe it will be particularly useful to have at least some sample visualizations that can be used for testing and demonstration purposes within this documentation. That means that while we are editing the code, we can have interactively updated feedback on the work as it progresses.
Perhaps rather than letting the /docs package in the monorepo get bloated, it'd be worth having some packages/layers, packages/hooks sooner rather than later...
Part of the shared interests we have as collaborators are around potential different ways of representing points in the SpatialData spec itself, and it may be that we have a package relating to experimental prototypes around such features.
]]>Short answer to the above question is probably "no" in the long term, and "I don't see why not, as long as we're clear that we expect stores to be treated as immutable for now". I'd welcome any feedback on my stance there.
In the meanwhile, some thoughts on breaking out of my current procrastination loop.
Development has felt somewhat stalled as I've been in a protracted procrastination loop about how to process and represent metadata, which is something of a pre-requisite to parsing transformations, which in turn is a pretty essential aspect of getting the library to actually be capable of doing anything useful or interesting.
Perfectionism and other distractions with MDV demos etc have somewhat disrupted my focus on this. At the same time, I've been doing a bit more work on avivatorish state in the demo ImageView component and I'm not really convinced that the way I'm modelling state there is right - and also haven't thought very much about how this'll be exposed as a public API. That's not what I want to address here, but I suppose it has been another detriment to focus.
I wasn't quite sure why zarrita had chosen to make the raw consolidated metadata so private... I've been allowing this to block me more than I should. Somehow, despite me having made a stated decision weeks ago to separately parse zmetadata and get on with life, it still hasn't quite been sitting right that it wasn't exposed so I raised an issue. I think my brain has been somewhat more focused on the question of "we need a coherent representation of the hierarchy and associated metadata" than "can we display an image with cell segmentation shapes overlaid, even if it does involve a few more decode(await store.get(...)) than I'd like". I am very wary of releasing a version of this library that bakes in technical debt in the form of unergomic patterns for users.
My default stance is that I still have a relatively limited understanding of zarr and that others like Trevor Manz probably know better. There clearly is a rationale for the design of zarrita, and hopefully it will evolve to more comprehensively and ergonomically expose relevant things. That said, I find myself wanting to expand some of the things in zarrUtils.ts into a more generally re-usable "zarrextra" package built on top of it.
It should be stated that as of this writing, none of the data I have been using has been generated with the latest version of spatialdata (0.6.x) and corresponding zarr v3 (with the associated changes to consolidated metadata, which in v3 are still not entirely stable as far as I understand). So the requirements of supporting each of those has been somewhat on my mind, and hopefully will not be too difficult, but for the sake of breaking procrastination cycles it should be clearly stated that the initial implementation will work against the older specification. On the MDV side, this difference first came to my attention when a collaborator tried to use the mdvtools.spatial.conversion script with outputs from 0.6.0, and they failed to open with 0.5.0 that is currently used there.
We should aim to support both versions soon, and related to that, establish a set of test fixtures associated with them. The defaultUrl = 'https://storage.googleapis.com/vitessce-demo-data/spatialdata-august-2025/visium_hd_3.0.0.spatialdata.zarr' used by the demo app is actually not able to be opened with the version of spatialdata we have in the MDV environment currently.
>>> import spatialdata as sd
>>> sdata = sd.read_zarr('https://storage.googleapis.com/vitessce-demo-data/spatialdata-august-2025/visium_hd_3.0.0.spatialdata.zarr')
...
pyarrow.lib.ArrowInvalid: Unrecognized filesystem type in URI: https:/storage.googleapis.com/vitessce-demo-data/spatialdata-august-2025/visium_hd_3.0.0.spatialdata.zarr/shapes/Visium_HD_Mouse_Small_Intestine_square_002um/shapes.parquet
Given that this is my go-to default datasource when working on this, that's not really a very reassuring thing to have lingering in the back of my mind.
Also as of this writing, there is no testing in place here to speak of, and limited coverage in MDV. While I don't necessarily want to go full TDD, the intention is definitely to have good coverage - and for that to stand as a useful reference as to the implementation status and use of various features. We might use Python in the spatialdata.js repo for the purposes of using spatialdata mock-data features for generating test fixtures. Given the known vagueries of output from different versions, there may even be a case to be made for swallowing the added test-environment-complexity of having tests based around different versions.
I do intend to update the version used in MDV at some point soon - particularly as this will clear up some issues we currently have with older python dependencies being brought in from there. It will make sense from my point of view to align spatialdata.js with MDV - and this is quite likely to mean that an initial 0.0.1 release of this will still be tied to older outputs, followed shortly after by an update aimed at supporting both. Alternatively, it may be that we prioritise support for the newer version to allow us to update the MDV dependency sooner rather than later.
Wrap up how we parse consolidated metadata in zarrUtils.ts and try to follow Mark's recommendations:
Mark Keller: We should be able to use https://github.com/hms-dbmi/zod-ome-ngff, either directly, or by exporting sub-schemas that can be reused via composition of zod schemas https://github.com/hms-dbmi/zod-ome-ngff/issues/23 Mark Keller: I recently updated the schemas there to support OME-NGFF v0.5 Mark Keller: A long time ago I started to implement typescript types for the transforms here. They are probably out of date now that the transformations proposals have evolved but maybe worth using as a starting point.
I had a few sketchy things in my working copy that are probably mostly redundant WRT that but may be worth stashing somewhere... the zod-ome-ngff should be a good approach on the face of it.
The initial version of this library and its use in MDV will be explicitly limited to stores that are considered effectively immutable - but this is actually not something that is likely to align with our goals for MDV in the mid-term. From my current understanding, consolidated metadata should support this use-case well.
Mutable stores of course open a whole new set of rabbit-holes... on the face of it, I think we'll need to seriously investigate icechunk if and when the time comes for this. Ideally, we'd be able to support workflows where a user makes a new project (with an associated empty spatialdata store), adds some image data, invokes a cell-segmentation job which after some time leads to more elements in that coordinate-system, annotates some of the data thus produced, etc etc. This is of course not to be taken at all lightly when it may involve multiple simultaneous users working with the same underlying store.
.js files with // @ts-check and quite extensive type-annotations in jsdoc form.
While moving the code across, I've been been changing this so that it is more regular TypeScript - which has mostly been a fairly straightforward process, and I think the result is somewhat cleaner... but some of these changes are also intermingled with what should be very minor changes to functionality...
return null to just return where another function was assinging the returned value to something annotated as 'optional' for example. I also noticed the mechanism for getParquetModule was being used in a way that would cause it to be re-fetched for instantiation of SpatialDataTableSource - I think the pattern used may have been following the way I'd implemented importing openjpeg-wasm in this draft viv PR, which I now notice I haven't updated with the equivalent of how I changed this in MDV.I don't think I've done anything that should intefere with functionality - but I've been through this process before adopting any actual tests or uses of the underlying code in any active code-path. I've circled the sun enough times to be very wary of assuming that everything will be fine without proper verification... and whether the changes are improvements or not, the fact that they exist is worth flagging.
It means that there is now the potential for the existence of two divergent implementations to exist - I believe the intention is that this library should be able to essentially stand-in for the existing implementations of SpatialData in Vitessce - but for that to work from their perspective would probably mean reviewing the extent to which code is ported, how it should be structured etc... The way I have so far structured things here is somewhat loose in certain ways, and I haven't (so far) comprehensively included everything that will ultimately be necessary.
//@ts-expect-error to //@ts-ignore.There are a few places in the code where I've left //@ts-expect-error annotations, with comments. I prefer this to ts-ignore as it means that in such cases where later changes to other parts of the code make them redundant, they then appear as errors, showing that the annotation can be removed. It's less easy to know whether a ts-ignore is still relevant. There is only one ts-ignore in Vitessce so in a sense a bit of a moot point there - as of this writing there are a few here, and I would like to aim to eradicate them (there are quite a few in MDV sadly, and I don't know when I'll manage to get around to those).
any history at some point.I would definitely prefer that we can type things well enough to avoid this.
There are a few places where better use of generics on things like Chunk<DataType> etc would help... here is a concrete example using Zarrita's is to type-guard in a loadNumeric method:
async loadNumeric(path: string) {
const { storeRoot } = this;
const arr = await zarrOpen(storeRoot.resolve(path), { kind: 'array' });
if (!arr.is("number")) {
throw new Error(`Expected a numeric array at ${path}, but got ${arr.dtype}`);
}
// now we have a type-guarded array, the return type will be correctly inferred
return await zarrGet(arr);
// return zarrOpen(storeRoot.resolve(path), { kind: 'array' })
// .then(arr => zarrGet(arr));
}
This will change runtime behaviour in cases where the arr.is("number") check doesn't pass. It seems like the occassions on that would be different should only be when the underlying assumptions in the code are false, and that it will result in better error-handling, but it is also possible that there is something wrong in my understanding here.
This type design in Zarrita seems like a really good thing, encouraging the use of types in a way that properly ensures actual runtime validation is reflected in what the LSP is able to understand about the working of the code (even if the above code was written in JS, a modern editor would likely have enough information to accurately reflect the types, aside from the trivial path: string).
If we do end up extensively using Vitessce code in SpatialData.js, hopefully it is useful to have another pair of eyes on it - and better levereging this feature is definitely something that I advocate.
class SpatialDataShapesSource extends SpatialDataTableSource
/**
* This class is a parent class for tables, shapes, and points.
* This is because these share functionality, for example:
* - both shapes (the latest version) and points use parquet-based formats.
* - both shapes (a previous version) and tables use zarr-based formats.
* - logic for manipulating spatialdata element paths is shared across all elements.
*/
export default class SpatialDataTableSource extends AnnDataSource
This means that the interface for Shapes inherits everything from AnnData, which is not an accurate way of modelling the actual spec - there are a lot of things about AnnData that are not inherent to GeoDataFrame, which in python is what is used for class Shapes(Elements[GeoDataFrame]).
I find this distracting as my habbit when learning and reasoning about code tends to rely heavily on exploring what the editor thinks it 'knows' about what interface a given symbol...
I posit that the sharing of functionality between tables/points/shapes mentioned here could probably be accomplished through composition than inheritence; I haven't formulated a precise design for this just yet.
So... as I write this, on top of my local changes to rev 8ff418e in this repository, I am investigating how the code looks if we start trying to use this SpatialDataShapesSource for Elements<'shapes'>
export type Elements<T extends ElementName> = Record<string, Promise<
T extends 'tables' ? Table
: T extends 'shapes' ? SpatialDataShapesSource : SpatialElement>>;
...
class SpatialData {
...
this.shapes[key] = (async () => {
// we already have a store in scope at this point and want to open a path within it, but I'm not sure we're allowed?
const store = await tryConsolidated(new zarr.FetchStore(`${this.url}/shapes/${key}`));
return new SpatialDataShapesSource({ store, fileType: '.zarr' });
})(); // side-note, would rather these not be immediately-invoked
}
When I start trying to write some hooks that access SpatialData.shapes, while there are undoubtedly ways of using it to get the actual data we're interested in, it is somewhat obscured by other inherited things. In order to use the loadPolygonShapes() method I need to know a path to it...
So, working through my thought process... perhaps I should indeed be using the root store from the outer scope, and providing a more focused API for interfacing with the actual internal parquet data, with any relevant path captured within an appropriate scope - but as of this writing I seem to have a broken build, HMR not working as expected, and I think I need to get some fresh air/excercise/rest before a bit of other refactoring.