OME-Zarr Pulse Check: Spec, Tooling, and Community
Summary
As part of our work on OME-Zarr, we have been meeting with members of the community: developers, analysts, and users in both academia and industry. Everyone we spoke to expressed excitement about the prospect of a well-supported push to 1.0. We also heard a strong desire towards stability and long-term support, particularly from prospective users and vendors interested in the format. Finally, we found room to improve the onboarding experience for new folks to get on board, and converge on standardized validation utilities to improve the development experience. As we work together towards OME-Zarr v1.0, one thing is clear - the welcoming spirit of the community and openness and transparency with which the format is being built are fundamental pillars of the ecosystem, and we must all celebrate and protect them!
The following findings and discussion are our current impressions, based on both our prior experience and our work with the community over the past few weeks. But engaging with the community is an active and ongoing effort. If you have any more feedback for us, you can get in touch at omezarr@image.coop.
Full details
As foreshadowed in our April blogpost announcing our new partnership, IC members have been busy scuttling about (physically, and virtually), chatting to folks to learn about their experience with OME-Zarr, what they need and want from the format, and any concerns they may have, or reasons for not adopting it (if they hadn’t).
We’ve been gathering feedback by participating in the hallway track at recent community events, chatting to folks in small groups, looking at aggregate image.sc info and GitHub repositories/documentation websites, and hosting mini-surveys at the May OME-NGFF Community Calls. If we didn’t get a chance to chat to you about OME-Zarr, we’d still love to hear from you! Reach out to omezarr@image.coop to get in touch.
For tool developers and spec contributors, both validation and conformance testing were key concerns. As we approach the v1.0 release, the specification and implementations will see rapid updates. Testing a new implementation of an updated specification feature requires data compliant with the updated spec, but this data can only be written once tools are updated. How do we ensure there is compliant data available for new spec versions, so that tools may test against it? Who validates the validators? In an ideal world, new versions of the specification would be published in tandem with exemplar datasets (e.g. building on this work from John Bogovic, gathering exemplar datasets for the different coordinate transforms in RFC-5) and a conformance testing utility (Chris Barnes has made a start here on tool for RFC-5) that can be executed across the different programming languages common in the ecosystem (and illustrated below) e.g. Python, Typescript/Javascript and Java.
Newer users and vendors brought up concerns about the complexity and usability of the format. Some folks mentioned that getting started was difficult without computational experience, while others, including vendors, pointed out the many-file issue as one example of the steeper learning curve when it comes to working with OME-Zarr data. Check out the word-cloud below for other current drawbacks of the format raised by folks in the community call (Figure 3).
Participants in the community call surveys had mixed levels of confidence in finding tools and resources to help new users get started (see Figure 4 below). With the ecosystem of OME-Zarr tools continuing to grow, and existing work already in-flight to improve some aspects of these concerns (e.g. RFC-9 for zipped OME-Zarrs), we think there is an opportunity to improve the onboarding experience for new users by building on educational materials that describe common OME-Zarr “happy-paths” e.g. updating this resource introducing OME-Zarr for big bioimaging data, and working to standardize how tools can advertise their compatibility with various parts of the specification.
You may already know this, but the OME-Zarr community is very active! We analyzed OME-Zarr-related activity on image.sc, the bioimage analysis forum we all know and love, and found that the frequency of OME-Zarr posts has been increasing (Figure 5). Further, we see that the community is quite responsive. 50% of posts receive a response within 2.3 hours and 90% receive a response within 1.7 days. While the on-boarding materials can be further developed, the image.sc forum is a great place to get help in the meantime, for new and experienced users alike.
All members of the community, whether contributing to the spec and tooling, actively using OME-Zarr, or simply keeping an eye on it for now, expressed that spec maturity and stability were the two most important considerations when thinking about widespread adoption of OME-Zarr as a format. Vendors described the need for a stable, robust specification to justify spending time and money on implementing support for OME-Zarr in proprietary software (something their users have said they want!), while prospective users want assurance that data written with a given OME-Zarr version is metadata-complete, and will remain usable with tooling for the foreseeable future without major breaking changes. In order to achieve this, the OME-Zarr community needs a plan for both sustainable funding and maintenance of the core parts of the OME-Zarr ecosystem.
While there’s still work to be done to make OME-Zarr accessible and robust for all who want to use it, the format’s potential for scalable, cloud-native storage and its vision for an interoperable open standard were repeatedly mentioned as key advantages of OME-Zarr as a format - “deposit somewhere once, access/index it from everywhere”.
It was evident throughout our various conversations that whether someone was just dipping their toes into what OME-Zarr has to offer, or they were seasoned contributors to the ecosystem, the momentum and trajectory of OME-Zarr, and its welcoming community spirit were compelling reasons to stay engaged and continue working towards our collective shared vision of an open, scalable and interoperable standard for the world’s imaging data.
Image Coop will continue working with the community to achieve this collective vision and address some of the concerns raised here by:
- helping to generate conformance testing datasets and utilities for the current spec and updates leading up to v1.0
- improving onboarding and getting-started materials to provide a smoother experience for new users
- ultimately, joining forces with current contributors to deliver a stable OME-Zarr 1.0 version that meets the community’s needs for years to come.