State of the Upgrade

General
1

Open Referral: State of the HSDS Upgrade

Open Referral Community,

Through 2022, the Open Referral Initiative engaged in a community-led process of upgrading the Human Service Data Specifications in order to properly support the increased use and new technologies emerging throughout our ecosystem of human service informatics.

Last summer, we convened a workgroup of active adopters and community members with particular expertise, and since then we have been regularly meeting to set an agenda, deliberate, and arrive at recommendations for improvements to HSDS and core support for it. This workgroup is now in its final stages before releasing a formal proposal to the community.

There are five interrelated areas being considered:

  • Updating the data model of HSDS (i.e. the fields, tables and relationships)
  • Updating the Human Service Data API (HSDA) specification (i.e. the query format and responses)
  • Potentially changing the basic format of our schema (i.e. the default machine-readable language for our standards, in which resource data is to be published for use with other tools)
  • Developing support for communities that want to adapt the standards – through extensions and/or ‘profiles’ that tailor the standard while maintaining compatibility across the ecosystem.
  • Developing tools that can support adoption of the standards (e.g. validators, transformers, mapping tools and preview tools).

These areas will form one proposal with five parts, adding up to a prospective version 3.0 of HSDS.

Feedback on the proposed version 3.0 is welcome, via:

We expect version 3.0 to be the last major upgrade for a while, so if you have any questions or concerns about its suitability for your current or future potential use of HSDS, please raise them now.

Data Model Upgrade

The fundamental modelling of HSDS - the tables, field names and data types - have largely remained unchanged since its inception. However, we’ve heard from implementers that there are some areas where the model could be improved to make the standard easier to use.

Open Referral’s Technical Stewards (Open Data Services Cooperative) are currently updating the documentation site for comment and feedback, which now features a staged version of a still-in-development proposal for HSDS version 3.0 on the docs site.

An overview of changes made can be found here. Moving forward, these changes, and wider changes to the standard will be added to the changelog page on the documentation site.

The most recent conversations, decisions and proposals in relation to the data model can be found in the technical section of the Open Referral Forum.

Please join the discussion in the forum and weigh in on any points that are of interest to you!

Schema Language

HSDS currently uses the JSON Datapackage specification (incorporating the JSON Table Schema) for describing how data should be stored within a set of CSV tables. Although this is a useful format, it does have some limitations –- most notably that API responses are almost universally JSON, not JSON Tables.

We are therefore proposing switching our primary schema language to JSON Schema: this allows us to carry out more nuanced validation (such as requiring one of two fields to be present, but not both) and makes API standardization easier.

The JSON Tables representation is an important artefact for many HSDS users, and so will be maintained: we automatically generate a JSON Table Schema from a JSON Schema, so we expect that the shift will not cause any issues for any implementations that expect the datapackage.

API Specification Upgrade

The workgroup has been considering how Open Referral’s API specification should work, taking into account lessons learned from Open Referral UK and elsewhere. We have heard that APIs are becoming increasingly important to the Open Referral community, and that there are opportunities for collaboration around open-source tooling particularly around API development.

The current API specification (HSDA) is based on a previous version of HSDS (v1) and was never updated for version 2. Meanwhile, Open Referral UK has developed a simpler set of API protocols which is adopted across the UK’s ecosystem, but their work hasn’t yet been re-integrated into the global specification, which limits interoperability among the tools in use across each part of our network.

The proposal is to make HSDS an API-first standard, with endpoints designed around the core tables (service, location, organization, service_at_location). The datapackage format will still be supported for bulk data interchange use cases, and will be updated and maintained in sync with the API specification.

A draft API specification has been created by Open Data Services and shared for comment and feedback via this post. Once the API specification has been reviewed and agreed by the workgroup, documentation will be updated accordingly.

Extensions & Profiles

When Open Referral UK published their work, they shared details of how their schema differed from HSDS: they had added fields that were UK-specific, re-worked some tables to offer a neater solution, and decided explicitly not to use aspects of the standard that weren’t relevant in their country. This was helpful for the Open Referral community overall, as it helped us to understand differences across countries, and also identified new issues that may be relevant universally.

We’ve agreed that we need a common method for publishing data models that are derived from HSDS, in ways that adapt the core schema to specific purposes, while preserving compatibility.

Toward that end, we’ve discussed three complementary concepts which will inform our ultimate conclusion about how ‘extensions’ and ‘profiles’ should work:

  • Constraints: identifying a subset of the fields that are relevant to a particular case
  • Expansion: identifying additional fields that are relevant in a particular case and can be represented in the same way across applications
  • Description: identifying some properties of the data (such as fields being present or absent, information being presented in a certain way) that mean that it is appropriate for use in a particular case

Other data standards have different approaches to this, including Open Contracting’s “extensions” mechanism (which starts with a minimal set of fields and specifies additions that are useful in particular cases), and FHIR’s “profiles” mechanism (which starts with a wide range of potentially useful fields and removes those not relevant to a particular case). Any approach for Open Referral will be informed by others’ experience.

The workgroup will continue to discuss approaches to extensions/profiles, and once broad consensus has been reached, Open Data Services will prepare documentation for the community to review ahead of adoption.

Thanks

Many thanks to our workgroup members – Devin Balkind of Sarapis, Mike Thacker of Porism, Sarah Pottelberg of AIRS, Skyler Young of SiteSavvy, Paul Brown of PlaceCube, Ian Singleton of Digital Gaps, and Shelby Switzer of the Beeck Center – for their support in this process!

2 Likes