Thanks everyone for the engagement on this! This is very humbling. Responses to individual points made below.
From my experience with the Open Referral UK implementation, and the wider International communirty, I believe that overly prescriptive schemas can actually hold back adoption. Forcing providers to map internal data models to a rigid external schema often feels like trying to fit a square peg in a round hole.
I understand what you’re saying and agree to an extent; but I do think having stricter schemas for data transmission is the way forward. Ultimately, the hard work of addressing data quality and interoperability has to be done by either: the data owner, an intermediary, or a data user.
I think that slightly stricter schemas are worthwhile in the long run to shift the burden away from end users towards data owners. By strict I mean that there are validation rules in place for a minimally-useful object.
Historically, it’s been the data users and a few well-positioned intermediaries who do the hard work. The stricter the validation rules of a Standard, the more this burden shifts to the Data Owner (or perhaps an intermediary who is converting it) and away from the data users. In our findings across a number of Standards, having data owners take on the work of adopting the Standard results in more resilient ecosystems of data publishers, tooling, and actual data use.
In this way, methodologies for using the data can be built around the guarantees that a schema provides, and tools can be built to support particular use cases based around the minimum guarantees that a schema provides.
This isn’t to say that there won’t be square pegs in round holes; there will never be a perfect model! I see a Standard as a Boundary Object which is fundamentally sociotechnical; it allows people to build tools around a structure and the promises a schema makes and in that process people can ask questions about models and assumptions which allow people to engage with things critically.
- Human-Data Interaction (source for explanation of Boundary Object . Academic paper alert, sorry, thought I’d justify my use of the term by citing it!)
I feel that as long as a provider ships their feed alongside an explicit API specification—be it OpenAPI, Swagger, or YAML—consumers have everything they need to validate and consume the data reliably. Self-describing data endpoints give us both validation safety and provider flexibility.
I think I understand your point here and to an extent I believe this is true; if done properly everything is all declarative and self-describing. However I’m not sure how far that takes us wrt to data use without a shared standard for the data model. Perhaps LLMs can help with this in terms of mapping between data sets, but if I was building a tool/script for a particular use-case of HSDS I’d need a guarantee of particular fields to be there. Currently, the schema is that guarantee. If I consume a bunch of APIs to ingest some data for my analysis, it doesn’t do me any good if they’re all presenting different models of a service even if they’re all self-documenting. I’d have to go in and look at everything and come up with a bunch of conditional logic to map things across depending on which model I’m looking at. Contrast that to the promise of “these datasets from these APIs all validate against the HSDS Schema”; I just need to understand the HSDS schema and my tool will work with any conformant API.
I fully understand the long-term desire to align data exposure across the entire sector. However, that level of standardization shouldn’t happen overnight or via a forced migration. Real alignment happens naturally over iterative cycles as business cases evolve for each individual adopter—allowing organizations to adopt richer standards at a pace that actually makes sense for their resources.
Definitely agree here. So with the context of everything I’ve noted, I do think having a (any) Standard with a sensible set of lightweight minimal requirements is the correct balance. Being able to achieve minimal compliance should be straightforward for everyone, and then the sell becomes “Hey, you benefit from the shared tools/methodologies for data use. You can get more out of it the more of the spec that you meet”.
With HSDS Lite as an example, I ignored a large amount of the depth-and-breadth of HSDS (which is useful, btw) and left what I felt is the core. The trade-off is that I made the schema stricter to pass validation; you need to have more fields to be minimally compliant, but if you don’t have those fields in your data, then maybe it wasn’t particularly useful anyway.
The identifier issue is an interesting one for our comunity to wrangle, and I suspect there is no universal answer. The use of UUIDs is great if a service is only described in one place. We’re finding that more than one registrar may describe the same service. If that’s the case, the UUID is less helpful since the selected ids will (by design) be different. The UK government preferred approach is to use persistent resolvable http URLs as identifiers.
Yes the id thing is interesting. For me there are a few things that identifiers need to help with in a dataset like this:
- Identifying the object in the local dataset
- Identifying the object in a mixed/global dataset
- Identifying the thing that the object claims to be describing (the actual service, rather than the record of it)
My belief is that UUIDs are only useful for the first two, and actually cause problems for the latter; for better or worse, organizations and services are currently not legally/formally identified by UUIDs (or at least, not in any context I’m aware of).
I heartily agree that having persistent resolvable URLs as identifiers is the way forward. HSDS Lite has a url property on its schema to account for this.
The interim was to loosen the requirements for the formatting of the id field to allow any string. This could contain an UUID (or a URL… a man can dream) But it didn’t make sense to make it a UUID if I didn’t believe that this was a reliable way of identifying a service.
Making record/entity identifiers globally unique is undeniably useful. I think this should be seen as a data quality concern in the short-term, until there is some consistent way of minting identifiers for services.
The Open Contracting model relies on being able to identify a single contacting process across a series of different releases, which can then be combined into a final “record” of the contract. Therefore publishers must “mint” an ocid to use for a contracting process. This tends to take the form of ocds-[publisher-prefix-from-a-registry-of-publshers]-[contracting-process-identifier-from-local-systems] (source). The local identifier is taken from things such as public journals, procurement systems, etc. This enables it to be semantically useful (it identifiers the publisher, and has a local context) while being globally unique.
In the UK we have identifiers (and some are resolvable) for some organizations , e.g. those from Companies House or the Charity Commission. These do not exist for all organizations and some organizations may have more than one. Hence we have the organization_identifier element to record identifiers and help match records from different publishers
Agreed.
For organisations, I might be biased but I think the Org ID format is the most useful model I’ve seen in practice: it allows for identifiers to be taken from official registries where available. More registries are being added all the time from different open data initiatives. I think this is another case where a UUID on an organisation object isn’t useful; it’d be much better if we just concatanated the name and scheme from the HSDS organization_identifier element to be the record’s id.
For organisations without official numbers in registries, then local database identifiers can be a decent fallback for the record identifier. Org-id provides guidance on how to use fallbacks to secondary, third-party, and local system identifiers:
Regarding services, we’re a long way from resolvable identifiers but we might introduce them based on the agreed UUID for a service AFTER deduplication of the same service (with different UUIDs) from different sources. If it’s simply a case of saying one record replaces another in a de-duplicated dataset, we can say that one service record replaces/isReplacedBy another.
I think Mike hit the nail on the head there. I don’t think having them based on UUIDs is necessarily the best way, though.
360Giving don’t benefit from formal journals and eGP systems the same way that Open Contracting do, but they still have the same format for grant identifiers: 360G-[publisher-prefix]-[internal-identifier-from-a-system]. It works really well, and also can be reliably used to create a URL for a grant in GrantNav:
Although one key lesson learned from 360Giving was “don’t allow vanity prefixes”. If a publisher changes their branding/name/structure etc. then there are problems with updating these identifiers. This is why Open Contracting maintain a registry of randomly generated prefixes: