I’ve been sitting on this for a while and finally had a few moments to finish it up and post it to github:
The details are in the issue. I essentially make an argument that the “compiled” schemas in HSDS are actively adding complexity which harms the ecosystem and creates a burden on profile developers, maintainers, and creates confusion for data users.
This thread is to advertise the issue, and also create a space for discussion for those who don’t want to/can’t participate on Github
This is an outstanding write-up, and I couldn’t agree more with the core thesis that the differentiation between “compiled” and “uncompiled” schemas is not useful and adds unnecessary complexity to both the specification and its tooling.
I have been a vocal advocate of not pre-resolving and compiling schemas since I first got my teeth into this project. From a developer’s perspective, forcing component re-use across separate, modular files is a massive win. It keeps schemas clean, highly maintainable, and remarkably easy to work with. Furthermore, keeping files modular unlocks highly efficient in-memory caching and drastically lowers network payload overhead.
As the developer of the current .NET-based validation engine for the Open Referral UK (MHCLG) platform, I wanted to share some technical insights on why moving away from compiled schemas is not just viable, but highly beneficial for performance and stability.
Dynamic, on-the-fly reference resolution is a solved problem. Modern schema validators do not need pre-flattened files. Our validation engine handles this natively:
Dynamic Base URI Mapping: We don’t need compilation tools dynamically doing find-and-replace swaps on our base URIs , which we know historically introduced fragile bugs in tools like hsds_schema_tools.
In-Memory Schema Registry: The engine registers unresolved schemas by their $id in a local cache. When it encounters a remote $ref in a schema, it resolves it instantly against this cache in milliseconds, entirely offline.
Component Re-use: When a profile extends a base schema, the engine loads the unresolved schema, dynamically resolves the base properties, and layers the profile’s overrides on top in memory.
The current compilation process relies on non-declarative scripts in hsds_schema_tools to strip out one-to-many properties to prevent recursion. This introduces severe discrepancies.
For example, compiled/service_list.json completely lacks recently added fields like service.additional_urls , despite them being fully defined in the base schemas. Because compiled/service_list.json is explicitly defined as the return schema for GET /services in the OpenAPI specification , validation tooling is completely blind to these new fields.
Standardizing on a single, declarative, uncompiled schema ensures one source of truth.
Managing both compiled and uncompiled versions adds immense complexity for profile authors and tool developers alike. Profile authors are forced to manually override $ref locations , and developers have to write incredibly complex translation layers to handle compiled vs. uncompiled paths.
If we eliminate compiled schemas:
Profile authors only need to understand the standard, human-readable base schemas.
Tooling can simply map namespace URIs directly.
Pre-compilation is an obsolete optimization that introduces massive maintenance friction and silent validation gaps.
I strongly support committing to the outright removal of compiled schemas in the next major upgrade , and adopting the patch-fix approach in the short term to treat compiled schemas as static, declarative files rather than programmatically modifying them with bug-prone scripts. This is especially true since we must accept that modifying openapi.json directly requires triggering governance. Doing this will instantly stabilize validation tooling and lower the barrier to entry for profile creators.