With the advent of Jeff’s re-working of the ORUK validator to be more flexible and suitable for multiple audiences, we’ve started discussion about what it looks like to have a hosted instance of the validator for the community.
It’d be good to get some ideas on what people’s needs are.
As far as I can see, there are two initial dimensions to understand (although please contribute additional things!!!)
How adhoc vs how systematic would you be wanting to use a hosted validator? (as opposed to hosting your own instance)
Are people wanting a web frontend for a hosted validator or are you happier interacting with its API? (the ORUK validator currently is back-end only, with a “bring your own front-end” approach.)
So here are my questions to the community; but again, if you think there’s something else to say about this topic then say it! These questions are just a starting point.
This obviously has links to some things we’ve covered in the User Stories around validation. This thread will be used to refine or add to the user stories, but since we’re discussing the possibility of a specific instance of a hosted community validator, we’re mostly focusing on this.
Q1: By what interface/mechanism are you likely to use the validator.
e.g.
“I’ll send my data directly to the API on an adhoc basis via something
like cURL and analyse/display the results myself on my server/computer”
“I want a web interface to input my API endpoint details, and to
display results to support debugging”
“I want to use the hosted validator as part of my production pipeline,
so I will be systematically making calls to its API via a software library”
etc.
Q2: If you want a web front-end, do you have specific needs for the
interface?
“There must be a clear way to input security details to access my API
data”
“I just want to put my root URL in a text box and hit a button and
then get my results”
etc.
Misc
If you’re planning on interacting with the API of the hosted validator programmatically, it’d be good to have some more details about this. What languages or environments are important? Do you prefer to use existing frameworks to do this, or would you like to see something that abstracts that away.
e.g. a Python developer might plan to take data from their Django web application and transform it into HSDS, and therefore might envision checking it with the validator at some point in its lifecycle. They might prefer to use the hosted validator with the popular requests library by just plugging in its details, or they might want an “official” library that handles taking data and passing it to the hosted intance of the validator and passing out the results; so they don’t need to manage access themselves.
Hello Matt. This is just my UK perspective, but of course we already have a hosted validator, so your questions apply to others who may or may not be in the same situation.
Essentially, I want to avoid developers/suppliers who publish compliant APIs “marking their own homework”. Many say they are compliant or “influenced by the standard” but their feeds don’t pass validation and so can’t be processed by others without extra work (if at all).
A hosted API allows anyone to check an API feed without having to develop and host their own software. The simplest example is that of an organisation commissioning a valid feed and passing the feed’s API endpoint through the validator before paying the supplier. An open validator that can be used equally by developers and commissioners removes any ambiguity on what a data feed should look like.
I think you know my opinions on this, and I have shared with you my own deployment of the complete validation process in AWS Lambda.
The validation possesses a Swagger UI, so basically it does have a documented UI that you can plug a URL into and validate it against known HSDS profiles. It won’t accept validation against (any old) json schema, or schema versions which are not pe-configured. I think that is the right approach.
I agree with Mike’s comments on “:marking their own homework”, which is why you should let them mark their own homework…but then validate against cached known HSDS profiles.
I have NOT tried building this into a library, but I don’t see how that would work well anyway as you need to pre-load all compliant profiles and cache them (IMHO). It works best as a hosted managed service tool.
By the way…
I worked through the memory issue which I have previously mentioned, implemented validation while streaming from the source instead of loading the entire dasta response into memory.
The results are promising.
I made some changes on how memory is allocated and ran an automated set of 10 validation request to the same source, sequentially.. The system is no longer “leaking” memory until it crashes (Good); instead, it’s staying within a predictable, healthy range.
The highlights: (from AI)
No more “runaway” growth: Instead of the memory usage climbing forever, it now levels off between 1.83 GB and 1.87 GB. (This is mostly cached, fully resolved and hydrated schema)
Faster performance: Once the system “warms up” after the first task, it’s running much faster—dropping from about 13 seconds down to 5 seconds per run. (because known schemas are cached in memory)
Self-cleaning works: The system is now successfully cleaning up after itself. We can see it reclaiming space after big tasks, which it wasn’t doing properly before. (oops….this was the big one that hit 9GB…SORTED)
Stable under pressure: After the first few minutes, the memory stays very flat, which means the “stress” on the system has been significantly reduced.
Incidentally, the Validation source can be deployed on an environment with 512MB storage and 2GB ram. I think I worked out a hosted validation on AWS Lambda would typically cost around $15 a month. There are other more efficient models that might prove more cost effective. I like Lambda as you have control over the environment. I have not deployed this on other environments.
(this is not the UK validator. It’s a fork. Through configuration it supports the same validation and output as the UK validator)
The AI had some opinions about how the validator could’ve make its job easier. Pasting them below:
Based on our experience debugging and integrating an existing dataset (Airtable) into the HSDS-UK 3.0 specification, here is a consolidated list of recommendations for the ORUK Validator team. These suggestions focus on making the tool more robust for developers, easier to use, and accessible for AI agents (LLMs) building compliant APIs.
1. Dependability & Error Handling
The Problem: The most significant hurdle we faced was the generic 503 Service Unavailable error surfaced by the frontend. This was caused by the Heroku router’s 30-second timeout limit. Because our Level 2 endpoints (like /taxonomy_terms and /organizations) contained hundreds of records, the backend validation took just over 30 seconds, resulting in a silent failure from the user’s perspective, even though the data was mostly valid.
Recommendations:
Implement Asynchronous Validation: For large directories, synchronous validation is prone to network and infrastructure timeouts. The validator should immediately return a job_id and allow the frontend (or API client) to poll for status (e.g., Processing Level 2…), eventually returning the full report.
Cap Record Validation: Instead of validating every single record in a paginated list, the validator could randomly sample or cap the validation at the first 50-100 records per endpoint to ensure the response returns within synchronous timeout limits.
Surface Partial Results: If the validator hits a timeout while processing optional Level 2 endpoints, it should still return the successful Level 1 validation results with a warning that Level 2 timed out, rather than crashing the entire report.
2. Usability & Developer Experience
The Problem: Debugging strict data-type constraints (like enforcing UUID formats for IDs, or stripping URLs from email fields) required a tight feedback loop of deploy → test → fail. Doing this against a remote cloud validator is slow.
Recommendations:
Provide a Local CLI Tool: A local version of the validator (e.g., an npx openreferral-validator package or a lightweight Docker image) would be a game-changer. Developers could run validation locally in their CI/CD pipelines or locally during development before deploying to staging.
Explicit OpenAPI Spec Overrides: The validator’s auto-discovery (inferring the spec URL from the root / endpoint’s version) is clever, but developers should have a strict toggle to override this and provide an explicit openapi.json URL during testing.
Clearer “Required Field” Messaging: When a field like description is missing because it’s null in the source database, the error “Required properties are missing from object: description” is helpful, but highlighting the specific record ID that failed would speed up data-cleaning efforts immensely.
3. LLM & AI Agent Accessibility
The Problem: As an AI agent helping a developer build an HSDS-compliant API, I had to reverse-engineer the Heroku backend API payload ({“baseUrl”: “…”, “options”: {…}}) by executing curl commands to bypass the UI timeout.
Recommendations:
Provide an llms.txt file: Host an llms.txt (or .well-known/llms.txt) on the main openreferraluk.org domain. This file should contain:
A condensed summary of the HSDS-UK 3.0 schema rules (e.g., “All IDs must be UUIDs”, “Emails must pass strict URI checks”).
Documentation on how an LLM can programmatically call the validator API.
Publish an OpenAPI Spec for the Validator Itself: If the Validator API has a documented OpenAPI spec, AI agents can natively consume it and automatically validate the code they write without human intervention.
Record-Level Validation Endpoint: Allow LLMs to POST a single JSON payload (e.g., a single Service or Organization object) directly to an endpoint like /validate/service. This would allow an AI to generate a record, instantly validate its schema against HSDS-UK, and self-correct formatting errors before writing the data translation logic.
Thanks @devin . I’m hereby bringing this to the attention of @Paul_D , @oughnic , @craigwilkins56 and @ddat-russ who have been looking at the UK hosted Validator and will be involved in its ongoing development, with input from colleagues outside the UK. Watch this space .
.