Proposal: API Field for Linking to Data Guides

Governance
1

Ahead of the Standing Technical Meeting scheduled for 2025-09-10, the authors would like to submit this proposal for consideration by the committee:

Action: Approval for the release of HSDS 3.2
2

A Retrospective on the Data Guide Field: Separating Schema Syntax, Data Semantics, and the Future of AI Auditing

Now that this proposal has been approved into the standard, it is worth sharing some retroactive commentary —especially from a validation engineering perspective.

When I first considered this update, my immediate engineering instinct was to push back. Having spent a significant amount of time building the core validation engine for the OpenReferral(UK) pipeline, I am naturally biased toward machine-readable, strictly typed, and centralized contracts. My initial view was that any local variations, profile extensions, or contextual nuances should live directly inside a well-documented local openapi.json (or Swagger/YAML specification) exposed by the individual data provider. The validation engine has the ability to review a local specification, ensuring it’s well-structured, complete, well-documented, and provide suggestions for improvement. One area that it cannot test for are endpoints supported by a data provider, but not included in either the HSDS profile version or the data provider’s local specification. I worried that introducing an external, human-readable document would act as a loophole for unstructured data descriptions.

However, looking closely at the operational realities of data publishing, I conceded my initial objections. The key is drawing a strict line between technical schema validation (syntax) and qualitative data provenance (semantics).

1. Syntax vs. Semantics: Why OpenAPI and the Data Guide Must Coexist

My initial skepticism failed to distinguish between validation syntax and data semantics. An OpenAPI specification is the ultimate source of truth for structural contract enforcement. It tells a machine how to parse a payload. It cannot, however, tell a human the real-world story behind that payload:

  • The Meaning Behind Empty Data: An OpenAPI spec can declare that a physical address property is nullable. What it cannot tell a consumer is why it is null. A user needs to know if an address is missing due to a technical error, or if a publisher intentionally redacted the locations of domestic violence shelters to protect victim safety.

  • The Data Supply Chain: An API payload can pass schema validation perfectly while leaving the consumer completely blind to its origin. Is the feed directly derived from an official database, or is it dynamically aggregated from third-party community directories? That qualitative difference changes how a developer interprets the reliability of the feed.

  • The Web of Accountability: There is often a complex chain of custody behind an API feed—involving Providers, Contributors, Assurers, Stewards, and Publishers. An OpenAPI definition has no native, clean way to articulate these distinct operational roles.

Trying to force this level of qualitative narrative into the description fields (my initial thoughts) of an OpenAPI schema clutters machine-readable files and creates severe maintenance friction for publishers.

2. The Critical Guardrail: Preventing “Shadow Endpoints”

This is where the distinction becomes vital for the community. Because the validation engine cannot test for endpoints supported by a data provider but completely omitted from the specifications, the data_guide field must never become a rug under which undocumented API customizations are swept.

  • No Out-of-Band Extensions: If a data provider extends the base HSDS profile to support custom local fields or entirely new endpoints, those structural changes must be explicitly declared in their local openapi.json.

  • Enforcing the Tooling Contract: We cannot allow a paradigm where a data provider exposes a “shadow endpoint,” leaves it completely out of the machine-readable spec, and simply writes a paragraph in a PDF Data Guide explaining how it works. If it isn’t in the schema, automated validation tooling is rendered blind, client SDK generation breaks, and integrations become incredibly fragile.

3. The Future: Using the Data Guide for AI-Powered Semantic Validation

Looking ahead, the separation of these two documents opens up a fascinating technical opportunity. With the rise of local Small Language Models (SLMs) running directly within local environments (e.g., via Ollama), the human-readable Data Guide can actually be used to automate semantic validation.

Instead of the Data Guide being a passive PDF, the validation engine can feed the guide’s text into an SLM as local context to perform intelligent, semantic audits on live API records:

  • Semantic Cross-Checking: If the Data Guide asserts, “This dataset only covers the London Borough of Camden,” the SLM can inspect sampled postcodes or coordinate fields in the live payload. If it spots records leaking into Manchester, it catches a geographic boundary violation that a standard regex postcode validator would completely miss.

  • Redaction Verifications: If the guide states that sensitive addresses are hidden for safety, the SLM can cross-reference the names of organizations against populated address fields to flag potential safeguarding leaks.

Can We Validate the Data Guide Itself?

Yes. To prevent the guide from drifting or becoming stale, the validation engine could be extended to automatically audit it:

  1. Structural & Completeness Checks: Enforcing a templated Markdown structure with a required YAML front-matter block (verifying fields like last_updated and publisher_roles ).

  2. Cross-Specification Alignment (The “Shadow” Check): Programmatically extracting terminology from the Data Guide text. If the guide discusses how a custom field is calculated, but that field does not exist in the local openapi.json file, the validator could immediately flag a specification mismatch.

  3. Internal Semantic Consistency: Utilizing an SLM to read the guide’s text to ensure the publisher isn’t contradicting themselves (e.g., claiming real-time data updates in one paragraph, and monthly scraping cycles in another).

Conclusion

The data_guide field is an exceptionally progressive framework, provided we maintain a strict division of labor:

  1. The Local openapi.json must remain the rigorous, machine-readable Technical Contract—fully declaring every single endpoint, method, and structural extension so the validation engine can validate syntax.

  2. The data_guide Field acts as the Dataset Story—providing the human-readable semantic guardrails, privacy policies, and operational workflows.

By keeping these boundaries clear, we can use traditional merthods to validate schema structure, while paving the way for lightweight, local AI tools to validate real-world semantic intent.