JSON Schema¶

A schema is a named picture of a payload: the shape a read is expected to return, written down once so a pipeline can be refused the moment the payload moves out from under it. Dirigent holds schemas as a first-class resource -- create, list, show, update, delete -- and validates every one as a JSON Schema, Draft 2020-12 when you store it. This page is that language, taught on the shapes a pipeline actually reads.

A schema is locally authored. It is the picture you hold of the payload, not something fetched or introspected from the source. That is the whole point: the source cannot tell you it changed, so you say what you expect, and dirigent tells you when the two stop matching.

A schema is applied on its own¶

A pipeline document carries a pipeline. A schema is not part of one -- it is applied by itself, the way a connection is:

dg schema create schemas/acme-sites.json   # from a file
dg schema create -                         # from stdin
dg schema list
dg schema show acme-sites
dg schema update acme-sites schemas/acme-sites.json   # the corrected file, labels and all
dg schema delete acme-sites

A schema reads its own identity from its keywords, so there is nothing else to pass:

Keyword	Becomes	If absent
`$id`	the `code` the schema is addressed by	the file's stem (`acme-sites.json` -> `acme-sites`)
`title`	the `name` shown beside the code	the schema has no name, and the code is its title
`description`	the body	the schema has no body

code is the only piece with identity: it is constrained (lowercase, digits, hyphens), unique on the instance, and what every reference uses. title and description are free text. A $id that is a full URI is accepted, and its last path segment is taken as the code.

Changing one that is already stored¶

A shape you got wrong is corrected in place. PATCH /api/v1/schemas/{code} changes the name, the description and the body -- never the code, which is what every reference names -- and the Schemas screen is that request with the schema in the editor it was written in. The name and the description are stored beside the body rather than read back out of it, so editing title in the body does not move the name; the box does.

dg schema update is the same request from a shell, and it changes a shape, its labels, or both:

# the corrected file replaces the shape, and its own title and description come with it
dg schema update acme-sites schemas/acme-sites.json
dg schema update acme-sites -                        # or from stdin

# labels alone, leaving the stored shape untouched
dg schema update acme-sites --name "Acme sites"
dg schema update acme-sites --description ""         # given empty clears what is stored

What this was not given, it leaves alone. A label named is set, a label given empty is cleared, and one left out stays as it stands -- so a corrected shape sent on its own keeps the name it was stored with. Where a file replaces the body, that file's own title and description come with it, exactly as dg schema create reads them, because the labels should belong to the shape the instance now holds rather than to the one it no longer does. Pass --name or --description to override what the file declares. This is the one place dg and the Schemas screen part company on purpose: the screen edits a shape in the boxes beside it, so there title in the body moves nothing, while the command line hands over a whole authored file and its keywords travel with it. The Apply dialog does the same when it replaces a carried schema, for the same reason.

A run holds the shapes it started with. When a run is created it reads the body of every schema its document names -- each code under requires.schemas, and each one a step's config names -- and keeps those bodies on the run itself, the way it keeps its resolved parameters. Every attempt of that run, every later step of it, and every retry of any of its steps check against those bodies and not against the table. So an edit reaches the next run of a pipeline naming the code, and never a run in flight.

That is the same grain the rest of a run works at: the pipeline version, the parameters, the priority and the worker tags are all resolved once when the run is created and read from the run from then on. A schema is one more of them.

The bodies are held, not the codes, because there is no schema versioning here for a code to resolve back through: a run that kept only the code would be back to reading whatever the table now holds. Only the codes the document names are read, so a run carries the shapes it checks against rather than a copy of the instance's whole shelf.

A ${...} in a step's schema is refused when the document is applied, with document.step_schema_interpolated. A code that is not a code until the run has started is not one the run could have pinned, and it is not one anything else can name either -- neither the used_by that names a schema's dependants nor the requires preflight can see it. A gate names its shape by writing the code.

If a run is created while a schema its document names is not on the instance -- a state no apply can produce, since applying such a document is refused and removing such a schema is too -- the run is refused with run.schema_not_held rather than started to fail at its gate.

So an edit is reported and a removal is refused. Every schema response carries used_by: the codes of the stored pipelines whose current version names the schema, whether under requires.schemas or in a step's schema field. The edit itself still lands, because correcting a shape every pipeline should now check against is what editing a stored schema is for, the code cannot be changed to sidestep it, and whether a new body still admits what a dependent pipeline sends is not a question a schema can be asked. The Schemas screen names those pipelines beside the body instead, so the decision is made with them in view rather than blind, and the schema.updated record dg schema update writes carries the same list for the same reason. If the new shape is not what the other readers of that code expect, store it under a code of its own.

DELETE /api/v1/schemas/{code} is the half that is refused, with schema.in_use, for as long as used_by is not empty. Applying a document that names a schema the instance does not hold is already refused by the requires preflight, so a delete that went through would leave behind exactly the state no apply is allowed to create -- and the next apply of an untouched pipeline would be what discovered it. Only each pipeline's current version counts: a version nobody applies any more is history, and a pipeline stops naming a code by applying a version that does not, after which the code is free to remove.

The shape of a shape¶

Every example below describes the same read, GET /api/sites.json?fields=id,name,elevation, whose payload is an object wrapping a list:

{"sites": [
  {"id": "NO-BRGN-01", "name": "Bergen Florida", "elevation": 12},
  {"id": "NO-TRMS-02", "name": "Tromso Holt",    "elevation": 21}
]}

The smallest schema that says anything is a type:

{"type": "object"}

That accepts the payload above and rejects a list or a string. type is the first question a schema asks of a value, and the seven answers are object, array, string, number, integer, boolean, and null.

Objects: properties, required, and the closed door¶

properties names the fields of an object and gives each its own schema. required lists the ones that must be present:

{
  "type": "object",
  "required": ["sites"],
  "properties": {
    "sites": {"type": "array"}
  }
}

A field named in properties is described but not demanded -- absence is fine, a wrong type is not. required is the separate question of presence. A field that is present but not named in properties is, by default, allowed and unchecked: JSON Schema is open by default, so extra keys pass silently.

Say additionalProperties: false to close that door -- to reject any field you did not name:

{
  "type": "object",
  "required": ["sites"],
  "properties": {"sites": {"type": "array"}},
  "additionalProperties": false
}

Now {"sites": [], "pager": {}} is refused, because pager was not named. This is the difference between "the fields I need are here" and "these are the only fields there are." A read that pins a payload usually wants the closed door on the objects it fully understands, and the open door where the source may add fields you do not care about.

Arrays and the shape of an element¶

items gives one schema that every element of an array must satisfy. Nest it into the field to describe the list's contents:

{
  "type": "object",
  "required": ["sites"],
  "properties": {
    "sites": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["id", "name", "elevation"],
        "properties": {
          "id":        {"type": "string"},
          "name":      {"type": "string"},
          "elevation": {"type": "integer", "minimum": -500}
        }
      }
    }
  }
}

This is the whole acme-sites example. It accepts the payload at the top of the page and rejects each of these, each for one reason:

{"sites": [{"id": "x", "name": "Bergen Florida"}]} -- the element is missing elevation.
{"sites": [{"id": "x", "name": "Bergen Florida", "elevation": -900}]} -- elevation is below minimum.
{"sites": [{"id": 42, "name": "Bergen Florida", "elevation": 12}]} -- id is a number, not a string.

minItems and maxItems bound the length; uniqueItems: true forbids duplicate elements, with 1 and 1.0 equal, true and 1 different, and an object's keys in no order.

Numbers and strings: pinning the value, not just the type¶

A string takes minLength, maxLength, and pattern -- a regular expression the whole string is tested against. An Acme site id is a two-letter country code, a four-letter site code, and a two-digit index:

{"type": "string", "pattern": "^[A-Z]{2}-[A-Z]{4}-[0-9]{2}$"}

A pattern is read with Python's regular expression syntax and meaning, which JSON Schema's ECMA-262 patterns mostly share: $ also matches before a final newline, and \d and \w match Unicode digits and letters, so anchor with \Z or spell a class such as [0-9] where that matters. A pattern Python cannot compile is refused when the schema is stored or applied.

A pattern that backtracks, such as the nested repeat ^(a+)+$, can take time exponential in the string it reads. The engine matches without holding the worker, and a check stops at its bound: a validate.schema step at its timeout (10 seconds by default), failing as rejected with base.value_unchecked, and a run's parameters after one second, refusing the run with parameter.unchecked. A schema still fails that way on every value long enough, so write patterns that do not backtrack.

That accepts "NO-BRGN-01" and rejects "too-short" and "has a space here!!". A number or integer takes minimum, maximum, and the exclusive pair exclusiveMinimum / exclusiveMaximum.

enum: the value is one of a known set¶

enum lists the values a field may take. The Acme service's measure is a fixed vocabulary, and a payload carrying a word that is not in it is exactly the drift a schema exists to catch:

{
  "type": "string",
  "enum": ["TEMPERATURE", "PRECIPITATION", "WIND_SPEED", "WIND_DIRECTION", "HUMIDITY", "PRESSURE"]
}

"TEMPERATURE" passes; "WINDCHILL" is refused. enum works on any type -- "enum": [1, 2, 3] or "enum": [true, false] -- and const is the one-value case, "const": "HOURLY".

Reuse: $defs and $ref¶

When a shape appears twice, name it once under $defs and point at it with $ref. A site id is the same three parts wherever it appears, so it is written once:

{
  "type": "object",
  "required": ["sites"],
  "properties": {
    "sites": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["id"],
        "properties": {"id": {"$ref": "#/$defs/site_id"}}
      }
    }
  },
  "$defs": {
    "site_id": {"type": "string", "pattern": "^[A-Z]{2}-[A-Z]{4}-[0-9]{2}$"}
  }
}

#/$defs/site_id is a JSON Pointer into this document: # is the root, and the path walks to the named subschema. A $ref may point anywhere in the document, and a $def may $ref another, so a shape composes out of named parts.

Reuse across schemas: a $ref to another stored schema¶

A schema the instance holds can point at another one it holds, by code:

{
  "$id": "weather-reading",
  "type": "object",
  "properties": {"station": {"$ref": "station-code"}}
}

station-code is the code of a stored schema, and station-code#/$defs/x points inside it. The reference is resolved against the referring schema's $id first, the way JSON Schema resolves every $ref, and then reaches a stored schema in one of two ways:

A bare code -- station-code, or station-code.json -- reaches the schema stored under that code. This is what a reference in a schema whose $id is itself a code (or absent) gives.
An address -- https://example.org/schemas/station-code.json, or a relative $ref inside a schema whose $id is such an address -- reaches a stored schema only when that schema's own $id is exactly that address. A link to some other schema whose address happens to end in a stored code reaches nothing, so it is refused rather than checked against a local shape it never meant.

A run takes every schema its document names and every one they reach this way when it starts, so editing either changes the next run and never one in flight. Plan refuses a reference that reaches no stored schema (document.schema_reference_unknown, at the place the document names the schema), a schema another one references cannot be deleted (schema.referenced), and a pipeline that checks against a schema through another's $ref is listed among the pipelines using it. The Schemas screen's tester and dg schema check follow the same references.

Nothing is ever fetched. A $ref to a URL that reaches no stored schema is not retrieved, because a schema is validated on the server and the workers and a fetch would be a request from their network to an address the document chose. One to a JSON Schema draft's own metaschema resolves, because those ship with the validator. Anything else fails where it is used: a pipeline's params refuses the run, and validate.schema fails the step as rejected.

format: the string that is really a date, or a site id¶

format annotates a string with the kind of string it is -- a date, a UUID, an email. In JSON Schema, format is an annotation by default and asserts nothing; a validator only checks it when it is handed a format checker. Dirigent's engine always hands its validator a checker, so a format you write is enforced. Alongside the standard formats (date-time, date, time, email, uri, ipv4, ipv6, hostname, regex, and the rest), dirigent registers the ones its own data speaks:

Format	A string that is	Example
`ulid`	a Crockford base-32 ULID, 26 chars	`01ARZ3NDEKTSV4RRFFQ69G5FAV`
`uuid4`	a random UUID (version 4)	`f47ac10b-58cc-4372-a567-0e02b2c3d479`
`uuid7`	a time-ordered UUID (version 7)	`018f6d8e-1a2b-7c3d-8e4f-0123456789ab`
`md5`	a 32-character hex digest	`d41d8cd98f00b204e9800998ecf8427e`
`sha1`	a 40-character hex digest	`da39a3ee5e6b4b0d3255bfef95601890afd80709`
`sha256`	a 64-character hex digest	`e3b0c442...b855` (64 hex characters)
`sha512`	a 128-character hex digest	`cf83e135...3e27` (128 hex characters)
`base64`	standard base-64, decodable	`aGVsbG8=`
`humane-duration`	a duration in dirigent's own spelling, never ISO 8601	`30s`, `1h30m`

So a record's observed_at is pinned as a real timestamp, not just any string:

{"type": "string", "format": "date-time"}

"2026-01-01T09:30:00Z" passes; "last Tuesday" is refused. A format only ever narrows a string: a value of the wrong type is caught by type, and format speaks only once the value is already a string.

A pack adds its own formats the same way it adds blocks: a Contribution carries a formats map of name to checker, and every format an installed pack contributes joins the ones above in the checker the engine hands its validator. So a dirigent-acme pack contributes acme-site-id and acme-period, and a schema that writes format: acme-site-id asserts wherever that pack is installed. This keeps a schema portable: a format no installed pack contributes stays a passing annotation -- the value is valid, just unchecked -- so the same schema asserts on an instance that has the pack and passes on one that does not. A pack that does this for a real system is dirigent-dhis2.

Where a schema is enforced¶

Storing a schema validates the schema -- that it is itself a legal Draft 2020-12 document. A schema validates a payload at a validate.schema gate, which passes a value through when it fits and fails the run at the boundary when it does not. A gate never writes a shape inline: its schema is a code, and the code resolves one of two ways.

The code names a schema the instance holds, applied on its own the way a pipeline's connection is created rather than carried:

check:
  block: validate.schema
  depends_on: [fetch]
  config:
    input: ${steps.fetch.output.body}
    schema: acme-sites

Because a named schema is a resource with a code, a document that references one declares it, and applying that document on an instance that does not hold the schema is refused by the same requires preflight that guards blocks and connections:

requires:
  schemas: [acme-sites]

Or the document carries the shape itself, in a top-level schemas: section keyed by code -- the same section, and the same rules, as the top-level connections: a document can carry:

schemas:
  site-record:
    type: object
    required: [sites]
    properties:
      sites: { type: array }

steps:
  check:
    block: validate.schema
    depends_on: [fetch]
    config:
      input: ${steps.fetch.output.body}
      schema: site-record

A carried schema resolves the gate's code before any instance-held one, so a document that carries every shape it names runs on its own; it needs no requires.schemas entry, because it satisfies its own reference. Its body is checked to be a valid schema when the document is applied, the way the parameter schema is. A server refuses a document that carries a schema, exactly as it refuses one that carries a connection -- storing it would put the shape in every version of the pipeline -- so the carried form is for a standalone or --local run, and a shared instance holds its schemas as their own resource. A document applied into a local run with --also-apply keeps the schemas it carries too, so a child started through pipeline.run resolves its own gates. A dg run --local run holds no instance schema until one is handed to it:

dg run --local --schema examples/schemas/ou-record.json \
  examples/validate/the-shape-is-wrong.yaml

A pipeline's own params schema is the other place a payload is validated: the parameters of every run, whoever asked for it -- the API, the CLI, a schedule's pins, a backfill, a parent's pipeline.run -- are checked against it before the run is created, and against the same assembled checker a gate uses. So a parameter declared format: date refuses 2026-13-40 at the door rather than carrying it into the first step, and a parameter declared format: acme-site-id asserts on an instance where that pack is installed and passes on one where it is not.

The reference shelf¶

Draft 2020-12 is the full language; the working set, by what a keyword is doing:

Doing	Keywords
Naming the kind	`type` (`object`, `array`, `string`, `number`, `integer`, `boolean`, `null`)
Objects	`properties`, `required`, `additionalProperties`, `patternProperties`, `propertyNames`, `minProperties`, `maxProperties`
Arrays	`items`, `prefixItems`, `contains`, `minItems`, `maxItems`, `uniqueItems`
Numbers	`minimum`, `maximum`, `exclusiveMinimum`, `exclusiveMaximum`, `multipleOf`
Strings	`minLength`, `maxLength`, `pattern`, `format`
A fixed value	`enum`, `const`
Combining	`allOf`, `anyOf`, `oneOf`, `not`, `if`/`then`/`else`
Reuse	`$defs`, `$ref` (a JSON Pointer, `#/$defs/name`, or a stored schema's code or exact `$id`)
Identity	`$id` (-> code), `title` (-> name), `description` (-> body), `$schema`, `$comment`

examples/schemas/ and examples/validate/ are this language exercised end to end: a schema applied with dg schema create and no pipeline in sight, a document that carries the same shape itself, and a payload refused at the gate.