Skip to content

Concepts

Dirigent has about a dozen nouns, and every one of them is either a thing you write or a row the engine keeps. That split is the whole model, so it is worth stating before the definitions: a pipeline is a definition, and a run is what happened when the engine executed one. Definitions are documents you author, version, and put in git. Runs, and everything under them, are rows the engine writes as it works -- and because they are rows rather than a process, a run survives every restart, can be resumed from where it broke, and is queryable long after the machine that executed it is gone.

Everything below hangs on that. The first half of the vocabulary describes what you write. The second half describes what the engine writes down about it.

   what you author                     what the engine records
   ---------------                     -----------------------

   Pipeline  (versioned)  ---starts--> Run
     |                                   |
     +-- Step ---------------------------+-- RunItem       (one per fan-out element)
     |     |                             |     |
     |     +-- references a Block        |     +-- StepAttempt   (one per try)
     |                                   |            |
     +-- Trigger (schedule)              |            +-- ArtifactRef  (the output)
     |
     +-- names a Connection by code (a credential the instance holds)

Pipeline

A coded, versioned definition: a parameter schema, a set of steps, the edges between them, and a concurrency policy. It is data -- one YAML or JSON document, stored whole as one immutable JSON value per version.

Editing a pipeline never mutates a version; it creates the next one. Runs pin the version they started from, which is what makes a run's history readable years later: the definition it executed is still there, exactly as it was. It also means fixing a broken document does not fix a run that already started -- that run is still pinned to the version that was broken.

A code is how documents address things (dg run regional-load): a kebab-case key, unique per instance, and the only identity a document carries. Beside it a pipeline may carry a name, a free-form human title that nothing ever references, and a description, the long-form text under it. An id is a UUID the instance mints and uses internally. A code travels between instances; an id never does.

A tags: list is what the pipeline is for, in the words a grown instance filters by rather than by name alone. Tags live in the document, so they version and apply like everything else about a pipeline, and a second apply says what they are now rather than what to add.

tags: [acme, nightly]

A tag is normalised at apply: it is lowercased, and what is left must be letters, digits and hyphens, start with a letter or digit, and be at most 32 characters. tags: [Acme] is stored as acme; tags: [nightly import] is refused at validation, at tags[0], with the tag named and the rule stated. Comparison is exact from then on, so nothing downstream has to reapply the rule. A document wears at most 16 of them, and says each one once.

Tags narrow: GET /pipelines?tag=acme&tag=nightly, dg pipeline list --tag acme --tag nightly and two chips on the pipelines screen all list the pipelines wearing both, because "the nightly acme imports" is an intersection. The runs listing takes the same filter through the run's pipeline -- GET /runs?tag=nightly&status=failed, dg runs list --tag nightly --status failed -- and asks what that pipeline wears now: a run pins the version it started from, never its pipeline's tags, so retagging a pipeline moves its whole history with it.

A requires: section is the preflight a document declares: the blocks it calls, the connections, pipelines, storage schemes and schemas the instance must hold, and workers, the capability tags a worker must carry before it may claim this pipeline's work.

requires:
  blocks: [docker.build]
  workers: [docker]        # only a worker started with --tag docker claims a run of this

Everything but workers refuses the apply when it is missing, because it cannot be made true by waiting. workers is checked against the live registry and reported, never refused: workers come and go, and a run whose tags nobody carries queues until one that carries them registers.

Beside concurrency, which decides what may be in flight at once, a document declares a priority: low, normal (the default) or high, which is how far ahead of every other run the claim takes this one's attempts.

concurrency: allow
priority: low            # a nightly bulk job, behind whatever a person is waiting on

Priority layers the way parameters do. The document declares the default, a schedule may override it for what it fires, and an ad hoc run may override it again -- dg run bulk-load --priority high is the same pipeline during an incident. Whichever wins is pinned on the run at creation, so editing the document never reorders a run already in flight. Nothing is preempted: a high run's attempts are claimed before any other's the moment a slot frees, but an attempt already running is never cancelled to make room.

A claim orders by priority first, then by round-robin fairness between the runs in flight, then by due time. Fairness numbers every attempt within its run by how many that run has already been served, so a four-hundred-item fan-out interleaves one attempt at a time with a two-step run rather than holding every slot until it drains.

A document also declares an importance: routine, normal (the default) or critical. It is how much this pipeline matters when it goes wrong, and it is deliberately written from a different vocabulary than priority so the two are never read as one scale. A pipeline can be both at once: a nightly reconciliation is low priority, because nobody is waiting on it, and critical importance, because the morning's payments are drawn from what it writes.

priority: low            # behind whatever a person is waiting on
importance: critical     # and worth waking somebody for when it breaks

Priority is read by the claim, once per attempt. Importance is read by the alert rules, when a run settles: a rule may name the least importance a pipeline must carry before it fires, so one rule pages for the critical failures while another logs every failure there is. It is mirrored onto the pipeline row at apply, so what an alert matches is what the current document says. Nothing else reads it yet.

A document may declare its own alert rules in an alerts: section keyed by rule code, each watching this pipeline and naming its connection by code:

alerts:
  nightly-failed:
    event: run_failed
    connection: ops-slack
    throttle: 1h

The document owns them: an apply creates, rewrites and removes them, the API and dg refuse to change or delete one, and pausing one stays an operator's choice that an apply keeps.

Detail: design.md section 12 for the document format, the code and step-name grammars, and what an apply does; operations.md for the rule that reads an importance, and examples/patterns/importance-critical.yaml for both words in one document.

Step

One node in a pipeline's DAG. A step names a block, gives it config, declares which other steps it depends_on and under what rule, and carries the engine-level knobs: retry policy, timeout, sensor poll and deadline, an optional for_each.

Steps live inside the pipeline document -- they are not rows. Only attempts get rows. Step names are snake_case, because they are read back inside ${steps.<name>.output.*} where a hyphen would look like a minus sign. A step may also carry an optional name: for a reader, but the map key stays the only thing depends_on and ${steps....} can name. A step is an addressable thing like any other, so it wears the same quartet: the map key is its code, the name is the optional title nothing references, and a node of a run's graph carries exactly those two -- code, which every edge and every attempt references, and name when the document gave one.

The knobs that belong to the engine are deliberately step-level rather than buried in a block's config: retry, timeout, poll, deadline, on_timeout, for_each, items, continue_on_failure. They mean the same thing for every block, so they are written in the same place for every block.

Source and reply

A step is exactly one of three forms: a block, a source or a reply. A source is where an input enters: steps: ask: source: {kind: endpoint, path: /greet/{name}}. It takes a name, a description and a sample beside its kind's own fields, and nothing a block step takes. A run of a pipeline with sources starts with the source it entered at already settled, its output the input -- for the endpoint kind, the request's method, path, path parameters, query, headers and body -- and every other source settled skipped as not entered.

The kinds are a catalog, contributed the way blocks are: dirigent ships endpoint, storage and sensor, dirigent-block-queues adds kafka and rabbitmq, and a pack may add its own. GET /api/v1/blocks lists them under sources, each with the fields it writes, and so does list_blocks over MCP. dg pipeline schema and the instance's document schema make source.kind a choice between the installed kinds and hold each to its own fields, so an editor completes kind: storage and then uri and poll, and squiggles a field the kind does not take.

Sources are roots, and every other step descends from one: a root beside them would run on every input from every source, and is refused. A step joining the branches of two sources says all_done or always, since a run enters at one of them only. A pipeline with sources carries no schedules, because what they started would enter no source.

A reply is a leaf that answers whoever sent the input: a status, headers, a content_type, and a body (or body_from a storage URI), any of them referencing the run's outputs. It takes depends_on, rule, name and description, and the server renders it the moment the steps it depends on have settled; no worker claims it, and neither a source nor a reply is retried -- running an input again is a new run.

A run is entered by hand with dg run --local, dg run <code>, or the API's $run: --source names the source when there are several, and --input gives the input, in the shape of the source's output, which its kind makes whole -- the endpoint fills in the method its route declares, the path its template fills, and the body's type and size. Without --input the source's sample is taken. A hand run holds no caller, so a reply records what it would have answered. See examples/sources/.

Every source has a row on the instance, reconciled on each apply by its step code. Its declaration follows the pipeline's current version; whether it is paused, its token and what it has counted are the instance's, and survive the apply. A source is listening; backlogged once as many runs from it are unsettled as its max_in_flight allows; draining when it or its pipeline is paused with runs still in flight; then paused. dg source list shows each one with its state. dg apply --paused and dg dev --seed create a source a worker polls or holds paused, and dg source resume starts it; an endpoint they create stays listening.

An endpoint source is called at its route under /endpoints: path: /greet/{name} answers POST /endpoints/greet/Bergen. Unless it writes auth: {kind: public}, its caller presents the token the apply minted for it as Authorization: Bearer; the token is shown once, in the apply's result, and dg source rotate-token replaces it. With auth: {kind: hmac} the caller presents no token: it signs <timestamp>.<raw body> with HMAC-SHA256, sends the Unix seconds in X-Dirigent-Timestamp and sha256=<hex> in X-Dirigent-Signature, or in the header auth.header names. A timestamp outside source_signature_window and a signature already taken are both refused 401. The apply mints the signing secret and shows it once, dg source rotate-token replaces it, and dg source set-secret sets one the sender issued; a webhook.post step with sign_with produces exactly that signature. A route is answered by one source on the whole instance, and an apply claiming one another pipeline answers is refused. A call is checked in one order, each check answering for itself: the instance's rate for the address and route (429), the route (404, 405), the token (401), or for a signed source the body in time (408) and under max_body (413) and the signature (401), the source's own rate (429), the source and its pipeline taking input (503), the content type (415), the body in time (408) and under max_body (413), the body parsing (400), the runs in flight (503), and, when a reply can answer, a free seat among the callers this server holds (503); then the run is created. Every call that reaches a source's route is journaled, and dg source inputs reads them: the method, the path, the caller's address, the status it was answered with and the refusal's code and reason, never a header or a body, so nothing a caller presented is kept. A call refused before it was let in -- no token, a bad signature, too fast, too large, too slow -- is journaled too, but only 60 a minute for one source; past that it is answered and counted in telemetry alone, so a flood from nobody cannot fill the journal.

A source counts what came of its calls in three numbers that add up to its journal:

Count What it counts
accepted A run was started, and its caller was not answered with the run's failure: answered 202, answered by a reply, or still held when its timeout passed
failed A run was started, and the held caller's answer was the run's failure: 422 when a step on the way to a reply refused the input, 502 otherwise
refused No run was started: 401 and its signature kin, 404 for a source gone mid-call, 408, 413, 415, 400, 429 and 503

poll_failures is something else: how many polls in a row of a polled or held source have failed, which is what makes it failing and doubles its backoff.

When no reply is downstream of the source, the caller is answered 202 with where to read the run. When one is, the caller is held for the longest timeout among the replies its input can reach, capped by source_reply_max, and the first reply to settle is the answer: its status, headers, content type and body, or the object body_from names, streamed, and the journal row ends replied. When no reply can answer any more, the run's failure is the answer, as a problem naming the run and the step: 422 when a step on the way to a reply refused the input as rejected, 502 otherwise. What the step failed with stays on the run, for whoever may read it, and a reply on a failure path that means a caller to see it says so with ${steps.<name>.error.message} (.code and .params beside it); examples/sources/reply-with-the-error.yaml answers 201 on success and 422 with the check's own message on a one_failed reply. A caller still held when its timeout passes is answered 504, and the run carries on. Each server process holds its callers on one reader, woken when a run settles, and a held input's journal row ends replied, failed or timed_out.

A storage source is polled rather than called: a worker lists its uri, a prefix or a glob, every poll (at most ten years, past which apply refuses it), and each object it has not handed over before becomes one run, ${steps.<source>.output.uri} naming it. An object is known by its URI, so one copied in still carrying an old date is found and one re-copied with a fresh date is not run twice; rerun_changed: true runs an object again whenever its modification time changes, for a file that is overwritten in place. Every worker polls, and a lease on the source's row keeps two from polling one source at once. The source keeps a cursor, so a restarted worker carries on where the last stopped; moving the uri starts the cursor over. A polled source is starting until its first poll succeeds, and failing while its polls fail, each failure doubling the wait before the next. max_in_flight bounds how many of its runs are unsettled before the next poll waits. A poll is all or nothing: when a run cannot be made of one of the inputs it found, none of them is kept, the cursor stays, and the poll is retried after the backoff, so an input is never skipped. For the same reason a pipeline with a polled source may not say concurrency: skip or replace, which would drop an input already taken. Its config is resolved with ${artifacts} in scope and nothing else, since there is no run yet. Plan resolves its uri the same way and refuses a file:// one outside the artifact root as artifacts.outside_root, the refusal every poll of it would meet.

A kafka or rabbitmq source, from the queues pack, is held rather than polled: one worker keeps a consumer open on its topic or queue under a lease it renews, and each message becomes one run. A message is acknowledged -- a Kafka group's offsets committed, a RabbitMQ delivery acked -- only after its run is committed, so a message is never lost and may, after a crash between the two, arrive twice. A worker that dies lets its lease run out, and another takes the source over. Pausing, redeclaring or removing the source closes its consumer at once. Each worker holds at most source_held_max sources.

A sensor source is polled too, and what it polls is a sensor block, poked outside any run:

steps:
  arrive:
    source:
      kind: sensor
      block: playground.arrive      # a sensor that may be poked as a source
      config: {after: 5s, rows: 3}  # the sensor's own config
      poll: 2s                      # optional; the sensor's default_poll otherwise
      max_in_flight: 2              # optional
    sample: {...}                   # what a hand run enters with, in the sensor's output shape
  load:
    block: transform.jq
    depends_on: [arrive]
    config: {input: "${steps.arrive.output.messages}", program: length}

A poke that answers NotYet starts nothing and keeps the cursor it returned. A poke that succeeds becomes one run, its source node holding the sensor's output, and the next wait starts from the cursor the sensor's resume_cursor derives from that output and is poked again at once. So a sensor whose success says where it left off -- playground.arrive and kafka.consume both do -- is read on from where the last batch ended, run after run, and the runs overlap: batch N is being transformed while batch N+1 is being waited for. The sensor's ctx.started_at is when the source began waiting, the last success or the first poll, so a sensor that waits a duration measures it from there. A worker sleeps until the earliest polled source falls due, at most source_poll_tick and never less than 50 milliseconds, so a poll shorter than the tick is kept and "at once" is within 50 milliseconds. Only a sensor whose spec declares it a source may be named, and the apply refuses any other (source.sensor_not_a_source) or one no installed pack provides (source.sensor_unknown); a sensor that reads what only a run has -- scratch, work, runs, capture, a schema, the run id -- fails the poll with source.sensor_outside_run. For a topic or a queue, the held kafka and rabbitmq kinds are the main path; see examples/sources/sensor-arrivals.yaml for the sensor kind.

Block

The collective term for the pluggable things a step can be: operators and sensors. A block id is a dotted string like http.request or storage.exists, and it is public API -- stored pipelines reference block ids as strings forever, so ids are never renamed.

Both kinds share one namespace and a document never says which kind a step is, so the id has to. A sensor is named as a condition -- storage.exists, http.ready, time.window -- and an operator as an action -- storage.copy, http.request, shell.run. Reading a document is how anyone finds out what a pipeline does, so a plugin package that names a sensor as an action costs every reader of every document that uses it.

Blocks arrive from plugin packages through a single contribute() hook, and each publishes its config and output schemas. Those schemas are the catalog, and the catalog is what validates a document at apply time and what the UI renders as forms.

Detail: the block reference is generated from the live catalog, so it lists what an instance actually has rather than what the docs remember.

Operator

A block that does work: it changes the world. Call an HTTP API, copy an object between storage backends, run a container, start another pipeline.

An operator either finishes synchronously with an output, or returns a RemoteHandle -- a serializable claim on a job it submitted somewhere else -- and lets the engine take over the waiting. docker.run is the built-in example of the second kind: it creates and starts a container, hands back the container id, and never waits.

Sensor

A block that waits for the world: it changes nothing. A file appearing under a storage URI, an endpoint reporting ready, a wall-clock window opening.

The rule that matters is that a sensor never blocks a worker. Each check is one short, read-only poke; if the condition is not met yet the sensor returns NotYet, which is not a failure and costs no retry budget, and the engine writes down when to look again. A sensor waiting six hours costs a row with a due time, not six hours of a worker.

A poke that has already read some ground keeps a cursor: a small JSON value it returns with its NotYet, and the next poke receives. It is stored by the transaction that parks the attempt, so it advances at-least-once the way an operator's probe metadata does -- a worker that dies before that commit leaves the older cursor behind, and the next poke reads the same ground again. The cursor lives only as long as the waiting attempt; a poke that succeeds ends the step, and anything a downstream step needs belongs in the output. The one thing that carries a cursor past a success is a sensor source, and only as far as the sensor says.

Sensors carry their own deadline, and what happens when it expires is configuration: on_timeout: fail or on_timeout: skip. skip is the interesting one, because skipped is a real terminal outcome that downstream edges can see -- "no drop landed today" is not a failure, it is a day with nothing to do.

Detail: design.md section 2 has the precise contract for both, and why they stay two concepts on one mechanism.

Run

One execution of one pipeline version, with resolved parameters and an attributed trigger.

A run's status derives mechanically from its leaves: succeeded, failed, cancelled, or completed_with_errors -- that last one meaning something failed and the pipeline said that was tolerable. Every run points at whatever started it (a user, a token, a schedule, a source, or another run) through a real foreign key, so "who ran this?" is a join rather than a free-text field.

An operator may pause a queued or running run (dg runs pause, $pause): nothing more of it is claimed, an attempt already running finishes, and its deadlines stand still, so a resume (dg runs resume) moves every open deadline on by as long as the run was paused. A paused run still holds its pipeline's concurrency slot, so a skip schedule records a skipped firing that names it, and a queue one waits behind it, until it resumes or is cancelled.

Item

Also called a RunItem. When a step declares for_each, the run gets one item per element of that list, each with its own status, its own error, and its own retry.

A for_each that reads params.* or run.* is fixed when the run is created, so the item grid exists from the moment a run is visible: a run over five hundred inputs reads as a grid of five hundred outcomes rather than one opaque failure.

A for_each that reads an upstream step's output, ${steps.list_files.output.files}, is expanded late: "list, then fan out over what was listed". The run is created with the step waiting as one pending attempt with no item, so it reads pending rather than skipped and the run cannot settle before it; when the step becomes ready, the transaction that readied it writes one item per element of the list, in list order, and queues them all. The source may be any step upstream, and a fan-out's output -- the list of its items' outputs -- is a list like any other. An empty list skips the step, as an empty grid fixed at creation does. A value that is not a list, or a field the output does not have, fails the step as rejected with nothing written. The grid is read once: retrying one of its items retries that item, and retrying the source afterwards does not redraw it. rule: one_failed is refused on such a step, because it is ready before the output exists.

No grid may be wider than the instance's fan_out_max_items (10000 by default). A grid fixed at creation over it refuses the run; a late one fails its step as rejected with run.fan_out_too_wide.

Nor may the items together write more than the run may store. run_storage_quota, one gigabyte by default, holds everything a run and the runs its steps start write to storage, whichever item wrote it, and the write that would pass it fails its step as rejected with run.storage_quota. See what a run may store.

A step may also map over a grid another fan-out already has, by writing for_each: ${steps.spread.items} where spread is a fan-out it depends on directly. The two steps then share one grid: item 3 of one is item 3 of the other, ${item} is the same element in both, and the second reads its match with ${steps.spread.item.output.<field>} -- one file per district written from the district's own export, rather than the whole batch handed over as a list. An item whose match did not succeed is skipped, not failed, so the rest of the grid carries on. Adoption chains: a third step may map over the second's items and still read the first's. A step adopting a late grid is late too, and expands with the same indices once the grid it adopts has settled.

When the grid it adopts runs under items: continue, an item of the adopting step does not wait for the whole grid: it runs once its own match has settled, while the match's siblings are still in flight, so district 3's file is written as soon as district 3's export is done. It waits for the whole grid as before under fail_fast, where the batch stands or falls together, under a rule other than all_success, and when the step reads the grid's whole output, which holds only the items that have succeeded so far. Over a late grid the same holds: an item-wise step's grid is written the moment the grid it adopts is, rather than once that grid has settled, and each of its items then waits for its own match.

The step's items policy decides what one bad element means. Under the default, fail_fast, any failed item fails the step. Under continue, the step succeeds as long as one item did, the failures are recorded against their own items, and the run reports completed_with_errors.

Control flow

There are no conditionals in a dirigent document, and reading one is therefore reading what it does. Everything a pipeline can express about shape is in these five places, and the gaps below are as load-bearing as the features.

What is expressible

  • Branching on an outcome. Each edge in depends_on carries a rule: all_success (the default), one_failed, all_done, always. one_failed makes a step an error-handler branch; all_done makes it cleanup. The failure path is drawn in the graph rather than hidden inside a block. See examples/failure/error-handler.yaml.
  • Tolerating a failure. continue_on_failure: true on a step means the step failing does not fail the run: downstream all_success edges are skipped and the run ends completed_with_errors. See examples/failure/retries.yaml.
  • Fan-out with per-item isolation. for_each maps a step over a list, one item per element, each with its own status, error, and retry budget; items: continue lets one bad element through. See examples/graph/fan-out.yaml.
  • Fan-in as a list. A fan-out step's output is the list of its items' outputs, in item order, and the step after it runs once and reads the whole batch. Only items that succeeded are in that list. See examples/graph/fan-in.yaml.
  • Fan-out pairing. A fan-out whose for_each is ${steps.<name>.items} maps over another fan-out's grid and reads its matching item with ${steps.<name>.item.output.<field>}, which is how one item's value reaches one item of the next step rather than the whole list. See examples/patterns/fan-out-item-wise.yaml.
  • Fan-out over an output. A for_each that reads ${steps.<name>.output.<field>} maps a step over a list an upstream step produced, and its grid is written when that step has settled. See examples/patterns/fan-out-over-an-output.yaml.
  • Parallelism. Every step whose prerequisites are met is claimable, so the graph's width is the parallelism. What bounds it is worker concurrency, not anything in the document. See examples/graph/parallel-branches.yaml.
  • Gating on the world. A sensor holds a branch until a condition is met, with poll, deadline, and on_timeout: skip deciding what happens when it never is -- which is how a branch is made optional without a conditional. See examples/sensors/sensor-gate.yaml.
  • Gating on a shape. validate.schema checks a value against a JSON Schema and passes it through when it fits, so a payload from a service you do not control is refused where it enters -- naming the path that is wrong -- rather than surfacing as a confusing error in a step three edges downstream. A mismatch is rejected: the same value fails the same way, so it is never retried. See examples/validate/expects-a-shape.yaml.

What is not

  • No branching on a value. There are no expressions and no conditionals: ${...} reads a value, it never tests one. A step runs, or it does not, on the outcome of its prerequisites. Branching on what an upstream step returned means a block that decides, and fails or succeeds accordingly. $${...} is the escape in any string the resolver reads: it yields the literal ${...} and is never resolved, which is how a compose file, a shell command or a template reaches a tool with its own braces intact. The dollars collapse in pairs, so $$${x} is one literal dollar followed by the resolved ${x}.
  • No expression in a transform's place. Reshaping between two steps is a step of its own: transform.jq, map.jq, filter.jq and convert.std run a program over a value, and they need no allowlist because jq opens nothing. What they are not is a way to make the engine decide: their output is data a later step reads, never a branch.
  • No reusable step groups. A document holds one flat steps map; there is no macro, no include, and no sub-DAG. Composing means pipeline.run, which starts a whole other pipeline as a step. See examples/composition/composition-parent.yaml.

An expression layer is on the roadmap rather than ruled out; the trigger for adding one is a real pipeline that cannot be written without it.

Attempt

Also called a StepAttempt. One try of one step (or of one item of a fan-out step): its own row, its own number, its own resolved input, its own logs, its own outcome.

A retry is always a new attempt row, never a re-entry into an old one, which is what makes a retry auditable rather than a counter. Attempts are automatic when the retry policy created them and manual when a person asked, and a manual attempt is never retried automatically -- someone asked for exactly one more try.

Whether an automatic retry happens is decided by the error class the block assigned to the failure: transient and unknown are retried while budget remains, and rejected never is however much budget is left. That is why a step with max_attempts: 3 can still fail on its first attempt: a 4xx, or a config that cannot be resolved, is rejected. The one 4xx that is not is 429: the server is rate limiting, the same call succeeds once the window has passed, so it is transient like a 5xx.

Connection

A coded credential and settings record of a contributed kind the instance holds -- an HTTP base URL with its auth, an S3 endpoint with its keys. Blocks, storage backends, and notifiers all reference one by code, and a connection may carry a human name and a description beside it.

This is the reason documents are portable. A document says connection: modelling-api; it never says what the password is. The same document applies to staging and to production, and each instance resolves the code against its own credential. Fields a connection kind declares as secret are encrypted at rest and redacted in every API response.

An alert channel is one of these too: a slack or an email connection is what an alert rule delivers through, minted and sealed exactly like the credential a block uses.

A step's connection must be of a kind its block takes. Each block says so for each of its connection fields, and the block reference lists it: sql.query takes a sql connection, and http.request, http.ready and webhook.post take an http one or one of any kind carrying a base_url, such as a pack's dhis2 or fhir. Plan and apply refuse any other, naming the kinds the field takes, and dg validate does the same offline for a connection the document carries. A webhook connection is the one that trips people up: it is the alert channel's, a single url an alert is POSTed to, and webhook.post refuses it with document.step_connection_for_alerts. The block POSTs to an http connection's base_url plus the step's path, or to an absolute url with no connection at all.

A connection something uses cannot be deleted. DELETE /connections/{code} and dg connection delete answer 409 server.connection_in_use while a pipeline's current version names the code -- in a step, in a source, or under requires.connections -- while an alert rule delivers through it, or while the instance's storage_connections configures a scheme from it. GET /connections/{code} and dg connection show say which, under used_by. A past version holds nothing back, and a code written as a ${...} reference is not one the server can read until a run resolves it.

Detail: operations.md for the key that encrypts them, operations.md for the channels, and security.md for the lifecycle.

A document that carries its own

A document meant to run on its own -- a published example, a bug reproduction -- has no instance to resolve a name against. It may carry the connection instead, beside its steps:

connections:
  demo:
    name: Demo API
    kind: http
    description: The public demo API.
    config:
      base_url: https://demo.example.org/api
      timeout: 30s

A carried connection holds settings alone. One holding a field its kind declares secret, such as basic_password, or a URL with a password in it, is refused at plan with document.carried_connection_secret, wherever the document is planned: a document never holds a secret, however public. A connection that needs one is created with dg connection create, or handed to a local run in a connections file.

dg run --local seeds these into its throwaway instance, so the document runs with nothing to set up first. name and description are optional and label the connection it seeds; a connections file takes the same fields. --connections FILE replaces one by code, which is how the same document is pointed at your own instance without editing it.

A document carries its JSON Schemas the same way, in a top-level schemas: section keyed by code, and a validate.schema gate names one of them by schema. dg run --local seeds these too, and --schema FILE replaces one by code; a carried schema needs no requires.schemas entry, because it satisfies its own reference.

A server refuses to apply a document that carries a connection or a schema. An applied document is stored, versioned, exported and diffed, and a credential -- or a shape every version would then repeat -- would be in all four. Carry a connection only where the document has to stand alone and the connection needs no secret; anywhere else the instance holds it and dg connection create makes it. A shared instance likewise holds its schemas, applied once with dg schema create and named in requires.schemas.

Trigger

What starts a run on its own, or by hand. Two kinds:

  • Ad hoc -- dg run, the API, or the UI. Always available, always parameterized.
  • A schedule -- cron, a fixed interval of at most ten years, or a single instant, each with its own timezone and its own parameter overrides. Several per pipeline is the normal case: nightly against staging and weekly against production is two schedules on one pipeline, not two pipelines.

A run started by a call, a file landing, a message or a sensor's success is not a trigger's: the pipeline's first step is a source, and the input reaches its steps as that step's output.

Each carries a code unique within its pipeline, plus the same optional name and description a pipeline has. A schedule may pin a priority on the runs it starts, overriding the pipeline's.

Schedules can be declared in the document's triggers: section, in which case an apply materialises them and marks them managed. One created by hand with dg schedule create is not managed, and an apply that does not mention it leaves it alone.

A kind: triggers document is the third owner. It names one pipeline and declares clocks for it, which is how an operations team keeps a clock file over a pipeline another team defines:

format: dirigent/v1
kind: triggers
code: document-nightly            # this document's own key, unique among triggers documents
name: Nightly batch clocks        # optional
pipeline: managed-and-manual      # the pipeline these fire, defined in its own file
triggers:
  schedules:
    - code: ops-nightly
      cron: "0 4 * * *"
      timezone: Europe/Oslo
      params: {day: "2026-01-01", environment: production}

It has no steps and writes no pipeline versions. It is applied through the same dg apply and reconciles the same way, but only over the rows it declared: its own apply never touches the pipeline's inline triggers, the pipeline's apply never touches its, and neither touches a hand-made one. An empty triggers: retires everything it owned. A code already taken on that pipeline under a different owner is refused at apply, naming the owner, and a pipeline: no instance holds -- or one that is deactivated -- is refused at pipeline. A directory apply orders pipeline documents before triggers documents, so a directory carrying both converges in one pass.

The instance's triggers documents are read at GET /api/v1/trigger-documents and GET /api/v1/trigger-documents/{code}, which names the schedules the document owns, and with dg trigger-document list and dg trigger-document show CODE. Deleting one -- DELETE /api/v1/trigger-documents/{code} or dg trigger-document delete CODE -- takes its rows with it and leaves the pipeline, its inline triggers, and every hand-made row alone. Deleting the pipeline takes its triggers documents, which are meaningless without it.

Detail: design.md section 8 for the misfire policy and exactly what an apply does and does not touch.

Artifact

Also called an ArtifactRef: the durable record of a step's output. It carries the content type, the byte size, and a digest, plus either the value inlined into the row or a URI into pluggable storage.

The rule underneath it is that nothing ever passes a worker-local filesystem path between steps. A step's output has to be readable by whichever worker claims the next step, possibly on another machine, possibly days later on a manual retry -- so it is a URI or it is a value, never a path.

Each run also gets a scratch prefix (<artifact-root>/runs/<run-id>/...) for intermediate files, which a step names as ${run.scratch}. A pruned run takes its scratch with it; see retention.

${artifacts} names the root itself, and the two are the whole choice a document makes about where bytes go: ${run.scratch}/x.json is swept when the run is pruned, and ${artifacts}/kept/x.json is kept until somebody deletes it. Both resolve against whichever backend the instance is configured with, so neither puts a local path or a bucket name in the document.

Detail: design.md section 10.

Two more worth knowing

Trigger rule. The condition on an edge, deliberately using Airflow's names and meanings: all_success (the default), all_done, one_failed, always. This is how error handling is expressed -- an error branch is an ordinary step behind a one_failed edge, drawn in the DAG rather than coded inside a block. A prerequisite that failed makes an all_success edge unsatisfiable forever, so its dependent is skipped rather than left pending.

Concurrency policy. Per pipeline: allow, skip, queue, or replace. It decides what happens when a run is asked for while one is already in flight.

The full glossary, including the engine-internal terms (waiting, lease, lost-job policy) and a column mapping every name onto its Airflow and Prefect equivalent, is design.md section 17.