Conventions

The choices this codebase makes that a reader does not expect, and why they are choices rather than accidents. The binding rules live in CLAUDE.md at the repo root; this page is the human tour of the ones worth understanding before you read much code. docs/ui-conventions.md is the same thing for the web UI.

Documentation lives with the value, and tooling reads it

Two comment notations here are not comments to the next editor -- they are data other code reads.

A field docstring is a string literal on the line after a pydantic field:

class S3StorageConfig(BlockModel):
    region: str = "us-east-1"
    """The region signed into every request."""

BlockModel sets use_attribute_docstrings=True, so pydantic lifts that string into the field's JSON Schema description. From there the one sentence travels to the block catalog, the generated blocks.md table, and the config form the UI generates -- so a block author writes user-facing documentation once, beside the field, and every surface renders it. A string before the field, or a # comment, is invisible to all of it. Position is the mechanism, not taste: the field docstring comes after the field the way a function's comes after its def.

A docstring is written in reStructuredText and every reader renders markdown, so the two meet at the catalog boundary: dirigent_common.docstrings.as_markdown rewrites RST inline literals as markdown inline code on every description in a contributed model's schema. Write code in double backticks and it arrives as inline code everywhere.

A reference entry says what a thing is and stops. The generators behind blocks.md and settings.md render the first paragraph of a description and nothing after it, so tuning advice and the arithmetic behind a default belong in operations, not in the docstring's opening.

A constant docstring is a comment beginning #: on the line before a module constant:

#: How far into the container's log a probe has read.
LOGS_SINCE = "logs_since"

Same idea for a bare NAME = value where a following string would not read cleanly: #: attaches documentation autodoc can pick up. A plain # documents nothing.

Code comments say only what the code cannot

A comment states what a thing does, or a constraint the code cannot show -- nothing else. If a sentence still reads sensibly with "we chose this because" in front of it, it is rationale, and rationale belongs in the commit message or the roadmap, never in a file. No history, no pointing at other files, no restating the next line in prose. Examples are the one exception: they teach, so an example step may say why it is configured as it is.

kind, never type

The discriminator dirigent owns is always kind -- record kinds, block kinds, connection kinds. type is kept only where an external standard or the language fixes the word: JSON Schema, MIME content types, SQLAlchemy column types, Python's type().

Every addressable thing carries the same four fields

id, code, name, description, each with one job. id is the uuid a machine holds (a UUIDv7, time-ordered). code is the addressable key -- constrained, unique, and what a URL, a document and every reference carry. name is an optional human title with no identity semantics at all: nothing is ever referenced by it, and two rows may share one. description is long-form and markdown-capable. A screen renders the quartet one way everywhere: the title is the name if there is one and the code otherwise, the code is always on screen in mono and never drawn twice, and the description is the body. A user's username is their code; a step's map key is its own.

A custom format string is how a field declares dirigent meaning

JSON Schema's format keyword carries strings the standard does not define, and dirigent uses that for two jobs.

For discovery: a storage-URI field is typed StorageUri, which publishes format: storage-uri. At apply, the engine finds every storage field across every block by that marker -- without knowing any block's field names -- and checks the URI's scheme against what the instance's backends claim, so a scheme no backend serves fails at apply rather than at the first write.

For assertion: dirigent ships base formats (ulid, uuid4, uuid7, md5, sha256, base64, ...) on top of the standard ones, so a schema writing format: uuid7 rejects a v4. Both are the same mechanism -- a non-standard format string -- pointed at discovery in one case and value-checking in the other. A pack contributes its own through the formats surface, and the JSON Schema guide has the format-versus-pattern rule. JSON Schema itself is targeted at Draft 2020-12 throughout.

models are ORM, schemas cross a boundary

models.py holds SQLAlchemy classes and nothing else. A schemas module holds pydantic types that cross a boundary -- request, response, wire, configuration shapes -- and the package that owns a boundary owns its schemas, so the server imports its wire types from the client rather than redefining them. A pydantic type used by one module only is not a schema: it stays with the code that uses it, because there pydantic is standing in for a dataclass. A block's config and output models stay with the block, where a plugin author expects them.

A wire schema is named for what it carries: <Name>In and <Name>Out for an entity's request and response, <Name>Update for a PATCH body, <Name>Detail for the superset one read answers with, <Verb>Request for an action's body, and an answer named for what it is (ApplyResult, RunAccepted). Enums, nested components and singleton reads carry no suffix.

A measured quantity and a configured one are spelled differently

A measured timing in emitted data is duration_ms, an integer of milliseconds; a measured size is <thing>_bytes, an integer of bytes. A configured duration is a humane Duration (15m, 48h), and a configured size is a humane Size (16KB) -- the type says the unit, so a configured size carries no _bytes suffix. The reader always knows whether a number is a machine's measurement or a person's setting.

Every command speaks NDJSON, and the terminal decides who hears it

A command writes one record per line to stdout -- no banner, no table, no colour -- whenever stdout is not a terminal: a pipe, a container's log, an agent's shell and CI all read records without asking. At a terminal the same records are rendered, the way dg format renders them. --json asks for records on a terminal, -o console for the rendering into a pipe, the log_format setting -- DIRIGENT_LOG_FORMAT, a project's .env, or its dirigent.yaml -- names either once, and dg format renders a stream that was kept or piped. Every record carries a kind, which is what a formatter dispatches on and what jq selects by, and it carries what its rendering needs, so no renderer reads the run a second time. A table is a rendering of a record, and it lives in the formatter, never in the command. dg dev at a terminal renders its lines and dg init in a pipe writes records: the rule reaches every command.

Two commands answer with one plain line at a terminal and in a pipe alike, because the value is the whole output and there is nothing to dispatch on: dg --version, which behaves like --help, and dg secret-key, so DIRIGENT_SECRET_KEY=$(dg secret-key) works in a shell and in a .env.

Giving a command a record

A command states each fact it has once, by calling emit_fact(kind, message=..., **what it carries). The kind names the fact rather than the entity it happened to -- schedule.paused, pipeline.deleted, db.upgraded -- because the entity's own name is already a listing kind whose row rides under fields, and a reader that dispatches on kind must be able to tell a row from a change. The message is the verb in one word, and the fields are what the rendering needs and nothing else: the identity quartet where the fact has an entity, the value that changed, and for a check the list of problems it found. A refusal the CLI itself decided on is refuse(...), which writes the same error record the server's own problem shape becomes. No command calls console.print: emit_fact writes NDJSON or hands the record to the formatter, and which of the two happens is the invocation's business.

The rendering is the formatter's. A fact that fits one line needs nothing more -- the console line already carries the message and every field. A fact that does not fit a line carries the bulk in a field named in summaries.BULKY, so it stays off the line, and gets an entry in summaries.RENDERERS keyed by its kind that draws it: run draws the run's steps and timings, validation the problems and the document's graph. A kind with no entry renders as its line, which is why a record from a newer dirigent still reads. The test asserts on records -- only(stdout, "schedule.paused")["code"], refusal(stdout)["problems"] -- and never on the drawing; the few tests that cover a rendering call the formatter directly, in test_formatters.py, with colour set explicitly rather than taken from the terminal.

Refusals carry a code

Every refusal dirigent produces is a catalogued Message: a stable dotted code, and one English template with named params. A block raises BlockFailure(MESSAGE, error_class=..., **params), a domain refusal is a DomainError carrying its class's message or the one that applies, the engine's own failures are Failure.rejected(MESSAGE, **params), and the CLI refuses with refuse(MESSAGE, **params). Each carries code and params beside the rendered sentence, and the problem document, the attempt row and the error record all carry them onward.

A code is stable API, exactly as a block id is: adding one is compatible, renaming one is not. The text is not -- it may be reworded whenever a better sentence is found, because what a reader holds on to is the code and what a translation replaces is the template.

The surface that draws a refusal decides its words

A refusal is minted once and read in several places, so the sentence it was minted with is not the sentence a person reads. The code and the params cross the wire; each surface renders its own prose from them. dg renders from the Python catalogue. The web UI renders from LABELS.refused in src/lib/labels.ts, keyed by the same dotted code, and translated with every other word it says -- so a translated interface does not draw translated chrome around an English refusal. detail still crosses the wire, for a log, a curl and any consumer with no catalogue of its own, but the browser reads it only as the fallback for a code it has no sentence for.

That fallback is why scripts/check_refusal_labels.py exists: a code with no label still works, in English, silently, which is the fault put back. It sorts every code the workspace mints into the ones a pack serves through its labels contribution, the ones only a terminal reads (cli, health), the ones whose whole template is one param and so have no words to translate, the ones the UI says, and the ones recorded in scripts/unlabelled_refusals.txt as having no words yet. That record is a ratchet: a new code may not join it, and a code given a sentence must leave it.

A carrier a refusal reaches a person through carries the pair, not only the sentence: error_code and error_params on the run, the fan-out item, the attempt, the step report, the run report and the log line, and on the AlertMessage a notifier receives. A cancellation is a refusal too, so cancel_run takes a Message rather than a reason in prose.

The text names the remedy. no connection coded 'dhis2' (none are configured) says what to do next; invalid connection does not. The params carry the specifics the template interpolates, and never a secret, a credential, or the value that failed validation -- the pydantic mapping names the kind of the input it refused and drops the input itself.

The text names no tool, because more than one surface renders it. The CLI is free to tell a CLI user to run a command -- a cli.* or health.* message knows it is the CLI -- but a message the web UI also draws must not send a person somewhere they are not: one that says to type dg secret-key is a dead end in every surface but the one that command belongs to. A shared message says what is wrong and what must become true -- DIRIGENT_SECRET_KEY must hold one -- and each surface offers its own way there. Where a surface has no way to offer, the fact alone is still better than an instruction its reader cannot follow.

A refusal is written for the person reading it, so a Python class name is never the sentence. What an exception said about itself is an operator's detail: it rides in the params, in the step's log and in the record, behind a sentence somebody wrote. A block that fails in a way nobody anticipated is refused with run.block_raised, which names the block and says, after it, what the exception said about itself, scrubbed of the step's secrets; dirigent_common.raised_detail writes that text, and the class rides in the params and the step's log where an operator reads it. That helper is what a HealthReport.detail uses too, so an instance's own words come first and the class is the last thing left to say.

A prefix has exactly one owner. A runtime package owns its own name (common, plugin, client, server, cli, testing), and one package may own a second where a family of refusals is its own thing: validation beside common for the pydantic mapping, health beside cli for the checks, notify for the built-in notifiers. A block family owns its family name (base, execute, http, parquet, queues, sql, storage), an engine inside a family owns two segments (sql.duckdb, transform.jq), and the core owns one prefix per area (auth, pipeline, document, schedule, source, alert, secret, artifacts, run, reference, host, parameter, schema). A pack's prefix is its pack name -- dhis2 for dirigent-dhis2. dirigent-common/tests/test_messages.py walks every catalogue the workspace imports and fails when a prefix is unowned or a code is minted twice.

An exception a package raises for itself, and catches before it answers -- the CLI's ParamError, SourceError and ProfileError, a transform engine's TransformError -- is not a refusal until it is one. The refusal is the refuse or the BlockFailure that catches it, which carries the code; the sentence it caught rides along as that refusal's detail param. What must never happen is a refusal reaching a person under no code at all.

Log lines are not refusals. ctx.log.info, a process record, a worker's heartbeat: those are events, they carry no code, and nothing here applies to them.

dirigent_testing.check_pack_messages holds a package to all of this, and packages/dirigent-testing/tests/test_wording.py points it at this workspace. It ships from dirigent-testing rather than from scripts/ because a pack in its own repository is held to the same rule and cannot reach into this one; the plugin page is where a pack author reads it. The frontend's half of the same rule -- every word a person reads minted in src/lib/labels.ts -- is scripts/check_ui_labels.py, and the UI conventions is where that one is written down.

Plugins are pluginkit, and files end in .yaml

Any extension point -- blocks, connection kinds, storage backends, notifiers, format checkers -- is built on pluginkit through dirigent-plugin, never a hand-rolled registry. And a YAML file this project owns ends in .yaml, never .yml; a file a tool insists on naming, like mkdocs.yml, keeps the name the tool expects.

The test lanes, and the one that never gates

The fast lane (make test) runs on SQLite and is what every change is held to. Four markers are deselected by default and each has its own target and its own CI job: postgres for what only a real PostgreSQL proves, s3 for a real object store, docker for a real daemon, and queues for real brokers. e2e -- the product driven through dg as a person drives it -- is marked so make e2e can run it alone, but it is in the fast lane and gates every change with the rest.

A pull request does not run the whole fast lane. make test-affected asks scripts/affected_suites.py which packages the diff reaches -- the ones it touches, and every package that depends on one of them -- and runs only those suites. The graph comes from the packages' own pyproject.toml files, plus the two edges no manifest states: the plugin host discovers every dirigent.* entry point in the interpreter, so a pack reaches dirigent-core, and dirigent-testing ships a pytest plugin, so it reaches every suite. Anything outside packages/ runs all of them, because a document, an infra file or the root manifest is read by tests that reach out of their own package. A push to main runs every suite once under coverage, and that is where the 90 percent floor is held.

load is the exception that never runs in the gate. make load fans a run out over fifty attempts logging five thousand lines each, against a real PostgreSQL, and writes what the log path cost -- rows a second, flush sizes and durations, how long a line takes to become visible, connections held, and what the settled reads cost -- as load records on stdout. It is a measurement, not a threshold: it is run on purpose, by a person, and read.