Docker¶

The docker.* family runs containers as pipeline steps. It is four blocks:

docker.run runs one container and lets the engine probe it to completion. It speaks the Docker Engine HTTP API over the daemon socket.
docker.compose.up and docker.compose.down run a whole stack from a compose file. They shell out to the docker compose CLI -- only the CLI can orchestrate compose -- and read the stack's state back over the Engine API.
docker.build builds an image from a context with buildx, and pushes it where a connection holds a registry credential.

Each block's fields are the generated block reference; this page is the family: what they share, how the daemon is reached, and the compose lifecycle that ties up, a drive step, and down into one run.

These are unsafe blocks¶

Every block here declares local_execution, which means the engine refuses to run it unless the instance names its id in DIRIGENT_ENABLED_UNSAFE_BLOCKS. That gate exists because reaching a Docker daemon is reaching root wherever that daemon runs: anything that can talk to a daemon can start a privileged container on its host. "Can edit pipelines" must never silently mean "can run containers on the worker's host".

Enable them per instance, and only for a worker you trust with what its daemon controls:

dg run --local examples/docker/docker-compose-stack.yaml \
  --enable-unsafe docker.compose.up,docker.compose.down,docker.run
export DIRIGENT_ENABLED_UNSAFE_BLOCKS='["docker.compose.up","docker.compose.down","docker.run"]'

The daemon these blocks reach¶

A block talks to whatever daemon DOCKER_HOST names, reading it and the TLS variables beside it the way the docker CLI does:

`DOCKER_HOST`	Reached as
unset	the local unix socket `/var/run/docker.sock`
`unix:///path/to.sock`	that unix socket
`tcp://host:port`	`http://host:port`
`tcp://host:port` with `DOCKER_TLS_VERIFY=1` and `DOCKER_CERT_PATH`	`https://host:port` with the client `ca.pem`/`cert.pem`/`key.pem`
any other scheme, `ssh://` included	refused: the daemon is reached over HTTP here, and nothing tunnels one

The reference shape: a dind sidecar¶

infra/compose.yaml gives the worker its own daemon. A docker:dind sidecar serves tcp://docker:2376 with TLS required, generating its CA, server and client certificates into DOCKER_TLS_CERTDIR=/certs; the worker mounts the client half read-only and carries three variables, which nothing else in the stack does:

DOCKER_HOST: tcp://docker:2376
DOCKER_TLS_VERIFY: "1"
DOCKER_CERT_PATH: /certs/client

A pipeline's containers are then the sidecar's, and no host socket is mounted anywhere. The API blocks read DOCKER_HOST through the resolution above; the CLI blocks pass these three to the docker binary they run, so env_allowlist never has to name them and a document cannot point a step at another daemon.

The sidecar runs privileged: true, which docker-in-docker requires -- it has no unprivileged mode that starts containers. It keeps /var/lib/docker on a named volume, so an image pulled once survives a restart. It shares a compose network, docker, with the worker alone, so the names postgres and s3 do not resolve for a pipeline's container. It still reaches the internet, and whether it reaches the database by address depends on the host's Docker, not on this file:

Measured from the sidecar	Linux: Docker Engine on a GitHub `ubuntu-24.04` runner	macOS: Docker Desktop 29.9.2
The database, by name	does not resolve	does not resolve
The database, by address	no connection	connects
The same, from a container inside the sidecar	no connection	connects
A port published on the host's loopback	not through the network's gateway	not through the gateway, but connects through `host.docker.internal`

The Linux column is what the compose CI lane checks on every change. It says nothing about another distribution, Docker version or firewall setup. The macOS column was measured by hand with the sidecar's image on two networks, once.

So on Docker Desktop this network is no boundary: the name is gone and the route is not, and a pipeline's container that guesses an address reaches the database and the bucket. The stack does not close that route. A firewall rule inside the sidecar would not hold, because a pipeline allowed docker.compose.up can start a privileged container there and remove it. What stands in its way on such a host is the database password and the bucket secret, so neither should be a default. Where the data matters, run the worker and its daemon on their own host.

What allowlisting these blocks grants¶

The sidecar is not a security boundary. Allowlisting any docker.* block gives whoever can apply a pipeline root-equivalent control of the sidecar and everything mounted in it. Treat it as a decision about the host:

docker.compose.up runs the compose file a document supplies, with no filtering of what its services ask for, so a service can be privileged or mount any path on the sidecar's filesystem.
Every path mounted in the sidecar is reachable that way. That is the work volume, holding every run's working files on this worker, including credentials a step has written there while it runs, and /certs/client, the key that opens the daemon itself.
The sidecar is itself privileged, so a privileged container inside it is one step from the kernel it shares with the host. Assume a pipeline that can run one can reach the host.

The separate network narrows what is easy to reach. It is not a firewall, and it does not contain a container that has taken over the sidecar.

Run the docker blocks on a dedicated worker host. Give that worker --tag docker, or DIRIGENT_WORKER_TAGS=docker, declare requires.workers: [docker] on the pipelines that use them, and put nothing on that host you would not hand to whoever can apply a pipeline. See local execution for the rest of what that allowlist means.

One constraint the sidecar imposes: a bind mount is resolved on the daemon's filesystem, not the worker's. docker.run mounts inputs and outputs from the run's work directory, and hands the daemon those paths, so the sidecar mounts the same work volume at the same /var/lib/dirigent/work. Point a worker at a daemon that cannot see that path and every step with inputs or outputs mounts an empty directory.

Mounting the host's /var/run/docker.sock into a containerized worker is the discouraged fallback: it works, but it grants that container root on the host. See operations.

The worker image (infra/Dockerfile) always ships the docker CLI with the compose and buildx plugins, so the CLI blocks have something to run. A step's HOME is its own work directory, so the CLI blocks write the run a docker config that names the worker's own plugin directory: buildx and compose resolve wherever Docker Desktop or a package put them.

The `docker` connection kind¶

A step that names no connection reaches the daemon the worker's own environment names, which is the table above and nothing else. A docker connection is how a document names one instead, and how a build reaches a registry:

Field	What it holds
`host`	The daemon, in one of the two schemes the worker speaks: `tcp://host:2376`, with the TLS triple below, or `unix:///var/run/docker.sock`. Empty leaves the worker's own environment standing.
`tls_ca`, `tls_cert`, `tls_key`	The client TLS triple for a `tcp://` daemon, in PEM form. All three or none.
`registry`	The registry a push authenticates to, such as `ghcr.io`. Empty means Docker Hub.
`username`, `password`	The registry credential, together or not at all.

A connection names a daemon, a registry credential, or both, and one that names neither is refused. dg connection check proves whichever it holds: docker version against the daemon, and a docker login followed straight by a docker logout against the registry, in a config directory of its own so nothing about it persists.

dg connection create docker build-daemon \
  --set host=tcp://docker:2376 \
  --set "tls_ca=$(cat /certs/client/ca.pem)" \
  --set "tls_cert=$(cat /certs/client/cert.pem)" \
  --set "tls_key=$(cat /certs/client/key.pem)"
dg connection check build-daemon

tls_key and password are sealed: encrypted at rest, redacted in every API response, and never an argument or an inherited variable. Each reaches the CLI and the Engine API as a 0600 file in a 0700 directory in the system's temporary directory, outside the roots a sibling step can read -- the TLS triple as DOCKER_CERT_PATH, the login as a DOCKER_CONFIG of its own -- and both directories are removed when the step leaves, on success, on failure and on cancellation.

A cancel reaches a daemon only the worker can. $cancel runs where it was asked, and in the shipped stack the server has no route to the daemon, which shares a network with the worker alone. So an attempt cancelled without its remote agreeing stays marked, and a worker the run's tags allow tells the daemon on its next sweep, every DIRIGENT_SWEEP_INTERVAL for ten minutes.

Every step of one stack names the same connection. A stack exists only on the daemon that was told to create it, so a docker.compose.up on one daemon and a docker.compose.down on another tear down nothing. The up, the docker.run that drives it and the down all carry the same connection:, and examples/docker/docker-remote-daemon.yaml is that document.

A connection names the daemon; it does not name the worker. A compose file and a build context still live on the worker's own filesystem, so a pipeline that names one keeps requires.workers: [docker] beside it exactly as one that does not.

Files live in the run's work directory¶

A compose file, a build context and a bind mount are things a tool opens through the filesystem, so they live in the run's work directory on the worker's own disk -- whatever the artifact root is. Within it:

a compose file is either inline content or a path relative to that directory (one an upstream git.checkout put there). Inline content is written under compose/<step>[/<item>]/attempt-<n>, so two compose steps of one run, and two items of one fan-out, never write over each other's document. It takes no ${...} reference, refused at apply and when the step starts, because a value spliced into YAML can add keys. Write $${NAME} in the file and give the reference to env under NAME, and compose reads it as one value;
a build context is a relative directory, with its Dockerfile relative to that.

Which program runs is the host's decision. docker.compose.* and docker.build run docker from the worker's own PATH, so a host that wraps it, or spells compose another way, puts a wrapper named docker first on that PATH. A step's env cannot set PATH, HOME, XDG_CONFIG_HOME, DOCKER_CONFIG, DOCKER_CONTEXT, DOCKER_HOST, DOCKER_CERT_PATH, DOCKER_TLS, DOCKER_TLS_VERIFY, BUILDX_CONFIG, SSL_CERT_FILE, SSL_CERT_DIR or an LD_* or DYLD_* variable (execute.steering_environment), because the CLI holds a connection's TLS key and registry password while it runs. A daemon is named by a docker connection instead. The CLI reads its config, and so its plugins and credential helpers, from a private directory for that one step, never from the run's work directory, where an earlier step's files could plant one.

docker.run's outputs puts a file in either place. Each entry maps a name the container wrote in its output mount to a target: a target carrying a URI scheme is copied to storage, where it outlives the run, and a target without one is a path relative to the work directory, which is how one step produces the compose file or the build context the next step opens. inputs is always storage: a URI read into the read-only mount.

An image is not trusted about what it leaves in its output mount. Each declared output is opened without following a link, on the way to it or at its end, and has to be a regular file, so a link planted under the declared name fails the step as execute.output_not_a_file rather than copying a worker's file out. A target in the work directory is opened the same way and fails as execute.output_target_refused. A compose file, an env file, a build context or a Dockerfile named by a path that a link leads outside the work directory fails as execute.path_leaves_the_work_directory, so a repository a step checked out cannot hand one of the worker's own files to the CLI.

On the compose stack the daemon mounts that directory at the same path the worker sees it at, because a bind is resolved on the daemon's filesystem rather than the worker's. It is local to one worker, so a step that reads what another step put there must run on the same worker.

The compose blocks are written for compose files that reference pre-built images. A compose file with build: stanzas needs its build contexts already in the work directory -- getting them there is the pipeline's job, through a git.checkout or a docker.run output.

The compose lifecycle¶

This is the pattern the compose blocks are shaped around, and the reason up and down are separate steps rather than one self-contained block:

up brings the stack up detached, and it persists. The step returns once the stack is up (with wait: true, once every service is healthy) and reports the project's services and the network they share. The containers keep running for the rest of the run.
A later step drives it. A docker.run step joins the compose network by name -- the up step reports it as default_network (<project>_default) -- and reaches the services on it.
down tears it down, even on failure. A separate docker.compose.down step with rule: all_done runs whether the drive step passed or failed, so a stack is never left running. The engine already evaluates all_done; no "finally" primitive is needed.

Both blocks default their project name deterministically from the run id, so the down addresses the exact project the up created with nothing wired between them, and two runs never collide. On a bring-up that does not succeed -- a non-zero exit, a bring-up that ran past its deadline, or a step that was cancelled -- the block itself best-effort tears the half-built project down (cleanup: true) before leaving, so containers compose had already started are never orphaned. A successful up is never torn down that way: it persists for the rest of the run by design.

steps:
  stack:
    block: docker.compose.up
    config:
      wait: true
      content: |
        services:
          api:
            image: nginx:alpine

  drive:
    block: docker.run
    depends_on: [stack]
    config:
      image: alpine:3
      network: "${steps.stack.output.default_network}"
      command: wget -qO- http://api/

  teardown:
    block: docker.compose.down
    depends_on: [drive]
    rule: all_done
    config:
      down_volumes: true

The full example is examples/docker/docker-compose-stack.yaml.

Building an image¶

docker.build shells out to docker buildx build -- BuildKit, its cache and multi-stage builds come for free -- and reads the built image's id from an --iidfile rather than scraping the log. The image lands in the worker's own daemon store, so a later docker.run or docker.compose.up step on the same worker references it by tag.

Pushing¶

push: true sends every tag the build produced to the registry a docker connection names. It needs that connection: push with no connection is refused when the document is applied, and a connection with no username and password is refused when the step starts. Neither is a build that quietly produced a local image nobody can reach.

build:
  block: docker.build
  config:
    context: context
    tags: ["ghcr.io/owner/app:1.2.3"]
    connection: ghcr
    push: true

The step logs in against a DOCKER_CONFIG directory of its own under the run's work directory, with the password on stdin rather than in an argument, pushes each tag with docker push, and logs out again; the directory goes with the step. The worker's own docker config is never written to and no session survives the step. The output carries pushed, the tags that reached the registry, and digests, the digest the registry gave each of them where the CLI reported one -- a tag can be moved and a digest cannot, so the digest is the only exact name for what was pushed. The password is scrubbed out of every log line and every failure message.

examples/docker/docker-build-push.yaml is the worked document.

The orphan reaper¶

docker.compose.up names its project dirigent-<run id> and compose labels every container of it com.docker.compose.project, so a stack is addressable by the run that created it long after that run has gone. Usually nothing is left: a down step with rule: all_done runs on any outcome, and a failed up tears its own half-built project down. What is left over is what none of that reached -- a worker that died mid-run, a pipeline cancelled between up and down, a down step that never ran.

A worker sweeps for those on a cadence. Each pass lists the compose projects on its daemon, reads the run id out of each project's name, and looks that run up:

a run that is still active is never touched, whatever else is true;
a run that is terminal, and a run the instance no longer holds, are both orphans;
a project younger than the grace is left alone whatever its run says, because a stack whose up has only just returned is a stack whose down has not been reached yet. A project's age is its newest container's.

Each orphan is taken down with docker compose -p <project> down -v --remove-orphans, and each one is one record: kind: docker_reaped, carrying the project, the run id and the run's status, so dg worker shows what went. A project this instance did not create -- anything whose name is not dirigent-<32 hex digits> -- is never listed as a candidate at all.

Two settings govern it:

Setting	Default	What it does
`docker_reap_interval`	`5m`	How often a pass runs. Zero turns the reaper off.
`docker_reap_grace`	`10m`	How old a project must be before a pass will consider it.

A worker with no docker CLI on its PATH runs no pass at all, rather than one that fails every five minutes.

The same pass runs on demand:

dg docker reap --dry-run
dg docker reap

--dry-run writes the same records and takes nothing down.

The reaper only sees the daemon the worker sees. With one dind sidecar per worker that is exactly the stacks that worker created, which is the deployment this is written for. A worker pointed at a shared daemon sees every dirigent project on it, including projects other workers created -- which is correct, since the run lookup is the instance's own, but it means two instances must not share one daemon.

One daemon per pipeline¶

A stack brought up on worker A, or an image built there, is invisible to worker B and its daemon. Give the docker-capable workers a docker tag (dg worker --tag docker, or DIRIGENT_WORKER_TAGS=docker) and declare requires.workers: [docker] on the pipeline, so up, the steps that drive it, down, and any build all meet the same daemon.