Durable tasks and workers #

Raw markdown: /tasks.md

For humans #

Interlock Tasks lets an application submit work over HTTPS and lets a machine behind NAT run it without opening a port. Interlock remembers the task, leases one attempt to an eligible worker, records progress and logs, and keeps the canonical result. The always-open SSE connection makes claims fast; it is only a wake-up hint. A disconnected stream may add one polling interval and may never lose work.

Do this #

Mint two keys for the same project. They are deliberately different credentials:

interlock keys new "app server" --kind task-producer --project <project>
interlock keys new "garage worker" --kind task-worker --project <project>

Run either complete sample:

# Java worker, then Java submitter in another terminal
INTERLOCK_WORKER_KEY=il_... ./gradlew :tasks-java:run --args=worker
INTERLOCK_PRODUCER_KEY=il_... ./gradlew :tasks-java:run --args='submit Ada'

# Node 20+ worker, then Node submitter
INTERLOCK_WORKER_KEY=il_... node samples/tasks-node/worker.mjs
INTERLOCK_PRODUCER_KEY=il_... node samples/tasks-node/producer.mjs Ada

Or use the CLI from a connected project:

interlock tasks submit demo/hello --input '{"name":"Ada"}' --idempotency-key hello-ada
interlock tasks list
interlock tasks watch <task-id>
interlock tasks workers
interlock tasks doctor --spool /path/to/worker-spool

Open /tasks for the attention-first overview, /task-types for per-type capacity and history, and /workers for the fleet and one worker's registered handlers.

What will bite you #

Execution is at least once. Use taskId/effectKey as the idempotency key at any external effect that supports one. A lease can expire after the effect happened and before completion was acknowledged.
A handler that may have committed a non-idempotent effect must report an unknown outcome rather than request a retry. The task becomes outcome-unknown; it does not masquerade as failure.
A task type includes an integer schema version as a separate field. Submit audio/transcribe plus schemaVersion: 1, never the string audio/transcribe@1 as type.
Inline input and result envelopes are capped. Put large bytes behind a signed URL or an Interlock file reference; Tasks carries the reference, hash, size and type, not the blob.
Worker and producer keys do not overlap. SCOPE_DENIED means the key is for the other role (or is an ordinary SDK key with no Tasks authority).
NO_ELIGIBLE_INFERENCE_WORKER is a real answer for a self-hosted model: no healthy worker in this project serves it, no managed fallback was used, and nothing was billed.

Then read #

On this page

For humans
For robots
Java
Node
Outcomes and retry
Operations
Verify
Operations evidence

For robots #

The task row and ordered journal are truth. Treat SSE frames as notifications to re-read truth. Every worker mutation carries taskId, attemptId, fence, and leaseToken; a stale attempt gets 409 STALE_ATTEMPT and must stop reporting. Never reinterpret that refusal as an accepted duplicate.

Java #

HttpTaskProducer producer = new HttpTaskProducer(url, producerKey);
Wire.TaskView submitted = producer.submit(Wire.SubmitRequest.of(
    "demo/hello", 1, Map.of("name", "Ada")));
Wire.TaskView done = producer.await(submitted.id(), Duration.ofMinutes(2), Duration.ofSeconds(1));

try (HttpTaskTransport transport = new HttpTaskTransport(url, workerKey);
     TaskWorker worker = TaskWorker.builder(transport)
         .workerId("garage-mac")
         .maxConcurrent(2)
         .handle("demo/hello", 1, EffectSafety.PURE,
             (input, ctx) -> Map.of("message", "hello"))
         .start()) {
  worker.join();
}

The Java worker uses separate handler and housekeeping executors. Saturating handler capacity does not starve lease renewal. It writes a terminal completion to an atomic local spool before sending it and removes the record only after the control plane answers.

Node #

import { TaskProducer, TaskWorker } from 'interlock-node-sdk';

const producer = new TaskProducer({ serviceUrl, key: producerKey });
const task = await producer.submit('demo/hello', 1, { name: 'Ada' });
const done = await producer.await(task.id);

const worker = new TaskWorker({ serviceUrl, key: workerKey,
  workerId: 'garage-node', maxConcurrent: 2 });
worker.handle('demo/hello', 1, async (input, ctx) => {
  ctx.progress(50, 'working');
  return { message: `Hello, ${input.name}!` };
}, { effectSafety: 'pure' });
await worker.start();

The Node worker has the same correctness shape: HTTPS claims and reports, SSE hints plus polling, renewal independent of handler promises, cooperative cancellation, bounded concurrency, and an atomic disk completion spool.

Outcomes and retry #

Handler outcome	Task behavior
returns	attempt succeeds; current fenced completion becomes canonical
retryable failure	re-queued only when task policy and declared effect safety both permit it
unknown	`outcome-unknown`; no automatic repeat of a possibly committed effect
other exception	terminal failure

Submission is effectively once for a caller-provided idempotency key within one project. Reusing the key with the same canonical request returns the original task; changing the body returns IDEMPOTENCY_CONFLICT.

Operations #

interlock tasks doctor proves the Tasks API is reachable and authorized and reports clock drift, queue depth, online workers and outbox debt. Pass --spool on a worker machine to prove its completion spool is readable and writable.
interlock tasks workers distinguishes ready, full, draining and offline. Full means all slots are doing work; do not restart it as though it were offline.
Drain a worker locally (TaskWorker.drain() or SIGTERM in the supplied runners). It renews existing attempts and claims none. Revoking a worker key is an authority action, not a graceful drain.
If queued work has no matching worker, compare the literal type@version, pool and project. Interlock does not guess hardware capability or silently route across projects.

Verify #

These are the fast, reproducible gates:

./gradlew :interlock-java-sdk:test :tasks-java:compileJava
npm test --prefix interlock-node-sdk
node --check samples/tasks-node/worker.mjs
node --check samples/tasks-node/producer.mjs

For the control plane and browser surface:

./gradlew :interlock-java-api:test -Dinterlock.test.db=true
./run-web-test.sh

The browser run captures desktop and phone Tasks frames. Review them: no horizontal clipping, literal task types preserve case, status always has words in addition to color, and the next action for a no-worker task is visible above the metrics.

Operations evidence #

Tasks schema changes are versioned in Flyway migration V2__interlock_tasks.sql; Hibernate no longer has to invent this subsystem's production DDL. A consistent scheduler-only backup and a causal scratch restore use:

scripts/tasks-backup-restore.sh backup /secure/tasks-$(date +%F).sql
TASKS_RESTORE_ADMIN_USER=... TASKS_RESTORE_ADMIN_PASSWORD=... \
  scripts/tasks-backup-restore.sh verify /secure/tasks-$(date +%F).sql

The verifier restores into a purpose-named scratch database, checks orphan attempts/events and current-attempt references, reports counts and checksum, then removes only that scratch database. For repeatable latency and fairness measurements against one or several API nodes:

INTERLOCK_TASK_PRODUCER_KEY=... INTERLOCK_TASK_WORKER_KEY=... \
  node scripts/tasks-load.mjs --urls http://node-a:8090,http://node-b:8090 \
  --tasks 1000 --workers 16

That smoke reports enqueue→claim separately from enqueue→terminal. A production edge fleet gate will not label itself qualified without the requested stream tier, HTTPS, slow readers, a reconnect storm, and origin heap/file-descriptor snapshots:

INTERLOCK_TASK_PRODUCER_KEY=... INTERLOCK_TASK_WORKER_KEY=... INTERLOCK_METRICS_KEY=... \
  node scripts/tasks-load.mjs --gate 1000 --urls https://api.interlock.sh \
  --tasks 1000 --workers 1000 --slow-readers 10 --reconnect-streams 1000 \
  --metrics-url https://api.interlock.sh/api/metrics