# Durable tasks and workers

## For humans

Interlock Tasks lets an application submit work over HTTPS and lets a machine behind NAT run it
without opening a port. Interlock remembers the task, leases one attempt to an eligible worker,
records progress and logs, and keeps the canonical result. The always-open SSE connection makes
claims fast; it is only a wake-up hint. A disconnected stream may add one polling interval and may
never lose work.

### Do this

Mint two keys for the same project. They are deliberately different credentials:

```bash
interlock keys new "app server" --kind task-producer --project <project>
interlock keys new "garage worker" --kind task-worker --project <project>
```

Run either complete sample:

```bash
# Java worker, then Java submitter in another terminal
INTERLOCK_WORKER_KEY=il_... ./gradlew :tasks-java:run --args=worker
INTERLOCK_PRODUCER_KEY=il_... ./gradlew :tasks-java:run --args='submit Ada'

# Node 20+ worker, then Node submitter
INTERLOCK_WORKER_KEY=il_... node samples/tasks-node/worker.mjs
INTERLOCK_PRODUCER_KEY=il_... node samples/tasks-node/producer.mjs Ada
```

Or use the CLI from a connected project:

```bash
interlock tasks submit demo/hello --input '{"name":"Ada"}' --idempotency-key hello-ada
interlock tasks list
interlock tasks watch <task-id>
interlock tasks workers
interlock tasks doctor --spool /path/to/worker-spool
```

Open `/tasks` for the attention-first overview, `/task-types` for per-type capacity and history,
and `/workers` for the fleet and one worker's registered handlers.

### What will bite you

- Execution is **at least once**. Use `taskId`/`effectKey` as the idempotency key at any external
  effect that supports one. A lease can expire after the effect happened and before completion was
  acknowledged.
- A handler that may have committed a non-idempotent effect must report an unknown outcome rather
  than request a retry. The task becomes `outcome-unknown`; it does not masquerade as failure.
- A task type includes an integer schema version as a separate field. Submit `audio/transcribe`
  plus `schemaVersion: 1`, never the string `audio/transcribe@1` as `type`.
- Inline input and result envelopes are capped. Put large bytes behind a signed URL or an Interlock
  file reference; Tasks carries the reference, hash, size and type, not the blob.
- Worker and producer keys do not overlap. `SCOPE_DENIED` means the key is for the other role (or is
  an ordinary SDK key with no Tasks authority).
- `NO_ELIGIBLE_INFERENCE_WORKER` is a real answer for a self-hosted model: no healthy worker in this
  project serves it, no managed fallback was used, and nothing was billed.

### Then read

- [The wire and state contract](https://github.com/mufumbo/interlock/blob/main/docs/product/tasks.md)
- [Java sample](https://github.com/mufumbo/interlock/tree/main/samples/tasks-java)
- [Node sample](https://github.com/mufumbo/interlock/tree/main/samples/tasks-node)
- [Failure catalogue](/failures#tasks-says-stale_attempt-or-outcome-unknown)
- [Self-hosted AI workers](/self-hosted-ai)

## For robots

The task row and ordered journal are truth. Treat SSE frames as notifications to re-read truth.
Every worker mutation carries `taskId`, `attemptId`, `fence`, and `leaseToken`; a stale attempt gets
`409 STALE_ATTEMPT` and must stop reporting. Never reinterpret that refusal as an accepted duplicate.

## Java

```java
HttpTaskProducer producer = new HttpTaskProducer(url, producerKey);
Wire.TaskView submitted = producer.submit(Wire.SubmitRequest.of(
    "demo/hello", 1, Map.of("name", "Ada")));
Wire.TaskView done = producer.await(submitted.id(), Duration.ofMinutes(2), Duration.ofSeconds(1));
```

```java
try (HttpTaskTransport transport = new HttpTaskTransport(url, workerKey);
     TaskWorker worker = TaskWorker.builder(transport)
         .workerId("garage-mac")
         .maxConcurrent(2)
         .handle("demo/hello", 1, EffectSafety.PURE,
             (input, ctx) -> Map.of("message", "hello"))
         .start()) {
  worker.join();
}
```

The Java worker uses separate handler and housekeeping executors. Saturating handler capacity does
not starve lease renewal. It writes a terminal completion to an atomic local spool before sending it
and removes the record only after the control plane answers.

## Node

```js
import { TaskProducer, TaskWorker } from 'interlock-node-sdk';

const producer = new TaskProducer({ serviceUrl, key: producerKey });
const task = await producer.submit('demo/hello', 1, { name: 'Ada' });
const done = await producer.await(task.id);
```

```js
const worker = new TaskWorker({ serviceUrl, key: workerKey,
  workerId: 'garage-node', maxConcurrent: 2 });
worker.handle('demo/hello', 1, async (input, ctx) => {
  ctx.progress(50, 'working');
  return { message: `Hello, ${input.name}!` };
}, { effectSafety: 'pure' });
await worker.start();
```

The Node worker has the same correctness shape: HTTPS claims and reports, SSE hints plus polling,
renewal independent of handler promises, cooperative cancellation, bounded concurrency, and an
atomic disk completion spool.

## Outcomes and retry

| Handler outcome | Task behavior |
|---|---|
| returns | attempt succeeds; current fenced completion becomes canonical |
| retryable failure | re-queued only when task policy and declared effect safety both permit it |
| unknown | `outcome-unknown`; no automatic repeat of a possibly committed effect |
| other exception | terminal failure |

Submission is effectively once for a caller-provided idempotency key within one project. Reusing
the key with the same canonical request returns the original task; changing the body returns
`IDEMPOTENCY_CONFLICT`.

## Operations

1. `interlock tasks doctor` proves the Tasks API is reachable and authorized and reports clock
   drift, queue depth, online workers and outbox debt. Pass `--spool` on a worker machine to prove
   its completion spool is readable and writable.
2. `interlock tasks workers` distinguishes ready, full, draining and offline. Full means all slots
   are doing work; do not restart it as though it were offline.
3. Drain a worker locally (`TaskWorker.drain()` or SIGTERM in the supplied runners). It renews
   existing attempts and claims none. Revoking a worker key is an authority action, not a graceful
   drain.
4. If queued work has no matching worker, compare the literal `type@version`, pool and project.
   Interlock does not guess hardware capability or silently route across projects.

## Verify

These are the fast, reproducible gates:

```bash
./gradlew :interlock-java-sdk:test :tasks-java:compileJava
npm test --prefix interlock-node-sdk
node --check samples/tasks-node/worker.mjs
node --check samples/tasks-node/producer.mjs
```

For the control plane and browser surface:

```bash
./gradlew :interlock-java-api:test -Dinterlock.test.db=true
./run-web-test.sh
```

The browser run captures desktop and phone Tasks frames. Review them: no horizontal clipping,
literal task types preserve case, status always has words in addition to color, and the next action
for a no-worker task is visible above the metrics.

## Operations evidence

Tasks schema changes are versioned in Flyway migration `V2__interlock_tasks.sql`; Hibernate no
longer has to invent this subsystem's production DDL. A consistent scheduler-only backup and a
causal scratch restore use:

```bash
scripts/tasks-backup-restore.sh backup /secure/tasks-$(date +%F).sql
TASKS_RESTORE_ADMIN_USER=... TASKS_RESTORE_ADMIN_PASSWORD=... \
  scripts/tasks-backup-restore.sh verify /secure/tasks-$(date +%F).sql
```

The verifier restores into a purpose-named scratch database, checks orphan attempts/events and
current-attempt references, reports counts and checksum, then removes only that scratch database.
For repeatable latency and fairness measurements against one or several API nodes:

```bash
INTERLOCK_TASK_PRODUCER_KEY=... INTERLOCK_TASK_WORKER_KEY=... \
  node scripts/tasks-load.mjs --urls http://node-a:8090,http://node-b:8090 \
  --tasks 1000 --workers 16
```

That smoke reports enqueue→claim separately from enqueue→terminal. A production edge fleet gate
will not label itself qualified without the requested stream tier, HTTPS, slow readers, a reconnect
storm, and origin heap/file-descriptor snapshots:

```bash
INTERLOCK_TASK_PRODUCER_KEY=... INTERLOCK_TASK_WORKER_KEY=... INTERLOCK_METRICS_KEY=... \
  node scripts/tasks-load.mjs --gate 1000 --urls https://api.interlock.sh \
  --tasks 1000 --workers 1000 --slow-readers 10 --reconnect-streams 1000 \
  --metrics-url https://api.interlock.sh/api/metrics
```