Model fleets #

Raw markdown: /model-fleet.md

A model project owns its agreement, schemas, prompts, trainer, evaluator, and typed application client. Interlock owns immutable inputs and outputs, releases, channels, placement, machine credentials, downloads, process supervision, telemetry, and history.

The unit of operation is a model group, not a laptop. One group has many immutable releases and may use any number of training and inference machines.

Scaffold a model project #

From the application repository:

interlock models init my-model --trainer python --client java
interlock models register --dir my-model

The generated installation-notes.sh is the executable machine handoff. Run it from any location:

./my-model/installation-notes.sh

It uses the authenticated Interlock CLI and its own project directory to rotate the group's client and trainer keys, then prints two complete, server-bound commands ready to paste onto the respective machines. The file contains no secret; the newly created keys exist only in that command output and are shown once. Running it again creates a fresh pair and invalidates only older bootstrap keys—not machine credentials that have already been exchanged. Generated model bytes, datasets, checkpoints, and reports belong in Interlock Files, not Git.

Interactive terminals get a compact colored handoff with separate inference and training copy blocks. Redirected output is plain text, and the standard NO_COLOR environment variable disables ANSI styling explicitly.

Two keys, neither an SDK key #

Each group has exactly two active, rotatable bootstrap authorities:

interlock models key rotate client --dir my-model
interlock models key rotate trainer --dir my-model

The client key may enroll inference hosts for this group. The trainer key may enroll training and evaluation hosts for this group. Neither can administer Interlock, mutate a release, submit arbitrary Tasks, access another group, or act as an application SDK key.

The installer exchanges the group key once for a credential bound to one worker id, one group, and one role. It stores only that machine credential. Rotating a group key immediately stops new enrollments while already-enrolled machines continue. Revoke one machine credential to remove only that machine.

Install an inference machine #

The host needs Java 21, Python 3, and the selected runtime. For llama.cpp on macOS:

brew install llama.cpp

Then run:

export INTERLOCK_MODEL_KEY='<client-key>'
curl -fsSL https://PROJECT.interlock.sh/assets/install-model-host.sh | sh -s -- \
  --url https://PROJECT.interlock.sh --group my-model --role client
unset INTERLOCK_MODEL_KEY

Use --runtime mlx-lm when this host should serve MLX releases. A client install starts no default model. It downloads and starts only immutable deployments assigned by Interlock to this group.

Install a trainer #

export INTERLOCK_MODEL_KEY='<trainer-key>'
curl -fsSL https://PROJECT.interlock.sh/assets/install-model-host.sh | sh -s -- \
  --url https://PROJECT.interlock.sh --group my-model --role trainer
unset INTERLOCK_MODEL_KEY

The trainer receives each dataset, base model, checkpoint, and executable procedure as an immutable, hash-verified Interlock artifact. Auth remains in the generic bridge; procedure code receives only attempt-scoped file paths and the bounded metrics/checkpoint protocol.

On an Apple Silicon trainer, opt in to the pinned MLX-to-GGUF derivation toolchain when this model group needs NVIDIA-compatible candidates:

export INTERLOCK_MODEL_KEY='<trainer-key>'
curl -fsSL https://PROJECT.interlock.sh/assets/install-model-host.sh | sh -s -- \
  --url https://PROJECT.interlock.sh --group my-model --role trainer \
  --enable-mlx-gguf-deriver
unset INTERLOCK_MODEL_KEY

The flag is macOS-trainer-only and disabled by default. It installs exact MLX, Python and llama.cpp versions, computes a build identity from the complete converter toolchain, and advertises only that identity. Nothing runs until an explicit matching derivation is created. The trainer performs the offline fusion/conversion; an NVIDIA client only consumes the resulting immutable GGUF after its own release and gates. Derivation never changes an existing version, deployment, alias or channel.

A Python procedure declares its exact interpreter and package versions in its immutable dependency lock. On first use, the worker creates a content-keyed virtual environment under the model group's Nightshift directory, installs the hash-verified public Interlock Python SDK served by the same control plane, and reuses that environment only for an identical lock + SDK hash. macOS trainers use uv to resolve or install the exact pinned Python patch release rather than silently accepting a different Homebrew interpreter. Third-party Python packages currently come from their standard package index at this first-use boundary; datasets, procedures, SDK code, models, checkpoints, and releases come from Interlock. Exact Python binaries come from uv's managed runtime distribution. This distinction is intentional and visible—Interlock does not claim to mirror Python or PyPI today.

Do not pass a reusable key on the command line unless necessary. --client-key and --trainer-key exist for automation, but command arguments can remain in shell history and process listings.

What one bootstrap installs #

On macOS, the same command installs the worker LaunchDaemons and the compiled Nightshift menu-bar app. Nightshift is Interlock's local fleet companion: it discovers every model group enrolled on that Mac and lets the owner make each inference or training role always available, available only from 10 PM to 7 AM local time, or paused. It is a local availability ceiling; production versions, placements, and releases remain controlled by Interlock. Nightshift and its root helper are downloaded from the same Interlock origin, SHA-256 verified, and upgraded with the worker. There is no AskIdeal-specific app or source checkout.

Enrolled Macs do not compile Nightshift and need no Xcode or Swift toolchain. Interlock publishes a precompiled universal arm64+x86_64 app through visible assets/nightshift-macos-universal.zip and an atomic assets/nightshift-macos.json release pointer. The worker checks that manifest every 20 minutes, downloads the ZIP with its release hash as the cache key, verifies the exact SHA-256, and replaces Nightshift only when the bytes change. Nightshift releases are therefore independent of model versions and worker-harness releases.

One Mac may enroll several groups, including groups belonging to different Interlock projects. Each enrollment keeps its own origin and narrow machine credential. Runtime state lives under:

~/.nightshift/groups/<modelId>/
├── group.json
├── roles/<client|trainer>/
└── versions/<deploymentId>/

On Linux, the installer creates systemd services and a timer without the macOS UI. On both systems the worker:

• starts before login and restarts after failure;
• holds an outbound SSE connection for low-latency wakeups, with polling as the recovery path;
• stores its machine key in an owner-only environment file, never in a service definition;
• downloads the generic Task Client from the Interlock control plane;
• verifies its SHA-256 against /model-worker/manifest.json before an atomic replacement;
• checks for a new harness every 20 minutes and restarts the worker only after verification;
• downloads model and procedure artifacts from Interlock, not GitHub or a developer laptop.

Re-running the same origin, group, and role is idempotent: the install resolves its stable model id, retains its machine credential and local schedule, and does not exchange the group bootstrap key again.

Uninstall Nightshift from a Mac #

Choose Uninstall Nightshift… at the bottom of the Nightshift menu. The native confirmation is a whole-machine action: it stops every registered Interlock model worker on that Mac and removes their launchd jobs, local group versions, machine credentials, Nightshift services, helper, and app.

It does not delete shared model caches, source checkouts, unrelated services or databases, or any model, dataset, evaluation, training run, release, or history stored in Interlock. The host becomes offline in Interlock; revoke its machine credential from the Models fleet when the physical machine is no longer trusted. Installing again requires a fresh client or trainer bootstrap key.

Operate the fleet #

Use the Models fleet page to pause or resume a worker while keeping its durable installation. A paused inference host drains its model processes; resuming reconciles the current assignments.

interlock models status --dir my-model
interlock models key list --dir my-model

macOS service state is visible in Nightshift or with sudo launchctl print system/sh.interlock.model-worker.<modelId>.client. Linux state is visible with systemctl status sh.interlock.model-worker.my-model.client.

Security boundary #

Commodity hardware is untrusted infrastructure. The server therefore derives group and role from the exchanged credential, not from the capabilities a worker reports. Training placement, evaluation placement, deployment reconciliation, artifact downloads, claims, reports, and SSE all verify the exact machine identity. A compromised box can interrupt work assigned to that one box; it cannot expand its own authority.