Self-hosted AI workers #

Raw markdown: /self-hosted-ai.md

The promise #

Your Mac or Linux box runs the model. It opens outbound HTTPS/SSE to Interlock, claims only work for its project, and calls a model server bound to loopback. Interlock never opens a connection to your hardware or model port. Interlock does receive the ordinary prompt and result because it is the durable control plane; “self-hosted” means you own compute and uptime, not control-plane-blind end-to-end encryption.

Qwen3.6-35B-A3B Q8_0 is the first supported profile. Expect about 38 GB for model weights and a 64 GB Apple Silicon Mac or a Linux GPU host with comparable usable memory. Model source is unsloth/Qwen3.6-35B-A3B-GGUF; review its Apache-2.0 license and the upstream Qwen license before installing. Quantisation changes quality and memory use, so the deployment alias includes the exact profile rather than the vague word “Qwen”.

Install #

In Project settings, open Your hardware and choose Create install command. Run the one-time command on the machine. The key is shown once and has only tasks:register, tasks:claim, and tasks:report; it cannot submit application work or read another project.

The installer:

• supports macOS with a system LaunchDaemon and Linux with a systemd service, both running as the unprivileged installing user and starting without login;
• verifies a pinned public-key signature on both the release manifest and jar, then verifies the manifest SHA-256 and byte count before an atomic upgrade;
• stores the key in an owner-only environment file, never in a service definition or command line;
• binds/adopts llama.cpp on 127.0.0.1; non-loopback configuration is refused unless the operator deliberately sets LLM_ALLOW_REMOTE=1;
• keeps the previous jar for rollback and preserves secrets, logs, and models on uninstall.

Lifecycle commands use the same downloaded installer:

bash setup.sh status
bash setup.sh doctor
bash setup.sh update
bash setup.sh restart
bash setup.sh uninstall   # service/app only; env, logs and model cache remain

The host needs Java 21, curl, openssl, Python 3, and llama.cpp's llama-server. It needs outbound TCP 443 to api.interlock.sh and the asset host, plus outbound model download access on first boot. It needs no inbound firewall rule, public IP, tunnel, SSH exposure, port 8080, or port 11434.

Route policy #

Placement is explicit:

Java:

var options = Ai.ChatOptions.withManagedFallback(Models.QWEN_3_6_35B_A3B);
String answer = il.ai().chatWithOptions(options, "Summarise this session");

AiRun run = il.ai().submit(Ai.ChatOptions.selfHosted(Models.QWEN_3_6_35B_A3B), prompt);
AiRun.Snapshot done = run.await(Duration.ofMinutes(10));

Node:

const run = await client.aiRunSubmit(prompt, {
  model: 'qwen3-6-35b-a3b', routePolicy: 'self-hosted-only', timeoutMs: 600000,
});
const done = await client.aiRunAwait(run.id, { timeoutMs: 660000 });

HTTP uses POST /api/ai/runs, GET /api/ai/runs/{id}, and POST /api/ai/runs/{id}/cancel. /api/ai/runs/{id}/stream immediately sends an SSE hello, continues with durable task-event deltas and 25-second heartbeats, and may be disconnected at any time: GET by run ID remains canonical. A long inference is not coupled to a Cloudflare request deadline.

Operate it #

“Ready” means the worker registered ai/chat@1, was seen within 90 seconds, is not draining, and has capacity. “Full” is healthy—all slots are busy. “Drained” renews current leases and claims no new work. “Offline” means Interlock has not seen it recently; do not confuse that with a model that is still loading locally.

Before maintenance, drain and wait for running work to reach zero. Update, run doctor, then restart. For key rotation, create a new Task Worker key, replace it in the owner-only env, restart, prove the worker ready, then revoke the old key. For a model replacement, use a new deployment alias until readiness and a real completion pass; do not relabel different weights as the old deployment.

The billed route is ai.self-hosted.qwen3.6-35b-a3b. Evidence includes the task/run, worker, deployment, attempt, token counts, duration, terminal state, and route policy. An eligibility or admission refusal costs zero. One inference produces one AI charge; the underlying Task does not produce a second generic charge.

Diagnose and roll back #

1. Run setup.sh doctor: release signature/checksum, environment permissions, loopback model boundary, and boot service must all pass.
2. In Your hardware, distinguish offline, drained, full, and ready. Open /tasks for the exact ai/chat@1 attempt and event timeline.
3. Check local logs under ~/.interlock/task-worker/logs. Prompts/results are customer data; do not paste them into tickets. Worker logs redact them by default.
4. If the model is unavailable, self-hosted-only should return NO_ELIGIBLE_INFERENCE_WORKER, not silently spend money. Use managed fallback only when its cost and privacy boundary are acceptable.
5. Roll back an upgrade by stopping the service, replacing app/worker.jar with app/worker.prev, and starting it. uninstall removes the service/application without deleting the environment, logs, or model cache.