Project workers¶
Each project can run on its own worker: a separate service with its own disk for the project's repositories, governing workspace and agent sessions. The public web service has no workspace disk. It forwards a project's work to that project's worker over a private network, so a deploy of the web service does not stop agent work, and projects don't compete for one machine. The design is in specs/014-project-workers; Architecture & scaling shows how the pieces fit, and Deploying to the cloud sets up Railway from scratch.
Three platforms run the same layout:
| Platform | Use it for | The worker is | Its disk |
|---|---|---|---|
| Railway | production | a service spaces-proj-<code> (sleeps when idle) |
a Railway volume |
| Kubernetes | clusters, and minikube for engineers | a Deployment and Service spaces-proj-<code> |
a PersistentVolumeClaim |
| Docker | one machine, for engineers | a container spaces-proj-<code> |
a named volume |
SPACES_ROLE decides what a process runs: all (the default: everything in one service, as before), web, or project. A worker is created the first time its project needs one: when the project is created, or when it is first opened or run. The super admin sees every worker under the account menu → Project workers. That page also has a check (can Spaces reach the platform?) and a probe (may it create what a worker needs?).
Local: Docker¶
You need Docker and about 4 GB of free memory per busy project.
docker compose -f docker-compose.workers.yml up --build
open http://localhost:3000
- Create the first account, then add an LLM key under Organization → Models.
- Create a project. The web service starts its worker:
docker ps --filter label=spaces.dev/role=project-worker
docker logs -f spaces-proj-<code>
- Open the project and start a run. The run happens in the worker container, on its own volume (
spaces-proj-<code>-data).
docker compose … down stops the web service and Postgres but leaves the workers running, because compose didn't create them. To remove them, use Project workers → Remove, or:
docker rm -f $(docker ps -aq --filter label=spaces.dev/role=project-worker)
docker volume rm $(docker volume ls -q --filter label=spaces.dev/role=project-worker)
The web container uses the Docker socket, which gives it root on your machine. Use this setup only for local development.
Kubernetes, and minikube¶
deploy/kubernetes contains:
- the
spacesnamespace - a development Postgres
- the web Deployment with
SPACES_ROLE=web - a ServiceAccount with a Role limited to this namespace: Deployments, Services and PersistentVolumeClaims
The web service uses that account to create each project's worker.
minikube¶
minikube start --cpus 4 --memory 8g --disk-size 40g
eval "$(minikube docker-env)" # build straight into minikube's Docker
docker build --build-arg GIT_COMMIT="$(git rev-parse HEAD)" -t spaces:local .
cp deploy/kubernetes/secrets.env.example deploy/kubernetes/secrets.env
# edit secrets.env: ENCRYPTION_KEY and SPACES_INTERNAL_TOKEN (long random strings)
kubectl apply -k deploy/kubernetes
kubectl -n spaces rollout status deploy/spaces-web
kubectl -n spaces port-forward svc/spaces-web 3000:3000
open http://localhost:3000
After creating a project:
kubectl -n spaces get deploy,svc,pvc -l spaces.dev/role=project-worker
kubectl -n spaces logs -f deploy/spaces-proj-<code>
Workers request 250m CPU and 512 MiB and may use up to 2 vCPU and 2 GiB, so a few fit on a 4-CPU minikube. Change the limits with PROJECT_WORKER_VCPUS, PROJECT_WORKER_MEMORY_GB and PROJECT_WORKER_DISK_GB on the web Deployment. Change the requests with PROJECT_WORKER_CPU_REQUEST and PROJECT_WORKER_MEMORY_REQUEST.
To rebuild after a code change, build again with the new GIT_COMMIT (using minikube's Docker) and restart the web service. It updates each worker when that worker is idle:
docker build --build-arg GIT_COMMIT="$(git rev-parse HEAD)" -t spaces:local .
kubectl -n spaces rollout restart deploy/spaces-web
k3s in Docker (when minikube will not start)¶
minikube's binary crashes on start on some recent macOS versions. k3s runs a single-node cluster in a container, and its kubeconfig goes in its own file:
docker run -d --privileged --name k3s-spaces --tmpfs /run --tmpfs /var/run -p 16443:6443 \
rancher/k3s:v1.31.4-k3s1 server --disable traefik --disable metrics-server --tls-san 127.0.0.1
docker exec k3s-spaces cat /etc/rancher/k3s/k3s.yaml | sed 's#:6443#:16443#' > k3s.kubeconfig
export KUBECONFIG=$PWD/k3s.kubeconfig
docker build --build-arg GIT_COMMIT="$(git rev-parse HEAD)" -t spaces:local .
docker save spaces:local | docker exec -i k3s-spaces ctr -n k8s.io images import -
kubectl apply -k deploy/kubernetes
To try sleeping quickly, set PROJECT_WORKER_IDLE_SECONDS=30 and PROJECT_WORKER_SLEEP_SECONDS=60 on spaces-web. A worker then scales to 0 about a minute and a half after its last job. A request to the project, or queued work, starts it again.
Other clusters¶
- Push the image to a registry the cluster can pull from. Set
PROJECT_WORKER_IMAGEand the web container'simageto it, and setPROJECT_WORKER_IMAGE_PULL_POLICY=Alwaysif the tag moves. - Point
DATABASE_URLinsecrets.envat a managed Postgres 16 with pgvector, and removepostgres.yamlfromkustomization.yaml. - Set
PROJECT_WORKER_STORAGE_CLASSif the default StorageClass is not the one you want for worker disks. - Expose
spaces-webthrough your Ingress with TLS, and setPUBLIC_URLto that address. - To run the web service outside the cluster, give it
KUBE_API_URL,KUBE_TOKEN(a token for thespaces-provisioneraccount) andKUBE_CA_FILE.
Railway (production)¶
- Create a workspace token: Account Settings → Tokens → Create token, for the workspace that holds the Spaces project. A project token isn't enough: it can create services and volumes but not delete them, so removing a worker fails. Set
RAILWAY_PROVISIONER_TOKEN_TYPE=workspace. - Add it to the web service as
RAILWAY_PROVISIONER_TOKEN, and seal it. - Set
SPACES_INTERNAL_TOKENon the web service.
Workers are built from PROJECT_WORKER_REPO (default rayedbajwa/spaces). They get references to the web service's DATABASE_URL, ENCRYPTION_KEY, SPACES_INTERNAL_TOKEN and PUBLIC_URL, so those values stay in one place. They are sized 2 vCPU and 2 GB, with a volume at /data, and sleep when idle. A sleeping worker wakes when the web service sends it a request.
Sleeping, waking and the awake limit¶
A worker's job process exits after PROJECT_WORKER_IDLE_SECONDS (default 300) with nothing to do. That includes runs waiting on a person at a gate: answering restarts the stage on the next worker. The worker's server then makes no network traffic, so Railway puts the service to sleep. Docker keeps the container, but it uses almost no CPU.
On Kubernetes nothing sleeps a pod, so Spaces does it: once a worker has been quiet for PROJECT_WORKER_SLEEP_SECONDS (default 600; 0 turns it off), the web service scales its Deployment to 0 and marks it sleeping. Quiet means no job process running, nothing queued or running, not being moved, and not just started. The PersistentVolumeClaim keeps the files. Anything that needs the worker scales it back to 1 and waits for the pod: queued work, someone opening the project, or an approval. On a local cluster the pod is answering within about ten seconds. A sleeping worker gets new Spaces versions in its pod template without starting, so it wakes on the current version.
Reads that only decorate the board never wake a worker on any platform. They use what the worker stored the last time it was up.
When work is queued for a project, the web service wakes that project's worker, whichever path queued it: a run, an approval link, a Slack answer, or an intake conversion. It catches every job as it's added, and also checks every 30 seconds for queued work whose worker isn't running.
At most PROJECT_WORKERS_MAX_AWAKE (default 5) workers run jobs at once. Work for another project waits until one goes idle. The super admin gets a Slack DM the first time the limit is hit, then at most once an hour while it stays hit. The Project workers page shows how many workers are awake and which projects are waiting.
Updates only when idle¶
The web service runs the version every worker should run: the git commit it was built from. Every minute it updates workers on another version that have nothing running, at most two at a time. A worker that's busy stays on its version until its stage ends. While a worker is being updated it isn't woken; queued work runs on the new version, which starts its job process when it boots. The Project workers page shows each worker's version and when an update is waiting for it to go idle.
One deployment path on every platform: a push to main deploys the web service, and only the web service deploys workers, each with the web service's exact commit once it's idle. A worker never deploys by itself.
-
Railway: the web service deploys on a push to
main(its GitHub trigger). Workers are built from the same repository but have no branch and no deployment trigger. Spaces removes any trigger Railway attaches, when it creates a worker and on each update. Spaces deploys a commit to each worker through the API. A worker Railway has put to sleep isn't deployed just to update it, because that would start it. The rollout leaves it asleep, and when work wakes it, it's deployed on the current version, so the work runs on current code. That first start takes a build, a minute or so with Railway's cache. -
Kubernetes: the worker's Deployment is patched with the image and a
spaces.dev/versionannotation, so its pod is replaced even when the tag (spaces:local) stays the same. - Docker: the container is recreated from the image, on the same volume.
Locally, build with GIT_COMMIT so the web service knows its version. Without it the version is "unknown" and nothing is updated:
GIT_COMMIT="$(git rev-parse HEAD)" docker compose -f docker-compose.workers.yml up --build
Moving existing projects¶
Projects created before workers live on the shared service's disk. Move them one at a time from the account menu → Project workers → Existing projects. The shared service stays in SPACES_ROLE=all throughout: a moved project's requests and jobs go to its worker, and every other project carries on as before.
- Copy (dry run) creates the project's worker, copies the project's files to it (repositories, governing workspace, worktrees, agent sessions), and checks every path has the same files and bytes. The shared service keeps running the project.
- Move holds the project's jobs, copies again (so anything written since the dry run arrives), checks, then hands the project to its worker. A project with a stage running can't be moved: wait for the stage to end.
- Move back copies the worker's files back, checks them, and returns the project to the shared service.
Files keep their paths, since both sides mount /data, so nothing in the database changes.
On Railway, before the first move¶
- On the spaces service, set
SPACES_INTERNAL_TOKEN(a long random string, sealed) andHOST=::. Workers pull the files from the shared service over Railway's private network, which uses IPv6. Railway setsRAILWAY_PRIVATE_DOMAIN, which gives the shared service's address. SetMIGRATION_SOURCE_URLonly to override it. - Check that
RAILWAY_PROVISIONER_TOKENpasses Check the token and Probe create and delete. - Run a Copy (dry run) on one project and check the result, then Move it and use it for a while before moving the others.
Switching over¶
Once every project has moved, set SPACES_ROLE=web on the shared service. Nothing changes for moved projects, but the shared service stops running workers and stops sweeping its disk. Keep its volume for a week as a fallback. To go back, set SPACES_ROLE=all again and Move back the projects that need it.
Faster builds¶
Each Railway worker builds its own image from the repository with the Dockerfile. Railway has no shared registry between services, so there's no building once and pulling everywhere. Instead, the Dockerfile is ordered for Railway's build cache:
- system packages
- dependencies, then Chromium (about 1.7 GB, cached until
bun.lockchanges) - the code
- the commit, last
A worker's first build takes a few minutes. After that, an update rebuilds only the code layers.
The web service runs no agents, so it uses Dockerfile.web: 675 MB instead of 2.5 GB, with no Chromium, Docker CLI or psql. On Railway, set RAILWAY_DOCKERFILE_PATH=Dockerfile.web on the web service only.
Spaces sets each worker's build and start explicitly: dockerfilePath: Dockerfile and startCommand: bun run src/standalone.ts. Railway doesn't apply railway.json to services created through its API, so without these a worker builds with Railpack (no Chromium, psql or gosu) and starts only the server, and its job process never runs.
Settings¶
| Variable | Platform | Meaning |
|---|---|---|
SPACES_ROLE |
all | all, web or project |
SPACES_INTERNAL_TOKEN |
all | Shared secret between the web service and workers |
MIGRATION_SOURCE_URL |
all | The shared service's private address, which workers pull a moved project from (Railway: from RAILWAY_PRIVATE_DOMAIN) |
HOST |
all | Address to listen on; :: on Railway so workers can reach the shared service privately |
PROJECT_WORKER_PLATFORM |
all | railway, kubernetes or docker (Railway is assumed when RAILWAY_PROVISIONER_TOKEN is set) |
PROJECT_WORKER_IMAGE |
Kubernetes, Docker | Image the workers run |
PROJECT_WORKER_VCPUS, PROJECT_WORKER_MEMORY_GB, PROJECT_WORKER_DISK_GB |
all | Worker size (2, 2, 5) |
PROJECT_WORKER_PORT |
all | Port workers listen on (8080) |
PROJECT_WORKER_IDLE_SECONDS |
all | Idle time before a worker's job process stops (300) |
PROJECT_WORKER_SLEEP_SECONDS |
Kubernetes | Quiet time before the web service scales a worker to 0 (600; 0 keeps workers running) |
PROJECT_WORKERS_MAX_AWAKE |
all | Workers allowed to run jobs at once (5) |
PROJECT_WORKER_DATABASE_URL |
Kubernetes, Docker | Database address as the workers see it, if it differs from the web service's |
RAILWAY_PROVISIONER_TOKEN, RAILWAY_PROVISIONER_TOKEN_TYPE |
Railway | Provisioning token, and its type (project by default) |
PROJECT_WORKER_REPO |
Railway | Repository workers are built from (no branch: only Spaces deploys them) |
PROJECT_WORKER_POSTGRES_SERVICE |
Railway | Name of the Postgres service workers reference for DATABASE_URL (Postgres) |
PROJECT_WORKER_NAMESPACE, PROJECT_WORKER_SECRET, PROJECT_WORKER_STORAGE_CLASS, PROJECT_WORKER_IMAGE_PULL_POLICY, PROJECT_WORKER_CPU_REQUEST, PROJECT_WORKER_MEMORY_REQUEST |
Kubernetes | Where workers go and how they are scheduled |
KUBE_API_URL, KUBE_TOKEN, KUBE_CA_FILE, KUBE_INSECURE |
Kubernetes | API access from outside the cluster |
PROJECT_WORKER_DOCKER_NETWORK, DOCKER_SOCKET |
Docker | Network the workers join, and the engine's socket |