Runpod

RunPod Fast Provisioning

Rent a GPU that is ready in seconds, not minutes. The whole method is one rule: everything installable goes into a prebaked container image, so the pod's only job at boot is to pull that image, start sshd, and run the workload. Pod-side apt install, pip install, and source compilation are all billed minutes that buy nothing.

When to Use

• Work that is impossible or impractically slow on local hardware: needs more VRAM than the workstation has, needs CUDA specifically, needs N GPUs on one host.
• Benchmark grids, fine-tuning and distillation runs, short-lived model serving.
• Any repeat rental: the second pod is where a prebaked image pays for itself.

Do not rent for work that fits locally. Write the case for the pod first (what is locally impossible, what decides when it is done), and give the workload a hard stop before launching it.

The core rule: prebake, never install on the pod

Measured 2026-08-12 on RunPod secure cloud; "verified" means pod created, RUNNING, ssh accepted, and an on-pod check of GPU count, binary version, and the prebaked virtual environment all passed. Related measurements from the same session:

• A 17 GB model file pulled from HuggingFace on the pod in 34.8 s (about 490 MB/s) with hf_transfer preinstalled and enabled. Datacenter network beats any local upload.
• 2x A40 at $0.88/hr total served a 30B model layer-split across both cards (8.2 + 8.6 GiB resident) at 31 tok/s decode.

Always report the GPU type, GPU count, hourly price, and the date next to any number like these. Prices and stock move, and a throughput number without its machine spec is not a result.

Why RunPod over a managed serverless platform

Compared to managed alternatives such as Modal, RunPod runs roughly half the price for the same card, needs minimal prep work, exposes a usable REST and GraphQL API, and sits on a fast datacenter network. The trade-off is that the image, the lifecycle, and the cost discipline are yours to manage, which is exactly what the templates in this skill do.

Workflow

Step 1: Pick the GPU and confirm it is in stock

Availability is per (GPU type, GPU count): a type with stock at 1x may have none at 2x. Query before assuming, because a create call against an unavailable type fails with "There are no instances currently available".

bashCopy code
curl -s https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" -H 'Content-Type: application/json' \
  -d '{"query":"{ gpuTypes { id securePrice lowestPrice(input:{gpuCount:1, secureCloud:true}) { stockStatus uninterruptablePrice } } }"}'

The REST API has no gputypes path; stock and pricing come from GraphQL only. GraphQL returns HTTP 200 even for query errors, so check the body for an errors key rather than trusting the status code. For the pricing ladder, the aggregate-VRAM-per-dollar comparison, and the scale ladder from one card to a multi-node cluster, see references/gpu-selection.md.

Step 2: Build the prebaked image

Start from templates/Dockerfile. Five rules make it fast and stable:

1. Match the base image glibc to any prebuilt binaries copied in. Binaries built on Ubuntu 24.04 will not run on a 22.04 base.
2. Never compile, in the image or on the pod. COPY --from a digest-pinned official prebuilt image instead. Compiling CUDA code under buildx emulation on Apple Silicon is painfully slow; copying binaries is instant.
3. Prebake the dependency environment from the lockfile (uv sync --frozen --no-install-project --no-dev), so the post-sync install on the pod is a warm-cache no-op. Never let a tool's own requirements.txt drag in a second deep-learning framework install; that has filled a 60 GB disk before.
4. Preinstall hf_transfer, set HF_HUB_ENABLE_HF_TRANSFER=1, and point HF_HOME at the container volume so weight pulls run at datacenter speed and survive into the workspace.
5. Publish environment for ssh, not just for Docker. sshd spawns clean login shells that do not inherit Dockerfile ENV, so register library paths with ldconfig and write the variables to /etc/environment.

Validate locally before pushing. This gate costs nothing on a laptop and would otherwise be paid in billed pod-debugging minutes:

bashCopy code
docker buildx build --platform linux/amd64 -f Dockerfile -t "$IMAGE" --push .
docker run --platform linux/amd64 --rm "$IMAGE" <the-binary> --version   # runtime linkage check
docker run --platform linux/amd64 --rm "$IMAGE" ls /opt/<env-dir>/.venv  # prebaked env present

Step 3: Create the pod

templates/pod-up.sh creates the pod and prints a timestamped marker for every stage, exiting nonzero on the first failure. Four fields in the create call matter:

• env: {"PUBLIC_KEY": "<contents of the local .pub key>"}. Pods created through the REST API get no auto-injected ssh key; RunPod only injects account keys into its own templates.
• allowedCudaVersions: ["12.6","12.7","12.8","12.9"], pinning away from a host pool with a known container-start bug (see the pitfall catalog).
• containerDiskInGb sized for two copies of the model plus caches.
• gpuCount: N for a single-host multi-GPU pod. Same call, same pod, N cards.

Step 4: Sync the project and launch the job detached

templates/pod-run.sh has three modes: launch, status, fetch. Long jobs run detached under tmux with an explicit exit-code marker line, so the local session can disconnect without killing the run. pod-up.sh ends by printing the export POD_HOST=... POD_PORT=... line that pod-run.sh requires; export it before any mode.

Two non-obvious flags carry real scar tissue: rsync into a container fails chown (exit code 23) and kills a set -e script even though every file transferred, so always pass --no-owner --no-group. And every launch must be marker-checked, never fire-and-forget: echo an explicit STAGE / READY / FAILED marker and verify it. A launch that silently no-opped on a renamed path once burned 17 idle billed minutes.

Never suppress stderr on an orchestration path. A 2>/dev/null on a launch ssh hid a self-killing pkill for half an hour, disguised as network flakiness (pitfall 17). Capture per-pod logs instead.

Step 5: Fetch results, then terminate immediately

Copy results back to the local machine with scp or rsync. Do not stage them through a model hub; free-plan private storage caps have failed uploads mid-chain. Then templates/pod-down.sh, which reads the pod id recorded by pod-up.sh.

Terminating is part of finishing the job, not a separate cleanup task. Full detached-run, marker, and cost protocol: references/job-execution.md.

Fanning out and scaling

An evaluation grid or batch sweep is embarrassingly parallel: N independent single-GPU pods burn the same total GPU-hours as one pod and finish in 1/N of the wall-clock time. Reach for a fleet of independent pods, not a multi-node instant cluster: clusters are distributed-training infrastructure (collective communication, interconnect, launcher config) and the wrong shape for independent cells. For the rare run that genuinely needs a cluster, the scale ladder in references/gpu-selection.md includes the rung-4 pre-flight checklist: provision it as an instant cluster rather than hand-wired pods, prove the fabric with an NCCL all-reduce before the real job, make checkpoint-resume mandatory, and rehearse the full launch at the smallest shape because debugging bills every node at once.

Nine rules carry most of the value; the full method is in references/fanout.md.

1. Slice by estimated duration, not item count. A long-context bin can run 10x a short one, so equal item counts leave most of the fleet idle. Chain several short runs onto one pod (cmd1; cmd2; cmd3 inside one tmux session) and write one results file per slice, merged locally after fetch.
2. Provision what stock allows, then queue. Availability is per (GPU type, GPU count), so create pods until stock runs out and round-robin the unplaced slices onto the pods that did provision, rebalanced by duration. Do not stall a grid waiting for the last pod.
3. Verify one pod end to end before replicating. Session live, log line count advancing between two samples, GPU utilization engaged. Only then copy the launch to the rest of the fleet; an unverified launch multiplies one mistake by the size of the fleet.
4. Launchers are detached retry-with-backoff loops with a verification connection after every attempt. A transient network window can fail every pod in one wave and look exactly like a systematic bug.
5. ssh -n in every loop. Without it ssh eats the loop's stdin and only the first iteration runs, while the rest of the fleet boots and bills doing nothing.
6. Build command chains in Python, not in the shell. zsh does not word-split unquoted variables the way bash does, so a chain assembled from a variable can collapse into one malformed command.
7. Make relaunches idempotent. A retry loop that can fire twice must be safe to fire twice: kill the previous session and any orphaned worker first (with the character-class dodge, so the kill cannot match its own command line). An orphaned worker also holds the GPU busy, turning "GPU engaged" into a false launch-verification signal.
8. Monitor from one detached loop, and calibrate an ETA from measured throughput. End remote content pipelines with ; exit 0 (a zero-match grep otherwise reads as an unreachable pod), force PYTHONUNBUFFERED=1 so logs advance in real time, and convert an early tokens-per-second reading into per-cell ETA so silence can be told apart from a stall. Declare a pod unreachable only when a dedicated probe fails.
9. Reap on completion. A finished pod bills until something terminates it, and a monitor that only reports does not stop the meter. Run a local reaper loop: probe, detect chain-process-gone, fetch results and log, verify locally, and only then delete the pod. Never terminate on a failed fetch or an unreachable probe.

Per-pod startup runs about 4 min when the image is only partially prebaked (dependency sync, model pull, evaluation-dataset pulls, server load) and drops to about 90 s once the datasets and all dependency groups are baked in. That difference is paid once at build time and saved N times across the fleet.

Templates

Every template is parameterized through environment variables (RUNPOD_API_KEY, POD_IMAGE, POD_SSH_KEY, POD_NAME, POD_HOST, POD_PORT, POD_REMOTE_DIR, POD_JOB_CMD, POD_RESULTS, plus optional POD_TMUX_SESSION and POD_LOCAL_ROOT). Copy them into the project, fill in the job command, and commit them next to the code they run.

Pitfalls, in short

Each of these was hit for real. The full catalog, with symptoms and fixes, is references/pitfalls.md.

Cost discipline

• Write the case for the pod before creating it: what is locally impossible, and what decides that the run is done.
• Give the workload itself a budget or time cap flag as the hard stop, so an overrun ends the work rather than the billing.
• pod-down.sh runs immediately after fetch, in the same breath; for a fleet, a local reaper loop makes that automatic (fetch, verify, then delete on chain completion), capping the idle tail of every finished pod at the loop's poll interval.
• Record the spend where the work is tracked, with GPU type, count, hourly rate, and wall time.

Additional Resources

• Reference: references/pitfalls.md - the full provisioning pitfall catalog with symptoms, causes, fixes, and what each one cost
• Reference: references/gpu-selection.md - stock queries, the pricing ladder, VRAM per dollar, the single-card to multi-node scale ladder, and the cluster pre-flight checklist
• Reference: references/job-execution.md - detached tmux runs, marker discipline, result retrieval, and cost accounting
• Reference: references/fanout.md - fanning a grid across a fleet: duration-based slicing, partial-stock queueing, verify-one-then-replicate, retry-with-backoff launchers, idempotent relaunches, fleet monitoring with ETA calibration, and startup anatomy