Graphsignal Profiler

Graphsignal Profiler

Graphsignal observes inference workloads from a sidecar process — the profiler. It never shares a process with CUDA: the profiler watches the workload externally via CUPTI, OTLP/gRPC, Prometheus scraping, and NVML. Auto-instrumentation covers vLLM, SGLang, and PyTorch out of the box.

Install

Two install patterns depending on how you'll launch the profiler.

For graphsignal-run (CLI, recommended): install as a uv tool, isolated from your workload env.

bashCopy code
UV_TOOL_BIN_DIR=/usr/local/bin uv tool install 'graphsignal[cu12]'   # CUDA 12.x
# or
UV_TOOL_BIN_DIR=/usr/local/bin uv tool install 'graphsignal[cu13]'   # CUDA 13.x

UV_TOOL_BIN_DIR=/usr/local/bin puts graphsignal-run in a directory that is already on PATH for every shell, including non-interactive scripts and containers.

For graphsignal.watch() (in-process Python entry point): install into the app's own env.

bashCopy code
uv add 'graphsignal[cu12]'    # or pip install -U 'graphsignal[cu12]'

The cu12 / cu13 extras are Linux-only and only needed for GPU profiling.

Configure

The profiler reads its config from environment variables.

Set these before invoking graphsignal-run or calling graphsignal.watch().

Run

Option A — graphsignal-run CLI (recommended)

Wrap the launch command for your workload.

bashCopy code
export GRAPHSIGNAL_API_KEY="..."
graphsignal-run vllm serve Qwen/Qwen1.5-7B-Chat --port 8000

The CLI enables GPU profiling, starts the profiler sidecar, and launches the workload.

Option B — graphsignal.watch() from Python

For applications that bootstrap themselves (long-lived servers, scripts, notebooks), call graphsignal.watch() once during startup, before any CUDA work happens.

pythonCopy code
import graphsignal

graphsignal.watch()
# ... your application code (PyTorch, vLLM, SGLang, etc.) ...

It enables GPU profiling in this process and starts the profiler sidecar to observe it. Returns the subprocess.Popen so the caller can wait() or terminate() it.

OpenTelemetry tracing (opt-in)

Distributed traces (engine / scheduler / attention spans over OTLP/gRPC) are off by default. Enable them with --enable-otel, which must come before the workload command:

bashCopy code
graphsignal-run --enable-otel sglang serve --model-path Qwen/Qwen1.5-7B-Chat --port 8000

This captures the engine's request traces via a local OTLP collector. It requires OpenTelemetry installed in the engine's environment (e.g. pip install opentelemetry-sdk opentelemetry-exporter-otlp) — graphsignal can't provide it when installed in a separate env (e.g. uv tool), and SGLang ≥ 0.5.10 errors at startup if tracing is enabled without it. Prometheus metrics and GPU profiling are captured regardless of this flag; OTEL tracing applies only to graphsignal-run (not graphsignal.watch()).

Engine-specific notes

vLLM

bashCopy code
export GRAPHSIGNAL_API_KEY="..."
graphsignal-run vllm serve Qwen/Qwen1.5-7B-Chat --port 8000

Or from Python (before importing vLLM):

pythonCopy code
import graphsignal
graphsignal.watch()
import vllm
# ...

Captures vLLM's Prometheus metrics and CUPTI GPU profiling out of the box. Engine / scheduler / KV-cache / attention / output-processing OTEL spans are added with --enable-otel (see OpenTelemetry tracing above).

vLLM Docker (image without CUPTI):

bashCopy code
docker run --gpus all \
  -p 8000:8000 --ipc=host \
  -e GRAPHSIGNAL_API_KEY=YOUR_API_KEY \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint bash \
  vllm/vllm-openai:latest \
  -lc 'pip install --no-cache-dir graphsignal[cu12] \
       && exec graphsignal-run vllm serve \
           --model Qwen/Qwen2-VL-7B-Instruct \
           --trust-remote-code'

SGLang

bashCopy code
export GRAPHSIGNAL_API_KEY="..."
graphsignal-run sglang serve \
  --model-path Qwen/Qwen1.5-7B-Chat \
  --port 8000

Captures SGLang's Prometheus metrics and operation-level GPU profiling out of the box. OTEL spans are added with --enable-otel (see OpenTelemetry tracing above).

SGLang Docker:

bashCopy code
docker run --gpus all \
  -p 8000:8000 --ipc=host \
  -e GRAPHSIGNAL_API_KEY=YOUR_API_KEY \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint bash \
  your-sglang-image:latest \
  -lc 'pip install --no-cache-dir graphsignal[cu12] \
       && exec graphsignal-run sglang serve \
           --model-path Qwen/Qwen2.5-1.5B-Instruct \
           --port 8000'

PyTorch

Auto-instrumented for common PyTorch operator / module hot paths and CUDA memory metrics. Use either CLI or watch():

bashCopy code
export GRAPHSIGNAL_API_KEY="..."
graphsignal-run python my_app.py

pythonCopy code
import graphsignal
graphsignal.watch()
import torch
# ...

dstack

dstack runs inference as services. Use graphsignal-run around the launch command, exactly like bare-metal SGLang/vLLM.

yamlCopy code
type: service
name: deepseek-r1

image: lmsysorg/sglang:latest
env:
  - MODEL_ID=deepseek-ai/DeepSeek-R1-Distill-Llama-8B
  - GRAPHSIGNAL_API_KEY

commands:
  - |
    pip install --no-cache-dir 'graphsignal[cu12]' && \
    graphsignal-run python3 -m sglang.launch_server \
      --model-path $MODEL_ID \
      --port 8000 \
      --trust-remote-code

port: 8000
model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B

resources:
  gpu: 24GB

Deploy:

bashCopy code
dstack apply -f service.dstack.yml

Troubleshooting

Ensure outgoing connections to https://api.graphsignal.com are allowed.

Reference

• Full Profiler API: https://graphsignal.com/docs/reference/profiler-api/
• vLLM integration: https://graphsignal.com/docs/integrations/vllm/
• SGLang integration: https://graphsignal.com/docs/integrations/sglang/
• dstack integration: https://graphsignal.com/docs/integrations/dstack/