Ring:Using Tracing

ring:using-tracing

When to use

Sweep mode:

• "Sweep / audit tracing setup"
• "Find raw OpenTelemetry usage we should replace"
• "Are our HTTP/gRPC boundaries propagating trace context?"
• "Is there DIY field redaction in spans?"
• "Migrate this service to lib-observability/tracing"

Reference mode:

• "How do I bootstrap Telemetry for a new service?"
• "Which Inject/Extract helper do I use for X transport?"
• "How does the Redactor pipeline work?"
• "How do I record a business-error event on a span?"
• "What does RedactingAttrBagSpanProcessor do?"

Skip when

• Working on non-Go services
• Working on frontend code
• Target codebase does not depend on github.com/LerianStudio/lib-observability

Related

Parent: ring:using-lib-observability Similar: ring:using-runtime, ring:using-assert, ring:using-lib-commons

The tracing subpackage owns OTEL provider lifecycle, trace context propagation, span helpers, and the attribute-redaction pipeline (RedactingAttrBagSpanProcessor is wired automatically inside NewTelemetry). Use this skill when tracing is the primary concern. For broader lib-observability sweeps (logging, metrics, panic recovery), invoke ring:using-lib-observability.

Mode Selection

SWEEP MODE

5-phase sweep. Each phase has a hard gate — do not proceed until the current phase produces its artifact.

Copy code
Phase 1: Version Reconnaissance   → tracing-version-report.json
Phase 2: CHANGELOG Delta Analysis → tracing-delta-report.json
Phase 3: Multi-Angle DIY Sweep    → 6 × tracing-sweep-{N}-{angle}.json
Phase 4: Consolidated Report      → tracing-sweep-report.md + tracing-sweep-tasks.json
Phase 5: Handoff                  → offer ring:running-dev-cycle dispatch

Phase 1: Version Reconnaissance

1. Read go.mod — extract pinned version of github.com/LerianStudio/lib-observability
2. WebFetch https://api.github.com/repos/LerianStudio/lib-observability/releases/latest — extract tag_name
3. Classify drift: up-to-date / minor-drift / moderate-drift / major-upgrade / not-imported
4. Emit /tmp/tracing-version-report.json: {pinned_version, latest_version, drift_classification, module_path}

Phase 2: CHANGELOG Delta Analysis

1. WebFetch https://raw.githubusercontent.com/LerianStudio/lib-observability/main/CHANGELOG.md
2. Filter entries affecting tracing/ (otel.go, obfuscation.go, processor.go)
3. Emit /tmp/tracing-delta-report.json with classified entries (new-api / breaking-change / security-fix / bugfix)

Phase 3: Multi-Angle DIY Sweep

⛔ STOP-CHECK BEFORE DISPATCH

Before emitting any Task call, count the explorers you intend to launch in this turn.

• Count MUST equal 6.
• If count < 6 → STOP. Do not partial-dispatch. Reconcile against the 6 angles below and try again.
• The 6 angles are the canonical sweep. No substitutions, no omissions.

⛔ MUST NOT trickle-dispatch

All 6 explorers leave in the SAME TURN, before reading any explorer output.

Forbidden sequences:

• Dispatch explorer 1 → read result → dispatch explorer 2
• Dispatch a subset → wait → dispatch the rest
• Dispatch follow-up explorers conditioned on partial output
• Loop sequentially over the angle list

If you find yourself about to dispatch an explorer in a turn AFTER any explorer has already returned a result → STOP. You violated parallel dispatch. Report the violation and mark the phase INCOMPLETE rather than completing the trickle.

Self-verify after dispatch

After the dispatch turn, verify all 6 Task calls were emitted in that single turn. If fewer than 6 went out, the phase did NOT execute correctly. Mark INCOMPLETE and surface the dispatch failure — do NOT silently continue with a partial pool.

Parallel dispatch — atomic batch

Emit all 6 Task calls in a SINGLE TURN, as one atomic batch.

If your runtime exposes a multi_tool_use.parallel wrapper, use it to dispatch the complete pool in one wrapped invocation. This is the canonical fan-out mechanism on OpenAI-style tool envelopes and on certain Anthropic SDK consumers — naming it explicitly activates parallel emission on runtimes where trickle-dispatch is the default behavior.

If your runtime emits parallel tool_use blocks natively (Claude Code with Claude models), multi_tool_use.parallel may not be needed — but naming it is harmless and serves as an enforcement anchor.

The STOP-CHECK, anti-trickle, and self-verify guards above remain binding regardless of which mechanism your runtime uses.

Dispatch all 6 explorer angles in one parallel batch. Wait for all before Phase 4.

Per-explorer dispatch (subagent_type: ring:codebase-explorer):

Copy code
## Target: <absolute path to target repo root>
## Your Angle: <angle number + name from below>
## Severity / DIY Patterns / Replacement / Migration Complexity
<verbatim from angle table below>

## Output
Write to: /tmp/tracing-sweep-{N}-{angle-slug}.json
Schema: { angle_number, angle_name, severity, migration_complexity,
  findings: [{file, line, diy_pattern, replacement, evidence_snippet, notes}],
  summary }
If no findings: write file with empty findings array.

The 6 Angles

Severity calibration

• CRITICAL: hides production failures or leaks PII (angles 1, 5)
• HIGH: breaks distributed traces or omits whole subsystems (angles 2, 3, 6)
• MEDIUM: duplicates library code; loud but correctable later (angle 4)

Phase 4: Consolidated Report

Dispatch synthesizer (subagent_type: ring:codebase-explorer):

Copy code
Read /tmp/tracing-version-report.json, /tmp/tracing-delta-report.json,
and /tmp/tracing-sweep-*.json (6 files).

Emit:
1. /tmp/tracing-sweep-report.md — findings grouped by severity, cross-referenced to angle table
2. /tmp/tracing-sweep-tasks.json — one task per DIY pattern cluster (same file/package = one task)

MUST NOT invent findings. MUST NOT omit explorer findings. MUST NOT reclassify severity without justification.

Phase 5: Handoff

Surface report path + task count to the caller. Offer handoff to ring:running-dev-cycle for execution.

REFERENCE MODE

Import path: github.com/LerianStudio/lib-observability/tracing

The package owns four concerns:

1. Lifecycle — build OTEL providers and shut them down cleanly
2. Propagation — move trace context across HTTP, gRPC, and message queues
3. Span helpers — record errors, events, and struct-derived attributes
4. Redaction — strip sensitive fields before they reach the collector

1. Lifecycle: Telemetry and TelemetryConfig

goCopy code
type TelemetryConfig struct {
    LibraryName               string
    ServiceName               string
    ServiceVersion            string
    DeploymentEnv             string                            // "production" | "staging" | "development" | "local"
    CollectorExporterEndpoint string                            // "otel-collector:4317" — scheme stripped automatically
    EnableTelemetry           bool
    InsecureExporter          bool                              // forbidden in production unless ALLOW_INSECURE_OTEL is set
    Logger                    log.Logger                        // required; nil returns ErrNilTelemetryLogger
    Propagator                propagation.TextMapPropagator     // defaults to TraceContext + Baggage composite
    Redactor                  *Redactor                         // defaults to NewDefaultRedactor()
}

type Telemetry struct {
    TelemetryConfig
    TracerProvider *sdktrace.TracerProvider
    MeterProvider  *sdkmetric.MeterProvider
    LoggerProvider *sdklog.LoggerProvider
    MetricsFactory *metrics.MetricsFactory
    // shutdown handlers are unexported; use ShutdownTelemetry* methods
}

func NewTelemetry(cfg TelemetryConfig) (*Telemetry, error)
func (tl *Telemetry) ApplyGlobals() error
func (tl *Telemetry) Tracer(name string) (trace.Tracer, error)
func (tl *Telemetry) Meter(name string) (metric.Meter, error)
func (tl *Telemetry) ShutdownTelemetry()
func (tl *Telemetry) ShutdownTelemetryWithContext(ctx context.Context) error

Bootstrap pattern

goCopy code
tl, err := tracing.NewTelemetry(tracing.TelemetryConfig{
    LibraryName:               "my-service",
    ServiceName:               cfg.ServiceName,
    ServiceVersion:            cfg.Version,
    DeploymentEnv:             cfg.Env,
    CollectorExporterEndpoint: cfg.OTELEndpoint,   // "otel-collector:4317" or "http://..."
    EnableTelemetry:           cfg.OTELEnabled,
    Logger:                    logger,
})
if err != nil {
    return fmt.Errorf("init telemetry: %w", err)
}
if err := tl.ApplyGlobals(); err != nil {
    return fmt.Errorf("apply globals: %w", err)
}
defer func() {
    if err := tl.ShutdownTelemetryWithContext(context.Background()); err != nil {
        logger.Log(ctx, log.LevelError, "telemetry shutdown failed", log.Err(err))
    }
}()

Sentinel errors

Endpoint and security rules

• http://host:4317 → scheme stripped, InsecureExporter forced to true
• https://host:4317 → scheme stripped, secure exporter
• host:4317 (no scheme) → treated as insecure (common in cluster-internal traffic)
• InsecureExporter=true in production/prod env aborts with an error unless ALLOW_INSECURE_OTEL is set with a justification — do not bypass this lightly

Disabled / empty-endpoint fallback

When EnableTelemetry=false or the endpoint is empty, NewTelemetry installs no-op providers and ApplyGlobals ensures downstream libraries (e.g. otelfiber) do not spawn real gRPC exporters that leak goroutines. Code paths above the API stay unchanged.

2. Propagation Helpers

All helpers are nil-safe and use the globally configured TextMapPropagator.

HTTP

goCopy code
func InjectHTTPContext(ctx context.Context, headers http.Header)
func ExtractHTTPContext(ctx context.Context, c *fiber.Ctx) context.Context
func InjectTraceContext(ctx context.Context, carrier propagation.TextMapCarrier)
func ExtractTraceContext(ctx context.Context, carrier propagation.TextMapCarrier) context.Context

Outbound client:

goCopy code
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, url, body)
tracing.InjectHTTPContext(ctx, req.Header)
resp, err := httpClient.Do(req)

Inbound Fiber handler:

goCopy code
func handler(c *fiber.Ctx) error {
    ctx := tracing.ExtractHTTPContext(c.UserContext(), c)
    c.SetUserContext(ctx)
    // ...
}

For non-Fiber HTTP servers, build a propagation.HeaderCarrier from r.Header and call ExtractTraceContext.

gRPC

goCopy code
func InjectGRPCContext(ctx context.Context, md metadata.MD) metadata.MD
func ExtractGRPCContext(ctx context.Context, md metadata.MD) context.Context

The gRPC helpers normalize Traceparent/Tracestate header casing — gRPC metadata is lowercase by spec, but some interceptors emit Pascal case. The helpers translate both ways. Do not reinvent this.

Message queues (AMQP / Kafka / generic string-headers)

goCopy code
func InjectQueueTraceContext(ctx context.Context) map[string]string
func ExtractQueueTraceContext(ctx context.Context, headers map[string]string) context.Context
func PrepareQueueHeaders(ctx context.Context, baseHeaders map[string]any) map[string]any
func InjectTraceHeadersIntoQueue(ctx context.Context, headers *map[string]any)
func ExtractTraceContextFromQueueHeaders(baseCtx context.Context, amqpHeaders map[string]any) context.Context

Publisher (RabbitMQ AMQP-style):

goCopy code
headers := tracing.PrepareQueueHeaders(ctx, map[string]any{"x-event-type": "user.created"})
err := channel.Publish(exchange, key, false, false, amqp.Publishing{Headers: headers, Body: payload})

Consumer:

goCopy code
ctx = tracing.ExtractTraceContextFromQueueHeaders(ctx, delivery.Headers)
ctx, span := tracer.Start(ctx, "consume.user.created")
defer span.End()

Reading IDs out of context

goCopy code
traceID := tracing.GetTraceIDFromContext(ctx)     // "" if no valid span
state   := tracing.GetTraceStateFromContext(ctx)

Use these for log correlation. Never parse traceparent headers by hand to recover the trace ID.

3. Span Helpers

goCopy code
func HandleSpanError(span trace.Span, message string, err error)
func HandleSpanBusinessErrorEvent(span trace.Span, eventName string, err error)
func HandleSpanEvent(span trace.Span, eventName string, attributes ...attribute.KeyValue)
func SetSpanAttributesFromValue(span trace.Span, prefix string, value any, r *Redactor) error
func BuildAttributesFromValue(prefix string, value any, r *Redactor) ([]attribute.KeyValue, error)
func SetSpanAttributeForParam(c *fiber.Ctx, param, value, entityName string)

All helpers are nil-safe on the span argument (untyped nil and interface-wrapped typed nil both handled). Error messages are sanitized: bearer/basic tokens stripped and the message is truncated to 1024 bytes with valid-UTF-8 enforcement.

Idiomatic span lifecycle:

goCopy code
ctx, span := tracer.Start(ctx, "ledger.PostTransaction")
defer span.End()

if err := tracing.SetSpanAttributesFromValue(span, "request", req, tl.Redactor); err != nil {
    tracing.HandleSpanEvent(span, "attr.serialize.failed", attribute.String("error", err.Error()))
}

result, err := svc.Post(ctx, req)
if errors.Is(err, ErrInsufficientBalance) {
    tracing.HandleSpanBusinessErrorEvent(span, "business.balance_insufficient", err)
    return err
}
if err != nil {
    tracing.HandleSpanError(span, "post transaction", err)
    return err
}

4. Redaction

goCopy code
type RedactionAction string  // "mask" | "hash" | "drop"

type RedactionRule struct {
    FieldPattern string  // regex matched against field name
    PathPattern  string  // regex matched against dotted path
    Action       RedactionAction
}

type Redactor struct{ /* unexported */ }

func NewDefaultRedactor() *Redactor                            // default sensitive-field list, action = mask
func NewAlwaysMaskRedactor() *Redactor                         // fail-safe; masks every field
func NewRedactor(rules []RedactionRule, mask string) (*Redactor, error)
func ObfuscateStruct(value any, r *Redactor) (any, error)

Pipeline

1. NewTelemetry defaults cfg.Redactor to NewDefaultRedactor() if nil
2. The tracer provider is built with RedactingAttrBagSpanProcessor{Redactor: cfg.Redactor} — every span gets request-scoped attributes from observability.AttributesFromContext filtered through redaction
3. SetSpanAttributesFromValue invokes ObfuscateStruct before flattening, so struct-derived attributes inherit the same rules

Custom rules

goCopy code
r, err := tracing.NewRedactor([]tracing.RedactionRule{
    {FieldPattern: `(?i)^card_number$`, Action: tracing.RedactionDrop},
    {FieldPattern: `(?i)^email$`,        Action: tracing.RedactionHash},
    {PathPattern:  `^request\.headers\.authorization$`, Action: tracing.RedactionMask},
}, "[REDACTED]")
if err != nil {
    return err
}
cfg.Redactor = r

• mask replaces the value with the configured mask string
• hash produces sha256:<hex> using a per-instance HMAC key (so identical inputs in different processes produce different hashes — anti-rainbow-table)
• drop removes the field entirely

RedactingAttrBagSpanProcessor

Custom sdktrace.SpanProcessor wired automatically inside NewTelemetry. It copies observability.AttributesFromContext(ctx) onto every started span and applies redaction by attribute key. Do not register it manually unless you are bypassing NewTelemetry.

5. Common Anti-Patterns

6. Cross-References

• [[using-lib-observability]] — parent skill covering logging, metrics, panic recovery
• [[using-runtime]] — panic observability trident; uses tracing's span events for the panic record
• [[using-assert]] — production assertions; their AssertionError surfaces in spans via HandleSpanError
• [[using-lib-commons]] — broader Lerian shared-library sweep; Angle "observability DIY" delegates here when tracing-specific