Observability Setup

Observability Setup Skill

This skill provides patterns for setting up observability in Nais applications.

Required Health Endpoints

kotlinCopy code
import io.ktor.server.application.*
import io.ktor.server.response.*
import io.ktor.server.routing.*
import io.ktor.http.*

fun Application.configureHealthEndpoints(
    dataSource: HikariDataSource,
    kafkaProducer: KafkaProducer<String, String>
) {
    routing {
        get("/isalive") {
            call.respondText("Alive", ContentType.Text.Plain)
        }

        get("/isready") {
            val databaseHealthy = checkDatabase(dataSource)
            val kafkaHealthy = checkKafka(kafkaProducer)

            if (databaseHealthy && kafkaHealthy) {
                call.respondText("Ready", ContentType.Text.Plain)
            } else {
                call.respondText(
                    "Not ready",
                    ContentType.Text.Plain,
                    HttpStatusCode.ServiceUnavailable
                )
            }
        }
    }
}

fun checkDatabase(dataSource: HikariDataSource): Boolean {
    return try {
        dataSource.connection.use { it.isValid(1) }
    } catch (e: Exception) {
        false
    }
}

fun checkKafka(producer: KafkaProducer<String, String>): Boolean {
    return try {
        producer.partitionsFor("health-check-topic").isNotEmpty()
    } catch (e: Exception) {
        false
    }
}

Prometheus Metrics Setup

kotlinCopy code
import io.micrometer.core.instrument.Clock
import io.micrometer.core.instrument.binder.jvm.*
import io.micrometer.prometheus.PrometheusConfig
import io.micrometer.prometheus.PrometheusMeterRegistry
import io.prometheus.client.CollectorRegistry
import io.ktor.server.metrics.micrometer.*
import io.ktor.server.response.*
import io.ktor.http.*

val meterRegistry = PrometheusMeterRegistry(
    PrometheusConfig.DEFAULT,
    CollectorRegistry.defaultRegistry,
    Clock.SYSTEM
)

fun Application.configureMetrics() {
    install(MicrometerMetrics) {
        registry = meterRegistry
        // Production pattern from navikt/ao-oppfolgingskontor
        meterBinders = listOf(
            JvmMemoryMetrics(),        // Heap, non-heap memory
            JvmGcMetrics(),            // Garbage collection
            ProcessorMetrics(),        // CPU usage
            UptimeMetrics()            // Application uptime
        )
    }

    routing {
        get("/metrics") {
            call.respondText(
                meterRegistry.scrape(),
                ContentType.parse("text/plain; version=0.0.4")
            )
        }
    }
}

Business Metrics

kotlinCopy code
import io.micrometer.core.instrument.Counter
import io.micrometer.core.instrument.Timer

class UserService(private val meterRegistry: PrometheusMeterRegistry) {
    private val userCreatedCounter = Counter.builder("users_created_total")
        .description("Total users created")
        .register(meterRegistry)

    private val userCreationTimer = Timer.builder("user_creation_duration_seconds")
        .description("User creation duration")
        .register(meterRegistry)

    fun createUser(user: User) {
        userCreationTimer.record {
            repository.save(user)
        }
        userCreatedCounter.increment()
    }
}

OpenTelemetry Tracing

Nais enables OpenTelemetry auto-instrumentation by default. For manual spans:

kotlinCopy code
import io.opentelemetry.api.GlobalOpenTelemetry
import io.opentelemetry.api.trace.Span
import io.opentelemetry.api.trace.StatusCode

val tracer = GlobalOpenTelemetry.getTracer("my-app")

fun processPayment(paymentId: String) {
    val span = tracer.spanBuilder("processPayment")
        .setAttribute("payment.id", paymentId)
        .startSpan()

    try {
        // Business logic
        val payment = repository.findPayment(paymentId)
        span.setAttribute("payment.amount", payment.amount)

        processPaymentInternal(payment)
        span.setStatus(StatusCode.OK)
    } catch (e: Exception) {
        span.setStatus(StatusCode.ERROR, "Payment processing failed")
        span.recordException(e)
        throw e
    } finally {
        span.end()
    }
}

Structured Logging

kotlinCopy code
import mu.KotlinLogging
import net.logstash.logback.argument.StructuredArguments.kv

private val logger = KotlinLogging.logger {}

fun processOrder(orderId: String) {
    logger.info(
        "Processing order",
        kv("order_id", orderId),
        kv("timestamp", LocalDateTime.now())
    )

    try {
        orderService.process(orderId)

        logger.info(
            "Order processed successfully",
            kv("order_id", orderId)
        )
    } catch (e: Exception) {
        logger.error(
            "Order processing failed",
            kv("order_id", orderId),
            kv("error", e.message),
            e
        )
        throw e
    }
}

Nais Manifest

yamlCopy code
apiVersion: nais.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: myteam
  labels:
    team: myteam
spec:
  image: ghcr.io/navikt/my-app:latest
  port: 8080

  # Health checks
  liveness:
    path: /isalive
    initialDelay: 10
    timeout: 1
    periodSeconds: 10
    failureThreshold: 3

  readiness:
    path: /isready
    initialDelay: 10
    timeout: 1
    periodSeconds: 10
    failureThreshold: 3

  # Prometheus scraping
  prometheus:
    enabled: true
    path: /metrics

  # OpenTelemetry auto-instrumentation
  observability:
    autoInstrumentation:
      enabled: true
      runtime: java # Instruments Ktor, JDBC, Kafka automatically
    logging:
      destinations:
        - id: loki # Automatic Loki shipping
        - id: team-logs # Optional: private team logs

  # Resources (for metrics alerting)
  resources:
    limits:
      memory: 512Mi
    requests:
      cpu: 50m
      memory: 256Mi

Alert Configuration

Create .nais/alert.yml:

yamlCopy code
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-app-alerts
  namespace: myteam
  labels:
    team: myteam
spec:
  groups:
    - name: my-app-alerts
      rules:
        - alert: HighErrorRate
          expr: |
            (sum(rate(http_requests_total{app="my-app",status=~"5.."}[5m]))
            / sum(rate(http_requests_total{app="my-app"}[5m]))) > 0.05
          for: 5m
          labels:
            namespace: myteam # required
            severity: critical
          annotations:
            summary: "Error rate is {{ $value | humanizePercentage }}"
            consequence: "Users get errors"
            action: "Check logs in Grafana Loki"

        - alert: HighResponseTime
          expr: |
            histogram_quantile(0.95,
              rate(http_request_duration_seconds_bucket{app="my-app"}[5m])
            ) > 1
          for: 10m
          labels:
            namespace: myteam
            severity: warning
          annotations:
            summary: "95th percentile response time is {{ $value }}s"
            action: "Check Tempo traces for slow requests"

        - alert: PodCrashLooping
          expr: |
            rate(kube_pod_container_status_restarts_total{
              pod=~"my-app-.*"
            }[15m]) > 0
          for: 5m
          labels:
            namespace: myteam
            severity: critical
          annotations:
            summary: "Pod {{ $labels.pod }} is crash looping"
            action: "Check logs: kubectl logs {{ $labels.pod }}"

        - alert: HighMemoryUsage
          expr: |
            (container_memory_working_set_bytes{app="my-app"}
            / container_spec_memory_limit_bytes{app="my-app"}) > 0.9
          for: 10m
          labels:
            namespace: myteam
            severity: warning
          annotations:
            summary: "Memory usage is {{ $value | humanizePercentage }}"
            action: "Check for memory leaks, increase limits if needed"

Nais sends these to the team's default Slack channel. To use another channel, add an AlertmanagerConfig and label those rules alert_type: custom, or the default route posts them twice (see apps/copilot-api/.nais/app.yaml in navikt/copilot and the Nais alerting docs).

Complete Example

kotlinCopy code
import io.ktor.server.application.*
import io.ktor.server.engine.*
import io.ktor.server.netty.*
import io.micrometer.core.instrument.Timer
import io.opentelemetry.api.GlobalOpenTelemetry
import io.opentelemetry.api.trace.StatusCode

fun main() {
    val env = Environment.from(System.getenv())
    val dataSource = createDataSource(env.databaseUrl)

    // Run database migrations
    runMigrations(dataSource)

    // Setup metrics
    val meterRegistry = setupMetrics()

    embeddedServer(Netty, port = 8080) {
        configureHealthEndpoints(dataSource)
        configureMetrics(meterRegistry)
        configureRouting(dataSource, meterRegistry)
    }.start(wait = true)
}

fun Application.configureRouting(
    dataSource: HikariDataSource,
    meterRegistry: PrometheusMeterRegistry
) {
    val tracer = GlobalOpenTelemetry.getTracer("my-app")

    routing {
        get("/api/users") {
            val requestTimer = Timer.sample()
            val requestCounter = meterRegistry.counter(
                "http_requests_total",
                "method", "GET",
                "endpoint", "/api/users"
            )

            val span = tracer.spanBuilder("getUsersRequest")
                .setAttribute("http.method", "GET")
                .setAttribute("http.route", "/api/users")
                .startSpan()

            try {
                val users = userRepository.findAll()
                span.setAttribute("user.count", users.size.toLong())
                span.setStatus(StatusCode.OK)

                requestCounter.increment()
                requestTimer.stop(meterRegistry.timer(
                    "http_request_duration_seconds",
                    "method", "GET",
                    "endpoint", "/api/users",
                    "status", "200"
                ))

                call.respond(users)
            } catch (e: Exception) {
                span.setStatus(StatusCode.ERROR, "Failed to get users")
                span.recordException(e)

                meterRegistry.counter(
                    "http_requests_total",
                    "method", "GET",
                    "endpoint", "/api/users",
                    "status", "500"
                ).increment()

                // trace_id and span_id are auto-injected into MDC by OTel agent
                logger.error("Failed to get users", e)

                throw e
            } finally {
                span.end()
            }
        }
    }
}

Grafana & Loki & Tempo Queries

See references/grafana-queries.md for PromQL dashboard panels, LogQL query examples, and Tempo trace search patterns.

Monitoring Checklist

• /isalive endpoint implemented
• /isready endpoint with dependency checks (database, Kafka)
• /metrics endpoint exposing Prometheus metrics
• Health checks configured in Nais manifest
• Business metrics instrumented (counters, timers, gauges)
• Verify trace_id appears in logs (auto-injected by OTel agent via MDC)
• OpenTelemetry auto-instrumentation enabled in Nais manifest
• Alert rules created in .nais/alert.yml
• Slack channel configured for alerts
• Grafana dashboard created
• No sensitive data in logs or metrics (verify in Grafana)
• High-cardinality labels avoided (no user_ids, transaction_ids)
• Metric names are snake_case with unit suffix, counters end in _total
• Label values are bounded (no free-text, no unbounded ids)
• Logs go to stdout as JSON, never to files

Production Patterns & DORA Metrics

See references/production-patterns.md for real-world patterns from navikt repositories and DORA metric implementation examples.

More references

• references/metric-conventions.md: Prometheus naming rules (snake_case, unit suffixes, _total), label cardinality rules, and the Gauge and Histogram patterns.
• references/tracing-and-logging.md: trace context propagation (W3C, Kafka headers, DB), log levels, logging practice, and fixing broken log-to-trace correlation.
• references/auto-instrumentation.md: what Nais auto-instrumentation actually covers, runtime: sdk, sensitive-data masking, and the filtered noisy paths.
• references/alerting.md: raw Prometheus alert-rule schema, alerting practice, and the common Nais alert catalogue (availability, memory, DB pool, Kafka lag, DORA).
• references/rapids-rivers.md: event metrics, Kafka consumer-lag gauge, and event tracing for Rapids & Rivers rivers.
• references/typescript.md: Grafana Faro for frontends and prom-client metrics in Next.js API routes.