Whizard Telemetry Ruler

WizTelemetry Ruler

Overview

WizTelemetry Ruler is an extension component in the KubeSphere Observability Platform that provides event alerting and log alerting capabilities. It can define alerting rules for K8s native events, K8s/KubeSphere auditing events, and K8s logs, evaluate incoming event data and log data, and send alerts to specified receivers such as alertmanager, etc.

When to Use

• Installing or configuring the WizTelemetry Ruler extension
• Creating, updating, or deleting alerting rules (RuleGroup/ClusterRuleGroup)
• Managing alerting configurations
• Using the ruler API to manage alerting rules

Components

Dependencies

• WizTelemetry Platform Service (whizard-telemetry): Required
• WizTelemetry Events (whizard-events): Required if event alerting is enabled
• WizTelemetry Auditing (whizard-auditing): Required if auditing alerting is enabled
• WizTelemetry Logging (whizard-logging): Required if logging alerting is enabled
• WizTelemetry Notification (whizard-notification): Optional (for alert notification)
• WizTelemetry Data Pipeline (vector): Required if alerting persistence is enabled
• OpenSearch (opensearch): Required if alerting persistence is enabled

Installation

Prerequisites

REQUIRED: Complete all steps in order before generating InstallPlan.

Step 1: Get Available Clusters and Confirm Target

⚠️ CRITICAL: DO NOT proceed until target clusters are determined.

Step 1.1: Get available clusters

bashCopy code
kubectl get clusters -o jsonpath='{.items[*].metadata.name}'

Step 1.2: Determine target clusters

• If user explicitly specified target clusters in the request → Use those clusters directly, proceed to Step 2
• If user did NOT specify target clusters → Ask user to confirm which clusters to deploy to, then proceed to Step 2

Ask user (if not specified):

Copy code
Available clusters: host, dev
Which clusters do you want to deploy WizTelemetry Ruler to?

Step 2: Get Latest Version (if not provided by user)

MUST do this to get the latest version:

bashCopy code
kubectl get extensionversions -n kubesphere-system -l kubesphere.io/extension-ref=whizard-telemetry-ruler -o jsonpath='{range .items[*]}{.spec.version}{"\n"}{end}' | sort -V | tail -1

This outputs the latest version (e.g., 1.5.0). Note this down - you'll use it in the InstallPlan.

Step 3: Get AlertManager Host (if configuring sink)

Only perform this step if you need to configure sink for alert notifications.

The AlertManager proxy service (alertmanager-proxy) is deployed in the host cluster and exposed via NodePort (default port: 31093).

Step 3.1: Get a host node IP

bashCopy code
kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}'

Step 3.2: Confirm with user

Ask user to confirm the AlertManager host IP:

Copy code
Detected AlertManager host: <NODE_IP>
Detected AlertManager port: 31093
Alert URL: http://<NODE_IP>:31093/api/v1/alerts

Do you want to use this URL for alert notifications?

• If user confirms → Use http://<NODE_IP>:31093/api/v1/alerts as the sink URL
• If user provides different URL → Use the user-specified URL

Note: If using WizTelemetry Notification extension, ensure it is installed before installing WizTelemetry Ruler.

Install WizTelemetry Ruler

⚠️ IMPORTANT: Complete prerequisite steps BEFORE this step.

Based on your selections:

• Target clusters: User-confirmed cluster names
• AlertManager URL: From Step 3 (if configuring sink)

⚠️ CRITICAL: InstallPlan metadata.name MUST be whizard-telemetry-ruler. DO NOT use any other name.

⚠️ CRITICAL: config field is YAML format. You MUST:

• Use the config structure exactly as shown in the template
• DO NOT add configuration fields that are not shown in the template
• DO NOT modify the structure or hierarchy

⚠️ CRITICAL: All placeholders MUST be replaced with actual values. DO NOT leave them as placeholders.

Basic Installation Template (with AlertManager)

yamlCopy code
apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: whizard-telemetry-ruler
  namespace: kubesphere-system
spec:
  extension:
    name: whizard-telemetry-ruler
    version: <VERSION>  # From Step 2
  enabled: true
  upgradeStrategy: Manual
  config: |
    whizard-telemetry-ruler:
      config:
        sinks:
          - name: alertmanager
            type: webhook
            config:
              url: http://<ALERT_MANAGER_HOST>:31093/api/v1/alerts  # From Step 3
  clusterScheduling:
    placement:
      clusters:
        - <TARGET_CLUSTERS>

Installation with Custom Configuration Template (with all alerting types)

yamlCopy code
apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: whizard-telemetry-ruler
  namespace: kubesphere-system
spec:
  extension:
    name: whizard-telemetry-ruler
    version: <VERSION>  # From Step 2
  enabled: true
  upgradeStrategy: Manual
  config: |
    whizard-telemetry-ruler:
      auditingAlerting:
        enabled: true
      eventsAlerting:
        enabled: true
      loggingAlerting:
        enabled: false
      config:
        sinks:
        - name: alertmanager
          type: webhook
          config:
            url: http://<ALERT_MANAGER_HOST>:31093/api/v1/alerts  # From Step 3
  clusterScheduling:
    placement:
      clusters:
        - <TARGET_CLUSTERS>

Replace placeholders:

• <VERSION>: From Step 2 (e.g., 1.5.0)
• <TARGET_CLUSTERS>: User-confirmed cluster names
• <ALERT_MANAGER_HOST>: From Step 3 (auto-detected or user-confirmed node IP)

Enable Log Alerting Template

yamlCopy code
apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: whizard-telemetry-ruler
  namespace: kubesphere-system
spec:
  extension:
    name: whizard-telemetry-ruler
    version: <VERSION>  # From Step 2
  enabled: true
  upgradeStrategy: Manual
  config: |
    whizard-telemetry-ruler:
      auditingAlerting:
        enabled: true
      eventsAlerting:
        enabled: true
      loggingAlerting:
        enabled: true
      config:
        sinks:
        - name: alertmanager
          type: webhook
          config:
            url: http://<ALERT_MANAGER_HOST>:31093/api/v1/alerts  # From Step 3
  clusterScheduling:
    placement:
      clusters:
        - <TARGET_CLUSTERS>

Enable Alerting Persistence Template

yamlCopy code
apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: whizard-telemetry-ruler
  namespace: kubesphere-system
spec:
  extension:
    name: whizard-telemetry-ruler
    version: <VERSION>  # From Step 2
  enabled: true
  upgradeStrategy: Manual
  config: |
    global:
       alertingPersistence:
         enabled: true
    whizard-telemetry-ruler:
      config:
        sinks:
        - name: alertmanager
          type: webhook
          config:
            url: http://<ALERT_MANAGER_HOST>:31093/api/v1/alerts  # From Step 3
    alerting-persistence:
      sinks:
        opensearch:
          enabled: true
  clusterScheduling:
    placement:
      clusters:
        - <TARGET_CLUSTERS>

Configuration Parameters

Alerting Type Parameters

Sink Configuration Parameters

Alert Persistence Parameters (Optional)

Resource Parameters

Node Scheduling Parameters

Alerting Rule API

RuleGroup API (Namespaced)

List RuleGroups

bashCopy code
curl -X GET "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/logging.whizard.io/v1alpha1/namespaces/<namespace>/rulegroups?clusterName=host" \
  -H "X-Remote-User: admin"

Get RuleGroup

bashCopy code
curl -X GET "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/logging.whizard.io/v1alpha1/namespaces/<namespace>/rulegroups/<name>?clusterName=host" \
  -H "X-Remote-User: admin"

Create RuleGroup

bashCopy code
curl -X POST "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/logging.whizard.io/v1alpha1/namespaces/<namespace>/rulegroups?clusterName=host" \
  -H "X-Remote-User: admin" \
  -H "Content-Type: application/json" \
  -d '{
    "apiVersion": "logging.whizard.io/v1alpha1",
    "kind": "RuleGroup",
    "metadata": {
      "name": "<rulegroup-name>",
      "namespace": "<namespace>"
    },
    "spec": {
      "type": "events",
      "rules": [
        {
          "name": "test-rule",
          "desc": "Test rule",
          "enable": true,
          "expr": {
            "kind": "rule",
            "condition": "reason == \"FailedCreatePodSandBox\""
          },
          "alerts": {
            "severity": "warning",
            "message": "Pod sandbox creation failed",
            "labels": {
              "alert": "test"
            },
            "annotations": {
              "summary": "Pod sandbox creation failed"
            }
          }
        }
      ]
    }
  }'

Update RuleGroup

bashCopy code
curl -X PUT "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/logging.whizard.io/v1alpha1/namespaces/<namespace>/rulegroups/<name>?clusterName=host" \
  -H "X-Remote-User: admin" \
  -H "Content-Type: application/json" \
  -d '<UPDATED_RULEGROUP>'

Delete RuleGroup

bashCopy code
curl -X DELETE "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/logging.whizard.io/v1alpha1/namespaces/<namespace>/rulegroups/<name>?clusterName=host" \
  -H "X-Remote-User: admin"

ClusterRuleGroup API (Cluster-scoped)

List ClusterRuleGroups

bashCopy code
curl -X GET "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/logging.whizard.io/v1alpha1/clusterrulegroups?clusterName=host" \
  -H "X-Remote-User: admin"

Get ClusterRuleGroup

bashCopy code
curl -X GET "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/logging.whizard.io/v1alpha1/clusterrulegroups/<name>?clusterName=host" \
  -H "X-Remote-User: admin"

Create ClusterRuleGroup

bashCopy code
curl -X POST "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/logging.whizard.io/v1alpha1/clusterrulegroups?clusterName=host" \
  -H "X-Remote-User: admin" \
  -H "Content-Type: application/json" \
  -d '{
    "apiVersion": "logging.whizard.io/v1alpha1",
    "kind": "ClusterRuleGroup",
    "metadata": {
      "name": "<clusterrulegroup-name>"
    },
    "spec": {
      "type": "auditing",
      "rules": [
        {
          "name": "audit-rule",
          "desc": "Audit rule",
          "enable": true,
          "expr": {
            "kind": "rule",
            "condition": "verb == \"delete\""
          },
          "alerts": {
            "severity": "error",
            "message": "Delete operation detected",
            "labels": {
              "type": "audit"
            }
          }
        }
      ]
    }
  }'

Delete ClusterRuleGroup

bashCopy code
curl -X DELETE "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/logging.whizard.io/v1alpha1/clusterrulegroups/<name>?clusterName=host" \
  -H "X-Remote-User: admin"

Alert Query API

Query Alerts

Query alerts with filters and time range:

bashCopy code
curl -X POST "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/events.alerting.wiztelemetry.io/v1alpha1/query" \
  -H "X-Remote-User: admin" \
  -H "Content-Type: application/json" \
  -d '{
    "cluster": "host",
    "startTime": 1704067200,
    "endTime": 1704153600,
    "from": 0,
    "size": 10,
    "order": "descending",
    "parameters": [
      {
        "field": "severity",
        "operator": "=",
        "value": "error"
      },
      {
        "field": "alertname",
        "operator": "?",
        "values": ["pod*"]
      }
    ]
  }'

Query Statistics

Get alert statistics (overview, histogram, by severity, etc.):

bashCopy code
curl -X POST "http://whizard-telemetry-apiserver.extension-whizard-telemetry.svc:80/kapis/events.alerting.wiztelemetry.io/v1alpha1/statistics" \
  -H "X-Remote-User: admin" \
  -H "Content-Type: application/json" \
  -d '{
    "cluster": "host",
    "statisticsType": 501,
    "startTime": 1704067200,
    "endTime": 1704153600
  }'

Alert Query Parameters

Filter Parameter Structure

Filter Operators

Statistics Type Values

Available Alert Fields

Alert Query Examples

Query by Severity

jsonCopy code
{
  "cluster": "host",
  "parameters": [
    {
      "field": "severity",
      "operator": "=",
      "value": "error"
    }
  ],
  "size": 20
}

Query by Time Range

jsonCopy code
{
  "cluster": "host",
  "startTime": 1704067200,
  "endTime": 1704153600,
  "size": 100
}

Query by Namespace and RuleGroup

jsonCopy code
{
  "cluster": "host",
  "parameters": [
    {
      "field": "namespace",
      "operator": "=",
      "value": "default"
    },
    {
      "field": "rulegroup",
      "operator": "=",
      "value": "my-rule-group"
    }
  ]
}

Fuzzy Match Alert Name

jsonCopy code
{
  "cluster": "host",
  "parameters": [
    {
      "field": "alertname",
      "operator": "?",
      "values": ["pod*", "container*"]
    }
  ]
}

Query with Statistics

jsonCopy code
{
  "cluster": "host",
  "startTime": 1704067200,
  "endTime": 1704153600,
  "statisticsType": 507
}

RuleGroup API Query Parameters

Rule Type Values

Alert Severity Values

Rule Expression Kind Values

Condition Detailed Explanation

The condition field is used to filter events/logs/auditing that match specific criteria. It supports various operators and field references.

Operators

Available Fields by Type

Events Type (type: events)

Events Examples:

jsonCopy code
// Alert when pod sandbox creation fails
"condition": "reason == \"FailedCreatePodSandBox\""

// Alert on warning events for specific namespace
"condition": "type == \"Warning\" && involvedObject.namespace == \"default\""

// Alert when event count exceeds threshold
"condition": "count >= 5"

Auditing Type (type: auditing)

Auditing Examples:

jsonCopy code
// Alert on delete operations
"condition": "verb == \"delete\""

// Alert on failed requests (4xx/5xx)
"condition": "responseStatus.code >= 400"

// Alert on specific user activity
"condition": "user == \"admin\" && verb == \"delete\""

// Alert on sensitive resources
"condition": "resource.resource == \"secrets\""

Logs Type (type: logs)

Logs Examples:

jsonCopy code
// Alert on error keyword
"condition": "log contains \"error\""

// Alert on specific container
"condition": "container == \"nginx\""

// Alert on OOM kills
"condition": "log contains \"OOMKilled\""

// Alert on multiple keywords
"condition": "log contains \"failed\" && log contains \"connection\""

// Alert using regex
"condition": "log =~ \"error.*timeout|timeout.*error\""

Macro Usage

Macros allow reusable expressions:

jsonCopy code
{
  "expr": {
    "kind": "macro",
    "macro": "high_error_rate"
  }
}

List Usage

Lists allow grouping values:

jsonCopy code
{
  "expr": {
    "kind": "list",
    "list": ["error", "warning", "critical"]
  }
}

Alias Usage

Aliases provide descriptive names for complex expressions:

jsonCopy code
{
  "expr": {
    "kind": "alias",
    "alias": "Pod_Sandbox_Failure"
  }
}

Sliding Window Alert (Log Alerting)

For log alerting with sliding window:

jsonCopy code
{
  "name": "log-rate-rule",
  "desc": "Log rate alert",
  "enable": true,
  "expr": {
    "kind": "rule",
    "condition": "log contains \"error\""
  },
  "alerts": {
    "severity": "error",
    "message": "High error log rate"
  },
  "slidingWindow": {
    "windowSize": "5m",
    "slidingInterval": "1m",
    "count": 100
  }
}

Extension Operations

Check Extension Status

bashCopy code
kubectl get installplan whizard-telemetry-ruler
kubectl get extensionversions -l kubesphere.io/extension-ref=whizard-telemetry-ruler

Uninstall Extension

Uninstall from all clusters:

bashCopy code
kubectl delete installplan whizard-telemetry-ruler

Uninstall from specific cluster:

To remove WizTelemetry Ruler from a specific cluster, update the InstallPlan by removing that cluster from clusterScheduling.placement.clusters:

yamlCopy code
apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: whizard-telemetry-ruler
  namespace: kubesphere-system
spec:
  extension:
    name: whizard-telemetry-ruler
    version: <VERSION>
  enabled: true
  upgradeStrategy: Manual
  clusterScheduling:
    placement:
      clusters:
        - <REMAINING_CLUSTERS>  # Remove the cluster you want to uninstall from

Alert Notification Configuration

To send alerts through WizTelemetry Notification extension, configure the sink URL to point to the alertmanager-proxy service.

Auto-detection (recommended): Use the command from Step 3 to get the host node IP automatically.

yamlCopy code
whizard-telemetry-ruler:
  config:
    sinks:
    - name: alertmanager
      type: webhook
      config:
        url: http://<ALERT_MANAGER_HOST>:31093/api/v1/alerts

• <ALERT_MANAGER_HOST>: From Step 3 (auto-detected or user-confirmed)
• Default NodePort: 31093
• Service: alertmanager-proxy in kubesphere-system namespace