Kafka Connector Review

Kafka Connect Configuration Review

Reviews Kafka Connect connector configurations for common misconfigurations. Connectors are defined as JSON/YAML and are entirely language-agnostic.

Target environment: $ARGUMENTS

Workflow

Copy this checklist and track your progress:

Copy code
Connector Review Progress:
- [ ] Step 1: List all connectors
- [ ] Step 2: Inspect each connector's configuration
- [ ] Step 3: Validate configurations against plugin schemas
- [ ] Step 4: Audit for common misconfigurations
- [ ] Step 5: Generate report

1. List all connectors with status and task states
2. Inspect each connector's configuration in detail
3. Validate configurations against plugin schemas
4. Audit for common misconfigurations
5. Report findings with current and recommended configs

Step 1: List All Connectors

Use the Lenses MCP list_kafka_connectors tool to get all connectors with:

• Connector name, class and type (source/sink)
• Status (RUNNING, PAUSED, FAILED, UNASSIGNED)
• Task states and count
• Cluster info

Flag immediately:

• Critical: Connectors in FAILED state
• Critical: Tasks in FAILED state
• Warning: Connectors in PAUSED state with no obvious reason

Expected output: List of all connectors with name, class, status and task states.

Validation: If no connectors are returned, report that no Connect cluster is configured and stop.

Step 2: Inspect Configurations

Use the Lenses MCP get_kafka_connector_target_definition tool to get the full configuration YAML for each connector.

Step 3: Validate Configurations

Use the Lenses MCP validate_connector_configuration tool to validate each connector's config against its plugin's schema. This catches:

• Missing required configuration fields
• Invalid configuration values
• Type mismatches

Step 4: Audit Common Misconfigurations

Error Handling

• Critical: errors.tolerance=all without errors.deadletterqueue.topic.name (silently drops messages)
• Warning: Missing errors.tolerance (defaults to none, connector stops on any error)
• Warning: Missing errors.log.enable=true (errors not logged)
• Suggestion: Add errors.deadletterqueue.context.headers.enable=true for richer DLQ metadata

Converters

• Warning: key.converter or value.converter mismatch with topic serialisation format
• Warning: Using org.apache.kafka.connect.storage.StringConverter for structured data (should use Avro/JSON/Protobuf converter)
• Warning: Missing schemas.enable setting where schemas are expected

Transforms (SMTs)

• Warning: Complex transform chains (> 3 SMTs) that may be hard to debug
• Suggestion: Consider Kafka Streams or ksqlDB for complex transformations
• Verify transform class names are valid and in the correct order

Task Count

• Warning: tasks.max=1 on high-throughput connectors (limits parallelism)
• Suggestion: Source connectors - tasks.max should align with source partitioning
• Suggestion: Sink connectors - tasks.max should align with topic partition count

Connection and Retry

• Warning: Missing retry configuration for connectors that interact with external systems
• Warning: Missing connection timeout settings
• Suggestion: Set explicit consumer.override.* or producer.override.* for performance tuning

Naming Conventions

• Suggestion: Connector names should follow a consistent pattern (e.g., {source/sink}-{system}-{entity})

Success Criteria

Quantitative

• Triggers on 90% of connector-related queries (test with 10-20 varied phrasings)
• Completes review in under 10 MCP tool calls
• Catches 100% of validation errors reported by the plugin schema

Qualitative

• Failed connectors are flagged immediately in the status overview
• Error handling gaps (missing DLQ, silent drops) are always identified
• Report is useful for both connector operators and developers

Examples

Example 1: Routine connector review

User says: "Review all connectors in staging"

Actions:

1. List all connectors with status
2. Inspect and validate each connector's configuration
3. Audit for common misconfigurations Result: Full report covering all connectors with prioritised findings

Example 2: Investigating a failed connector

User says: "My sink connector is failing, can you check why?"

Actions:

1. List connectors and identify those in FAILED state
2. Get the full configuration for the failed connector
3. Validate config against plugin schema
4. Check for common issues (DLQ, converters, error handling) Result: Diagnosis of the specific failure with remediation steps

Example 3: Single connector deep dive

User says: "Review the config for the elasticsearch-sink connector"

Actions:

1. Fetch the target definition for the named connector
2. Validate against its plugin schema
3. Audit error handling, converters, task count Result: Focused report on a single connector

Troubleshooting

No connectors returned

Cause: No Kafka Connect cluster is configured in the environment or no connectors are deployed. Solution: Verify that a Connect cluster exists via Lenses UI. Check get_deployment_targets for available Connect clusters.

Validation fails with unknown plugin

Cause: The connector plugin class is not installed on the Connect cluster. Solution: Report the missing plugin. This is a deployment issue, not a configuration issue.

Connector shows RUNNING but tasks are FAILED

Cause: The connector framework is running but individual tasks have encountered errors. Solution: Check each task's status. Common causes include authentication failures, network issues or schema mismatches with the external system.

Output Format

Copy code
## Connector Review Report

### Environment: {name}

### Connector Status Overview
| Connector | Class | Status | Tasks | Failed Tasks |
|-----------|-------|--------|-------|-------------|
| name | class | RUNNING | 3/3 | 0 |

### Critical (must fix)
- [connector-name] Description of the issue
  Current: {current config} | Recommended: {recommended config}

### Warning (should fix)
- [connector-name] Description of the issue
  Current: {current config} | Recommended: {recommended config}

### Validation Errors
- [connector-name] {field}: {validation error message}

### Suggestion (consider improving)
- [connector-name] Description of the suggestion
  Recommendation: {how to improve}

### Summary
- X connectors reviewed
- Y critical issues found
- Z warnings found
- Failed connectors: N
- Failed tasks: M