Vulnerability Validation

Vulnerability Validation

Validate security findings by assessing whether they are actually exploitable in the context of this codebase. This skill filters false positives, confirms real vulnerabilities, and generates proof-of-concept exploits.

When to Use This Skill

• After commit-security-scan - Validate findings before creating issues or blocking PRs
• HIGH/CRITICAL findings - Prioritize validation of severe findings
• Before patching - Confirm vulnerability is real before investing in fixes
• Security review - Deep-dive validation of specific findings

Prerequisites

• .factory/threat-model.md must exist (from threat-model-generation skill)
• security-findings.json must exist (from commit-security-scan skill)

Inputs

Instructions

Follow these steps for each finding to validate:

Step 1: Load Context

1. Read security-findings.json from commit-security-scan
2. Read .factory/threat-model.md for system context
3. Identify which findings to validate based on inputs

Step 2: Reachability Analysis

For each finding, determine if the vulnerable code is reachable:

1. Trace entry points

   • Can external users reach this code path?
   • What HTTP endpoints, CLI commands, or event handlers lead here?
   • Is authentication required to reach this code?

2. Map the call chain

   • Starting from the entry point, trace the path to the vulnerable code
   • Document each function call in the chain
   • Note any branching conditions that must be satisfied

3. Classify reachability

   • EXTERNAL - Reachable from unauthenticated external input
   • AUTHENTICATED - Requires valid user session
   • INTERNAL - Only reachable from internal services
   • UNREACHABLE - Dead code or blocked by conditions

Step 3: Control Flow Analysis

Determine if an attacker can control the vulnerable input:

1. Identify the source

   • Where does the tainted data originate?
   • HTTP parameter, file upload, database query, environment variable?

2. Trace data flow

   • Follow the data from source to sink (vulnerable function)
   • Document each transformation or validation step
   • Note any sanitization, encoding, or type conversion

3. Assess attacker control

   • Can the attacker fully control the input?
   • Are there length limits, character restrictions, or format validation?
   • Does the data pass through any sanitization?

Step 4: Mitigation Assessment

Check if existing security controls prevent exploitation:

1. Input validation

   • Is the input validated before reaching the vulnerable code?
   • What validation rules are applied?

2. Framework protections

   • Does the framework provide automatic protection? (e.g., ORM parameterization, React XSS escaping)
   • Is the protection enabled and properly configured?

3. Security middleware

   • Are there WAF rules, rate limiting, or other controls?
   • Do CSP headers or other browser protections apply?

4. Reference threat model

   • Check the "Existing Mitigations" section for this threat type
   • Verify mitigations are actually in place

Step 5: Exploitability Assessment

Determine how difficult it is to exploit:

Consider:

• Attack complexity
• Required privileges
• User interaction needed
• Scope of impact

Step 6: Generate Proof-of-Concept

For confirmed vulnerabilities, create a proof-of-concept:

1. Craft exploit payload

   • Create a minimal payload that demonstrates the vulnerability
   • Use benign payloads (no actual damage)

2. Document the request

   • HTTP method, URL, headers, body
   • Or CLI command, file input, etc.

3. Describe expected vs actual behavior

   • What should happen (secure behavior)
   • What actually happens (vulnerable behavior)

Example PoC structure:

jsonCopy code
{
  "payload": "' OR '1'='1",
  "request": "GET /api/users?search=' OR '1'='1",
  "expected_behavior": "Returns users matching search term",
  "actual_behavior": "Returns all users due to SQL injection"
}

Step 7: Calculate CVSS Score

Assign a CVSS 3.1 score based on:

Example: CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N = 9.1 (Critical)

Step 8: Classify Finding

Based on analysis, classify each finding:

Step 9: Generate Output

Create validated-findings.json:

jsonCopy code
{
  "validation_id": "val-<timestamp>",
  "validation_date": "<ISO timestamp>",
  "scan_id": "<from security-findings.json>",
  "threat_model_version": "<from threat-model.md>",
  "validated_findings": [
    {
      "id": "VULN-001",
      "status": "CONFIRMED",
      "original_severity": "HIGH",
      "validated_severity": "HIGH",
      "exploitability": "EASY",
      "reachability": "EXTERNAL",
      "existing_mitigations": [],
      "exploitation_path": [
        "User submits search query via GET /api/users?search=<payload>",
        "Express router passes query to searchUsers() handler",
        "Handler passes unsanitized input to SQL template literal",
        "PostgreSQL executes malicious SQL"
      ],
      "proof_of_concept": {
        "payload": "' OR '1'='1",
        "request": "GET /api/users?search=' OR '1'='1",
        "expected_behavior": "Returns users matching search term",
        "actual_behavior": "Returns all users due to SQL injection"
      },
      "cvss_vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:N",
      "cvss_score": 9.1,
      "validation_notes": "Confirmed via code tracing. No input validation or parameterization."
    }
  ],
  "false_positives": [
    {
      "id": "VULN-003",
      "original_severity": "MEDIUM",
      "reason": "Input is validated by Joi schema in middleware before reaching this code. Schema enforces UUID format which prevents injection.",
      "evidence": "See src/middleware/validation.js:45 - Joi.string().uuid()"
    }
  ],
  "needs_manual_review": [
    {
      "id": "VULN-005",
      "original_severity": "HIGH",
      "reason": "Complex data flow through message queue. Unable to fully trace if sanitization occurs in consumer service."
    }
  ],
  "summary": {
    "total_analyzed": 10,
    "confirmed": 5,
    "likely": 2,
    "false_positives": 2,
    "needs_manual_review": 1,
    "by_severity": {
      "CRITICAL": 1,
      "HIGH": 3,
      "MEDIUM": 1,
      "LOW": 0
    }
  }
}

Success Criteria

The skill is complete when:

• All specified findings have been analyzed
• Each finding has a status (CONFIRMED, LIKELY, FALSE_POSITIVE, NEEDS_MANUAL_REVIEW)
• Confirmed findings have exploitation paths documented
• Confirmed findings have proof-of-concept exploits
• False positives have clear reasoning
• validated-findings.json is valid JSON
• CVSS scores are calculated for confirmed findings

Verification

Run these checks before completing:

bashCopy code
# Verify output exists and is valid JSON
cat validated-findings.json | jq . > /dev/null && echo "✓ Valid JSON"

# Check all findings have status
jq '.validated_findings | all(.status)' validated-findings.json

# Check confirmed findings have PoC
jq '.validated_findings | map(select(.status == "CONFIRMED")) | all(.proof_of_concept)' validated-findings.json

# Check false positives have reasoning
jq '.false_positives | all(.reason)' validated-findings.json

Example Invocations

Validate all findings:

Copy code
Validate the security findings from the last scan.

Validate specific findings:

Copy code
Validate findings VULN-001 and VULN-002 from security-findings.json.

Validate only HIGH/CRITICAL:

Copy code
Validate all HIGH and CRITICAL severity findings from the security scan.

Validate with specific files:

Copy code
Validate findings in security-findings.json using threat model at .factory/threat-model.md.

References

• CVSS 3.1 Calculator: https://www.first.org/cvss/calculator/3.1
• OWASP Testing Guide: https://owasp.org/www-project-web-security-testing-guide/
• Examples: validation-examples.md (in this skill directory)
• Upstream: commit-security-scan skill