Bedrock Operation Review

Amazon Bedrock Operational Review

Conduct a comprehensive operational review of Amazon Bedrock workloads aligned with the AWS Well-Architected Framework and Amazon Bedrock best practices.

This skill uses the AWS Bedrock, Bedrock Agent, CloudWatch, Service Quotas, and EC2 APIs only — all data is collected through native AWS control-plane APIs and CloudWatch metrics. It performs no data-plane model invocations and reads no prompt or response content.

When to Use

Activate this skill when the user asks to:

• Review, audit, or assess an Amazon Bedrock workload
• Check Bedrock best-practices compliance
• Evaluate Bedrock security, performance, cost, quotas, or resilience
• Perform a Bedrock operational readiness review (ORR)
• Investigate Bedrock configuration drift, throttling, or cost drivers

Step 1: Identify Target Scope

Ask the user which accounts and regions to review. Accept:

• Specific account IDs and regions
• "all regions" for a given account
• A specific pillar or set of checks (e.g. "just cost optimization")

If no scope is given, default to all configured account regions and all pillars. Default the analysis window to the last 7 days unless the user specifies a range (historical windows older than ~2 weeks may have reduced CloudWatch resolution).

Step 2: Discover Bedrock Resources

Per account/region, begin by checking whether the account actually uses Bedrock in this region. Do not use bedrock.ListFoundationModels as an activity signal — it returns the regional model catalog available to the account and is generally non-empty in every supported region regardless of usage, so it is treated as catalog data only (see ListFoundationModels).

Instead, determine activity from account-owned resources and AWS/Bedrock metrics:

• Account-owned resources: bedrockagent.ListAgents, bedrockagent.ListKnowledgeBases, bedrock.ListGuardrails, bedrock.ListCustomModels, bedrock.ListProvisionedModelThroughputs, bedrock.ListInferenceProfiles (application-scoped), and bedrock.ListModelCustomizationJobs.
• Invocation activity: cloudwatch.ListMetrics for the AWS/Bedrock namespace.

If no account-owned Bedrock resources exist AND CloudWatch shows no AWS/Bedrock metrics for the region, record "No Bedrock activity detected — skipping pillar analysis" for that region and move on. Do not generate empty findings tables for inactive regions.

For regions where Bedrock is active, collect the full resource inventory:

Copy code
bedrock.ListFoundationModels                   # catalog reference only: model
                                               # availability + lifecycle status
                                               # (NOT an account-activity signal)
bedrock.ListGuardrails / GetGuardrail          # configured guardrails
bedrock.GetModelInvocationLoggingConfiguration
bedrock.ListInferenceProfiles / GetInferenceProfile
bedrock.ListPromptRouters / GetPromptRouter
bedrock.ListProvisionedModelThroughputs / GetProvisionedModelThroughput
bedrock.ListCustomModels / GetCustomModel
bedrock.ListModelCustomizationJobs / GetModelCustomizationJob
bedrockagent.ListAgents / GetAgent             # agents (draft version)
bedrockagent.ListAgentVersions / GetAgentVersion  # deployed versions — read only the
                                               # per-version model ID, orchestration
                                               # type, and whether a prompt override is
                                               # present. Do NOT read the override
                                               # base-prompt template text.
bedrockagent.ListAgentAliases
bedrockagent.ListKnowledgeBases / GetKnowledgeBase
bedrockagent.ListDataSources / GetDataSource
bedrockagent.ListPrompts                       # Prompt Management: presence/metadata
                                               # only (identifiers, versions, counts).
                                               # Do NOT call GetPrompt — it returns
                                               # prompt template/variant content, which
                                               # this skill does not read.

Capture per resource: identifiers, ARNs, status, creation/update timestamps, encryption configuration (AWS-managed vs customer-managed KMS key), and any VPC configuration.

Step 3: Collect CloudWatch Metrics (namespace AWS/Bedrock)

Before querying metrics, load the authoritative thresholds reference:

Copy code
read_skill_resource(skill_id="bedrock-operation-review", path="references/metrics-thresholds.md")

Use the thresholds from that file when classifying metric values as Normal, Warning, or Critical throughout this step and Step 4.

Discover which models are actually invoked with cloudwatch.ListMetrics (dimension ModelId), then pull metric data with cloudwatch.GetMetricData. Use the user's selected window; default to 7 days.

Key model-level metrics (dimension ModelId):

GPU metrics for self-managed workloads live in the CWAgent namespace (nvidia_smi_utilization_gpu, nvidia_smi_memory_util).

cloudwatch.GetMetricData allows up to 500 metric-data queries per call — batch requests and paginate when a region has many invoked models.

Step 4: Analyze Against Best Practices

Before evaluating findings, load the best-practices checklist:

Copy code
read_skill_resource(skill_id="bedrock-operation-review", path="references/best-practices-checklist.md")

Use the checklist as the canonical list of items to evaluate for each pillar. Mark each item as ✅ Pass, ⚠️ Warning, ❌ Fail, or ➖ Not Applicable, and generate a finding for every Fail or Warning.

Evaluate all collected data across the pillars below and assign a severity to every finding: CRITICAL, HIGH, MEDIUM, LOW, or INFO. The pillars mirror the Bedrock operational review structure: Security, Performance, Service Quotas, Cost Optimization, Resilience.

4.1 Security

Ref: Security in Amazon Bedrock

• Guardrails: no guardrail configured on high-volume workloads → HIGH. Use guardrails to filter hate, insult, sexual, violence, and PII content and to block denied topics relevant to the use case. Guardrails
• Guardrail signals: high InvocationClientErrors (>5% of Invocations) indicates problematic content reaching models — filter before invocation → MEDIUM. InvocationsIntervened <5% of invocations on an active guardrail suggests under-configuration; >15% suggests policy over-tuning → review.
• Model invocation logging: for workloads handling sensitive content (PII, PCI, HIPAA), enabling invocation logging persists prompt/output on the account and may conflict with data-handling requirements. Flag logging configuration mismatches → MEDIUM. Model invocation logging
• Knowledge Base configuration: chunking (200–1000 tokens), embedding strategy, encryption, access controls, and VPC endpoints → MEDIUM where missing. Knowledge bases
• Knowledge Base data source encryption: data sources without a customer-managed KMS key when handling sensitive content → MEDIUM. Encryption of knowledge base resources
• VPC configuration for model customization jobs: customization jobs without a configured VPC expose training data to the internet → HIGH. (us-east-1, us-west-2) Configure a VPC for Bedrock
• IAM fine-grained access control: agent/alias IAM policies using resource * without justification → MEDIUM. Follow least privilege for inference endpoints. A comprehensive least-privilege audit of all IAM policies is out of scope for this skill — flag obvious *-resource findings only, at INFO severity. IAM for Amazon Bedrock
• Knowledge Base logging: KB ingestion log delivery not configured → LOW. CloudTrail and CloudWatch not enabled for anomaly detection → MEDIUM.
• Prompt injection: this skill does not read prompt template content, so it does not assess whether individual templates are hardened. Instead, treat a guardrail configured with prompt-attack filtering as the control signal — workloads with no guardrail (or a guardrail lacking a prompt-attack content filter) are exposed to prompt injection → MEDIUM. Provide general hardening guidance by reference only. Prompt engineering best practices
• Model access: Amazon Bedrock foundation models are enabled by default (no explicit access request needed) — model access should instead be controlled via IAM and SCP policies scoped to the essential models for the use case (least privilege). A comprehensive least-privilege audit is out of scope for this skill — report gaps at INFO severity only. Model access

4.2 Performance

Ref: Monitoring Amazon Bedrock

• Latency and throttling: InvocationLatency p95, TimeToFirstToken p95, throttle rate (InvocationThrottles / Invocations), and server error rate. High throttle rate → HIGH (add retries with exponential backoff + jitter, request quota increase, spread load, use CRIS). Prefer smaller models and streaming for latency-sensitive apps. Improve Bedrock performance
• Agent performance: agents can use a latency-optimized flow when they have a single knowledge base, no enabled action groups, don't ask follow-ups, and use the default orchestration template → flag agents that miss these conditions. Optimize agent performance
• Agent version configuration drift: compare deployed alias versions (from GetAgentVersion) against the draft. Aliases pinned to outdated versions with different model IDs or orchestration types than the current draft → MEDIUM (may miss performance or capability improvements).
• Invoked model versions: models in Legacy or End-of-Life state still receiving invocations → MEDIUM (plan upgrade; check the model card for its EOL date rather than assuming a fixed notice period — Legacy notice periods are either 6 months or 45 days depending on the model, with most models using 6 months). Model lifecycle
• Data automation: success rate <95%, error rate >5%, or throttle rate >1% for production workloads → MEDIUM. Bedrock Data Automation
• Bedrock service tier: verify tier selection (default, flex, priority, reserved) matches workload criticality. Latency-sensitive apps on Flex, or batch workloads on Priority, are misaligned → MEDIUM. Service tiers

4.3 Service Quotas

Ref: Bedrock quotas

• Model quotas: compare observed P95 invocations-per-minute and tokens-per-minute against the account RPM/TPM quotas (including CRIS and Global CRIS quotas) from servicequotas.GetServiceQuota. Utilization >75% → MEDIUM (request increase before throttling); sustained near the limit → HIGH.
• Guardrail service quotas: track ApplyGuardrail requests and text-unit consumption for content filter, denied topic, sensitive information, word filter, and contextual grounding policies. Utilization >75% → MEDIUM. Note: CloudWatch metrics don't distinguish Classic vs Standard policy versions, so review manually when both are configured. Guardrail quotas

4.4 Cost Optimization

Ref: Bedrock pricing

• Application inference profiles: not used for cost allocation / tagging → LOW. App inference profiles integrate with Cost Allocation Tags for usage tracking. Track cost and usage with inference profiles
• Custom model distillation: narrow, repetitive, high-volume tasks on premium large models are distillation candidates. Rule of thumb: InvocationLatency p99

  3000 ms AND InputTokenCount >1M/month AND throttle rate >5% → high-priority candidate → MEDIUM opportunity. (us-east-1, us-west-2) Model distillation

• Prompt caching: cache offload % = CacheReadInputTokenCount / (CacheReadInputTokenCount + CacheWriteInputTokenCount). Statuses: Not Supported, Not Enabled, Misconfigured (writes but no reads), Underutilized (<50%), Optimized (≥50%). Input-heavy repeated context that isn't cached → MEDIUM opportunity (up to ~85% latency and ~90% cost reduction on cached prefixes). Prompt caching
• Prompt management: presence indicates adoption maturity — INFO. Encourage versioning and variant testing for cost/quality tradeoffs. Prompt management
• Intelligent prompt routing: routes requests within a model family to the best-quality/lowest-cost model. Absence on mixed-complexity workloads → LOW opportunity. Intelligent prompt routing
• Provisioned throughput: balance provisioned commitments against on-demand. Idle or expiring provisioned models, or steady baseline load on on-demand → MEDIUM. Provisioned Throughput
• Batch inference opportunity: high-volume (>10K/day), latency-tolerant, or scheduled workloads on on-demand → MEDIUM (up to ~50% cost savings). Self-managed batch on EC2 GPU / SageMaker Batch Transform → migrate to managed Batch Inference. Batch inference
• EC2 GPU utilization (self-managed P4/P5/P5en/P6, requires CloudWatch Agent + NVIDIA DCGM plugin, 7-day minimum window):
  • Avg GPU util <40% → over-provisioned → migrate training to Trainium2 (~30–50%).
  • Avg GPU memory util >90% → OOM risk → model parallelism / larger instance.
  • GPU usage CV (StdDev/Mean) >0.5 → bursty → Spot Instances / Capacity Blocks.
  • Avg GPU util >80% sustained → commitment candidate → ML Savings Plans / RIs.

4.5 Resilience

Ref: Cross-region inference

• Cross-Region Inference (CRIS): CRIS adoption % = Profile InputTokenCount / (Profile InputTokenCount + Base Model InputTokenCount). Low adoption (traffic bypassing the inference profile by invoking base models directly) reduces burst resilience and, for global profiles, forgoes ~10% savings → MEDIUM.

Step 5: Generate Report

Generate a shareable report artifact for the review.

Artifact naming: bedrock-review-<account-id>-<region>-<YYYY-MM-DD>.md Example: bedrock-review-123456789012-us-east-1-2026-04-29.md

Structure the Markdown document with:

Report Header

Copy code
# Amazon Bedrock Operational Review — <account-id> / <region>
Date: <YYYY-MM-DD> | Analysis window: <start> to <end>
Pillars reviewed: <list>

Executive Summary

• Health: ✅ HEALTHY / ⚠️ WARNINGS / ❌ CRITICAL
• Finding counts by severity
• Top 3 critical/high items

Findings by Pillar

For each of Security, Performance, Service Quotas, Cost Optimization, Resilience:

| # | Finding | Severity | Current State | Recommendation |

CloudWatch Metrics Summary

| Metric | Model | Stat | Value | Status | Finding |

Service Quota Utilization

| Quota | Value | Observed P95 | Utilization % | Risk |

Cost Optimization Opportunities

| Opportunity | Signal | Est. Impact | Effort |

Priority Matrix

| # | Finding | Severity | Pillar | Effort | Impact |

Next Steps

• Immediate (CRITICAL/HIGH — 7 days)
• Short-term (MEDIUM — 30 days)
• Long-term (LOW — 90 days)

Appendix — Reference Links

• Amazon Bedrock User Guide
• Bedrock Guardrails
• Monitoring Bedrock
• Bedrock Quotas
• Prompt Caching
• Model Distillation
• Batch Inference
• Cross-Region Inference
• Well-Architected Framework

Severity Definitions

Region-Restricted Checks

Some checks only run in specific regions:

• VPC Configuration for Model Customization Job: us-east-1, us-west-2
• Custom Model Distillation: us-east-1, us-west-2
• Application Inference Profiles: us-east-1, us-east-2, us-west-2, ap-northeast-1, ap-northeast-2, ap-southeast-2, ap-south-1, eu-central-1, eu-west-1, eu-west-3, us-gov-east-1, us-gov-west-1

Skip a check silently in unsupported regions rather than reporting a failure.

Known API Quirks

• CloudWatch metrics for guardrail Content Filter and Denied Topic policies do not distinguish Classic vs Standard policy versions — utilization can be inaccurate when both are configured. Review manually.
• cloudwatch.GetMetricData caps at 500 metric-data queries per call — batch and paginate for accounts with many invoked models.
• Some models are only invocable through Cross-Region Inference profiles and will show 0 direct token usage (100% CRIS adoption by design).
• Prompt cache entries expire after ~5 minutes of inactivity; aggregate CloudWatch windows are directional, not per-session.
• EC2 GPU signals require the CloudWatch Agent with the NVIDIA DCGM plugin; without it, nvidia_smi_* metrics are absent and GPU signals can't be evaluated.

Data Source Boundaries

This skill collects data exclusively through native AWS APIs (bedrock, bedrockagent, cloudwatch, servicequotas, ec2). It does not:

• Invoke any foundation model (no data-plane calls).
• Read prompt or response content.
• Depend on any non-AWS tooling or internal scripts — the skill is self-contained on the DevOps Agent's primary cloud-source IAM role.