Investigating Azure Incidents

Investigating Azure Incidents

In Azure the control plane logs almost everything through Azure Resource Manager, so an incident is reconstructed from the Activity Log and the resource/data-plane logs, anchored on the identity that made the calls -- a user, a service principal, or a managed identity. The recurring trap is that identity lives in Entra while the damage lives in the subscription: you must correlate across both planes, because the Activity Log tells you what was done to a resource but the Entra sign-in log tells you who held the token and from where.

When to Use

• Suspected Azure resource compromise -- a subscription, resource group, or VM behaving as though someone else controls it
• Anomalous Azure Activity Log entries -- writes from unfamiliar callers, IPs, or regions, or an AccessDenied storm that looks like enumeration
• A Microsoft Defender for Cloud alert -- crypto-mining, anomalous resource deployment, suspicious sign-in, or IMDS token theft
• Managed-identity or service-principal abuse -- tokens minted for an app or VM being used from somewhere they should never appear
• A crypto-mining VM -- an unexpected spend spike, GPU/large SKUs, or new deployments in regions you do not operate in
• Storage-account or Key Vault exfiltration -- key regeneration, SAS-token minting, public-access changes, disk-snapshot sharing, or secret dumps

When NOT to Use

• The compromise is Entra identity / M365 mailbox, not Azure resources (though you will often need both) -- use investigating-m365-entra
• The incident is in AWS -- use investigating-aws-incidents
• The incident is in Google Cloud -- use investigating-gcp-incidents
• You are the attacker against the tenant, not the responder -- use attacking-entra-id
• The pivot is specifically into AKS / Kubernetes -- use attacking-eks-gke-aks; for defending the cluster, defending-kubernetes
• The general IR process and host-level response -- use responding-to-incidents

Log Sources and Where They Live

Establish what you have before you query. Missing logs are a finding, not a reason to skip the question.

The trap: the Activity Log is a control-plane record. Data-plane operations -- reading a blob, fetching a Key Vault secret, querying a Cosmos DB -- are not in the Activity Log at all. They exist only if a diagnostic setting was configured on that resource before the incident. Absence in the Activity Log is never evidence that data was untouched (see Rationalizations).

Confirm what is actually being logged before you trust a gap: az monitor diagnostic-settings subscription list (is the Activity Log exported beyond 90 days?) and az monitor diagnostic-settings list --resource <id> (does this storage account / vault ship data-plane logs anywhere?).

First-Hour Triage

Three moves, in order: scope the caller identity, pull its recent activity, preserve before you contain.

Scope the caller identity. Resolve the report -- a Defender alert, a billing spike, a suspicious deployment -- to the identity in the caller / identity fields of the Activity Log. That principal (a UPN, or a service principal / managed identity object ID) is the anchor for everything else.

bashCopy code
# Everything a specific caller did across the subscription control plane
az monitor activity-log list --caller attacker@contoso.com \
  --start-time 2026-07-01T00:00:00Z -o json

# Who holds what right now -- role assignments are the escalation surface
az role assignment list --all --include-inherited \
  --query "[?roleDefinitionName=='Owner' || roleDefinitionName=='User Access Administrator']" -o table

Pull recent activity from KQL if a Log Analytics workspace exists -- it is faster and richer than the CLI once you are past the first look:

kustoCopy code
AzureActivity
| where TimeGenerated > ago(7d)
| where Caller == "attacker@contoso.com"
| project TimeGenerated, OperationNameValue, ActivityStatusValue,
    CallerIpAddress, ResourceProviderValue, ResourceId, CorrelationId
| order by TimeGenerated asc

Preserve, then contain. An attacker who sees a role assignment revoked mid-operation will burn persistence you have not found. For anything but active, ongoing damage: snapshot disks, export the relevant logs, map persistence, then contain everything at once. Live mining or active exfil is the exception -- stop the damage and accept the trade.

Activity Log Deep-Dive (KQL)

The AzureActivity table is the authoritative control-plane record. Learn its fields:

• OperationNameValue -- the ARM operation, e.g. Microsoft.Authorization/roleAssignments/write. This is what you hunt on.
• Caller -- the UPN or object ID that made the call.
• CallerIpAddress -- external IPs on a managed identity that should only call from inside Azure are the IMDS-theft signature (below).
• ResourceProviderValue -- Microsoft.Compute, Microsoft.Storage, Microsoft.KeyVault, Microsoft.Authorization.
• ActivityStatusValue -- a run of Failure (often AuthorizationFailed) is enumeration: the attacker mapping what the stolen principal can reach.
• CorrelationId -- ties the sub-operations of one logical action together; pivot on it to expand a single suspicious event into its full sequence.

kustoCopy code
// Enumeration storm -- authorization failures by operation
AzureActivity
| where TimeGenerated > ago(7d)
| where ActivityStatusValue == "Failure"
| summarize n = count() by Caller, OperationNameValue, CallerIpAddress
| order by n desc

// Expand one event's full correlated sequence
AzureActivity
| where CorrelationId == "<correlation-id>"
| project TimeGenerated, OperationNameValue, ActivityStatusValue, ResourceId
| order by TimeGenerated asc

Canonical Attacker Operations to Hunt

Grep the timeline for these OperationNameValue patterns -- they are the shape of nearly every Azure intrusion.

• Privilege escalation via role assignment -- Microsoft.Authorization/roleAssignments/write granting Owner, Contributor, or User Access Administrator (UAA can grant itself anything). Watch for custom-role creation (Microsoft.Authorization/roleDefinitions/write) that hides * actions behind an innocuous name.
• Credential adds to a service principal or app -- done in Entra, not the Activity Log: a new secret or certificate on an app registration gives persistent, MFA-independent access. Correlate to the Entra audit log ("Add service principal credentials" / "Update application - Certificates and secrets management").
• Managed-identity token abuse -- a token minted for a compromised VM's system-assigned identity used to call ARM. The MI's object ID appears as Caller from an unexpected CallerIpAddress.
• Code execution on VMs -- Microsoft.Compute/virtualMachines/runCommand/action and Custom Script Extension (Microsoft.Compute/virtualMachines/extensions/write installing CustomScript) run attacker code as SYSTEM/root without any RDP/SSH.
• Compute/serverless persistence -- Microsoft.Web/sites (App Service / Functions), Automation Runbooks, and Logic Apps as scheduled backdoors that re-mint credentials or re-grant roles.
• Resource-level RBAC backdoors -- a role assignment scoped to a single storage account or vault rather than the subscription, easy to miss in a top-level review.

kustoCopy code
AzureActivity
| where TimeGenerated > ago(14d)
| where OperationNameValue has_any (
    "roleAssignments/write", "roleDefinitions/write",
    "runCommand/action", "virtualMachines/extensions/write")
| project TimeGenerated, Caller, CallerIpAddress, OperationNameValue, ResourceId
| order by TimeGenerated asc

The CLI equivalent filters the same operations: az monitor activity-log list --start-time <t> --query "[?contains(operationName.value,'roleAssignments/write')]".

Identity-Plane Correlation

The Activity Log names the principal but not the human behind it. Map the service principal or managed identity object ID back to Entra to see the authentication context -- this is the cross-plane step, and it usually means opening investigating-m365-entra.

kustoCopy code
// Where did this service principal / managed identity actually sign in from?
AADServicePrincipalSignInLogs
| where ServicePrincipalId == "<sp-or-mi-object-id>"
| where TimeGenerated > ago(30d)
| project TimeGenerated, AppId, ServicePrincipalName, IPAddress,
    ResourceDisplayName, ResultType
| order by TimeGenerated asc

// For an interactive user: sign-ins around the abusive Activity Log calls
SigninLogs
| where UserPrincipalName == "attacker@contoso.com"
| project TimeGenerated, IPAddress, Location, AppDisplayName,
    ConditionalAccessStatus, AuthenticationRequirement, ResultType

Check conditional-access status and whether MFA was actually satisfied -- a service principal bypasses interactive CA entirely, which is exactly why attackers pivot to SP/MI credentials.

Managed-Identity and SSRF Credential Theft

The Azure analogue of AWS IMDS theft: an SSRF or foothold on a VM / App Service reads the Instance Metadata Service to lift the managed identity's token, then uses it elsewhere.

Copy code
http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/

The signature is unmistakable: the managed identity's calls appear in AzureActivity from a CallerIpAddress that is not the resource's own outbound IP. The token is minted for that resource, so any call from an unrelated or external IP means the token left the box.

kustoCopy code
AzureActivity
| where TimeGenerated > ago(7d)
| where Caller == "<managed-identity-object-id>"
| summarize ops = count() by CallerIpAddress, OperationNameValue
| order by ops desc     // flag IPs that are not the VM's egress

Correlate the theft window with NSG flow logs (if enabled) for the outbound SSRF and the reuse source. On App Service the token endpoint uses IDENTITY_ENDPOINT with a header secret rather than 169.254.169.254 -- the same off-resource-use logic applies.

Defender for Cloud Alert Triage

Defender for Cloud is a starting pistol, not the investigation. Each alert maps to a hypothesis you confirm in the Activity Log and diagnostic logs.

bashCopy code
az security alert list -o table
az security alert show --location <loc> -n <alert-name> -g <rg>

An alert older than the 90-day Activity Log window still carries the principal and IPs -- pivot on those even after the raw events have aged out.

Data-Theft Detection

• Storage account key regeneration -- Microsoft.Storage/storageAccounts/listKeys/action and regenerateKey hand the attacker a full-access key that works outside RBAC and outside the Activity Log thereafter.
• SAS-token minting -- listAccountSas / listServiceSas produces a time-boxed exfil URL that leaves no per-object control-plane trail.
• Blob exfil -- visible only if data-plane diagnostic logging is on: StorageBlobLogs shows GetBlob volume by caller.
• Public-access changes on containers -- setting a container or account to allow anonymous/blob public access is exfil staging.
• Disk snapshot export / sharing -- Microsoft.Compute/snapshots/write then /beginGetAccess/action mints a SAS download URL for a full disk image.
• Key Vault secret dumps -- Microsoft.KeyVault/vaults/read (VaultGet) to enumerate, then data-plane SecretGet reads (in AzureDiagnostics / AZKVAuditLogs, only if logging was enabled).

kustoCopy code
// Control-plane data-theft indicators
AzureActivity
| where TimeGenerated > ago(14d)
| where OperationNameValue has_any (
    "storageAccounts/listKeys", "storageAccounts/regenerateKey",
    "listAccountSas", "listServiceSas",
    "snapshots/write", "snapshots/beginGetAccess")
| project TimeGenerated, Caller, CallerIpAddress, OperationNameValue, ResourceId

// Key Vault data-plane reads -- only present if diagnostics were on beforehand
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.KEYVAULT"
| where OperationName in ("SecretGet", "KeyGet", "VaultGet")
| project TimeGenerated, CallerIPAddress, identity_claim_upn_s, OperationName, id_s

Anti-Forensics the Attacker Attempts

A capable attacker tries to blind you. The key operations to hunt -- and their defeat:

• Diagnostic-setting deletion -- Microsoft.Insights/diagnosticSettings/delete stops data-plane logs from reaching the workspace.
• Activity Log export removal / narrowing -- deleting the subscription-level diagnostic setting that ships the Activity Log to a workspace or storage.
• Resource deletion -- deleting the VM, snapshot, or storage account to destroy the artifact and its logs.

The defeat is the same shape as an AWS org trail: a tenant-level export to a central, locked destination the compromised principal cannot reach -- immutable-storage (WORM/legal-hold) blob export, or Sentinel ingestion in a segregated workspace with delete protection and resource locks. When export is immutable, the attacker's own diagnosticSettings/delete call is logged there before it takes effect, so cleanup becomes evidence rather than a gap. If you lack it, record the blind window as a scoping limitation.

kustoCopy code
AzureActivity
| where TimeGenerated > ago(14d)
| where OperationNameValue has_any (
    "diagnosticSettings/delete", "Microsoft.Insights/diagnosticSettings/write")
| project TimeGenerated, Caller, CallerIpAddress, OperationNameValue, ResourceId

Evidence Preservation

• Snapshot the involved VMs' disks before touching them, and lock the snapshots so they cannot be deleted.

  bashCopy code
  az snapshot create -g <rg> -n IR-2026-042-osdisk \
    --source <os-disk-id> --tags case=IR-2026-042 legal-hold=true
  az lock create --name IR-2026-042-hold --lock-type CanNotDelete \
    --resource-group <rg> --resource-name IR-2026-042-osdisk \
    --resource-type Microsoft.Compute/snapshots
  

• Export the relevant Activity Log window before it ages out of 90-day retention -- run the CLI/KQL and save the JSON to a preserved, locked store.

• Apply resource locks / legal hold to evidence artifacts, and involve legal before collection if the incident may become a regulatory or litigation matter.

Containment

Do it all at once, after scoping. Partial containment alerts the attacker.

• Disable or rotate the compromised principal. For a service principal / app, disable it and roll its credentials:

  bashCopy code
  az ad sp update --id <app-id> --set accountEnabled=false
  # remove attacker-added secrets/certs
  az ad app credential reset --id <app-id>
  

  For a user, disable the account and force a reset in Entra, then revoke sessions (below).

• Revoke role assignments the attacker granted:

  bashCopy code
  az role assignment delete --assignee <object-id> \
    --role Owner --scope /subscriptions/<sub-id>
  

• Revoke sessions / refresh tokens so existing tokens die (an Entra action -- Revoke-MgUserSignInSession / az ad user ...), because disabling alone leaves issued tokens valid up to an hour.

• Isolate the VM by swapping its NSG to a deny-all outbound rule rather than deleting it, so disk and memory survive for analysis (az network nsg rule create ... --access Deny --direction Outbound --protocol '*').

• Rotate storage keys and regenerate SAS -- renewing both keys invalidates every outstanding SAS and access key at once:

  bashCopy code
  az storage account keys renew --account-name <acct> -g <rg> --key primary
  az storage account keys renew --account-name <acct> -g <rg> --key secondary
  

• Remove attacker persistence -- delete backdoor Functions/App Service, Automation runbooks, custom roles, and resource-scoped role assignments -- only after they are documented.

Reach for Azure CLI and KQL in Log Analytics / Sentinel for the investigation itself; MicroBurst and ROADtools for understanding the TTPs an attacker would run (and what each leaves behind); and Microsoft's Unified Audit correlation when the Azure story crosses into M365.

Rationalizations to Reject

• "The Activity Log shows nothing, so no data was touched." The Activity Log is control plane only. Blob reads, secret fetches, and DB queries are data-plane operations that are invisible unless diagnostic logging was enabled beforehand. Absence there is not evidence of no exfil.
• "Defender for Cloud didn't alert, so there's no compromise." Defender covers a subset of behaviors and depends on the right plan and coverage. Absence of a finding is not evidence of absence -- the Activity Log timeline is authoritative.
• "We disabled the user, so the account is contained." Issued access tokens stay valid up to an hour, and any service-principal credentials or role assignments the attacker created survive the disable. Revoke sessions and audit every persistence mechanism.
• "It's a managed identity, it can only be used from inside Azure." That is the design, not a guarantee. SSRF/IMDS theft lifts the token off the resource; the off-resource CallerIpAddress is precisely the signature to hunt.
• "The service principal only has Contributor, not Owner." Contributor can run runCommand on VMs, read storage keys, and mint SAS tokens -- code execution and data theft without ever touching role assignments. Trace what the role can reach; do not assume.
• "It's just crypto-mining, low priority." Mining means a principal with deploy rights was compromised -- the same access could exfiltrate data or escalate through roleAssignments/write. Mining is the visible symptom, not the scope.
• "Logs only go back 90 days, so the compromise started within 90 days." That is the Activity Log's default retention, your visibility limit -- not the attacker's timeline. Record it as a scoping gap and check the workspace / immutable export for longer retention.

References

• investigating-m365-entra -- the sibling identity-plane skill; open it to correlate the SP/MI back to Entra sign-ins and audit
• investigating-aws-incidents -- the sibling cloud-IR skill when the incident is in AWS
• attacking-entra-id -- the offensive side; how these tenant/subscription TTPs are executed
• attacking-eks-gke-aks -- when the pivot is specifically into AKS / Kubernetes
• responding-to-incidents -- the general IR process, evidence handling, and host-level response
• reporting-security-findings -- structuring the incident narrative and deliverable
• Azure CLI (az) -- drive the control plane and pull the Activity Log
• KQL in Log Analytics / Microsoft Sentinel -- query AzureActivity, AzureDiagnostics, and sign-in tables
• Microsoft Defender for Cloud -- alert source and finding-type reference
• MicroBurst and ROADtools -- emulate and understand Azure/Entra attack TTPs and what each leaves in the logs