CloudTrail

Record every API call made in your AWS account for auditing and forensics

AWS CloudTrail records every API call made in your AWS account, creating an immutable audit log of who did what, when, and from where. It is the foundation of AWS security auditing, incident investigation, and compliance. CloudTrail is automatically enabled for management events in every new account, but must be explicitly configured for data events and organizational coverage.

Trail Types: Management Events, Data Events, and Insights

Event type	What it captures	Cost	Example events
Management events (control plane)	API calls that modify AWS resources	First copy free per region; additional trails $2/100k events	CreateInstance, DeleteBucket, PutRolePolicy, AssumeRole
Data events (data plane)	Resource-level operations on data	$0.10 per 100,000 events	S3:GetObject, S3:PutObject, Lambda:Invoke, DynamoDB:GetItem
CloudTrail Insights	Anomaly detection on write API call rates	$0.35 per 100,000 events analyzed	Unusual spike in TerminateInstances calls

⚠️

Data events are not enabled by default. S3 data event logging can generate enormous volumes of events and cost hundreds of dollars per day on active buckets. Enable data events selectively - only for sensitive buckets (containing PII or credentials) or critical Lambda functions.

Organization Trails and Centralized Logging

An organization trail is created in the management account and automatically applies to all current and future member accounts. All events from all accounts and regions are delivered to a central S3 bucket.

bash

# Create an organization trail from the management account
aws cloudtrail create-trail \
  --name org-audit-trail \
  --s3-bucket-name aci-cloudtrail-logs \
  --is-multi-region-trail \
  --enable-log-file-validation \
  --is-organization-trail

# Start logging
aws cloudtrail start-logging --name org-audit-trail

# Verify trail is healthy
aws cloudtrail get-trail-status --name org-audit-trail

💡

Enable log file validation when creating a trail. This generates a hash digest file every hour that allows you to prove CloudTrail logs have not been tampered with - critical for compliance frameworks that require tamper-evident audit logs (PCI DSS, HIPAA, FedRAMP).

Querying CloudTrail Logs with Athena and CloudTrail Lake

CloudTrail logs can be queried in two ways: via Amazon Athena (SQL queries against S3-stored JSON files) or CloudTrail Lake (managed event data store with SQL interface, faster and simpler to set up).

	Athena + S3	CloudTrail Lake
Storage	Your S3 bucket (raw JSON)	Managed CloudTrail Lake event data store
Query tool	Athena SQL	CloudTrail Lake SQL (subset of ANSI SQL)
Setup	Create Athena table, partition by date	Create event data store, done
Cost	S3 storage + Athena $5/TB scanned	$0.75/GB ingested + $0.005/GB scanned per query
Retention	S3 lifecycle policies	7 years max in Lake
Best for	Existing Athena infrastructure, large-scale analytics	Quick forensic queries, compliance investigations

bash

-- Athena query: find all IAM role assumptions by a specific user in the last 7 days
SELECT
  eventtime,
  useridentity.arn,
  requestparameters,
  sourceipaddress,
  awsregion
FROM cloudtrail_logs
WHERE eventname = 'AssumeRole'
  AND useridentity.username = 'suspicious-user'
  AND eventtime > DATE_SUB(NOW(), INTERVAL 7 DAY)
ORDER BY eventtime DESC;

-- Find all API calls that resulted in Deny (AccessDenied errors)
SELECT eventtime, eventsource, eventname, useridentity.arn, errorcode
FROM cloudtrail_logs
WHERE errorcode = 'AccessDenied'
  AND partition_date = '2024-01-15'
ORDER BY eventtime DESC
LIMIT 100;

Protecting CloudTrail from Tampering

An attacker who gains admin access will often try to disable CloudTrail to cover their tracks. Defense-in-depth requires making this difficult and alerting on attempts.

Protection	How to implement
Log file validation	Enable --enable-log-file-validation on trail creation (detects tampering)
S3 bucket protection	Enable S3 Object Lock (WORM) on the CloudTrail bucket, block public access, restrictive bucket policy
CloudWatch alarm	Alarm on StopLogging, DeleteTrail, UpdateTrail API calls (CloudTrail -> CloudWatch Logs -> metric filter -> alarm)
SCP guardrail	AWS Organizations SCP that denies cloudtrail:DeleteTrail and cloudtrail:StopLogging for all principals except the security account
EventBridge alert	Route CloudTrail management events to EventBridge, alert on StopLogging

⚠️

GuardDuty has a finding type (Stealth:IAMUser/LoggingConfigurationModified) that fires when CloudTrail logging is disabled. Enabling GuardDuty provides a second layer of detection even if an attacker tries to destroy the trail before you can alert on it.

🎯

Interview Focus Points

1What is the difference between CloudTrail management events and data events? Why aren't data events enabled by default?
2How do you set up centralized CloudTrail logging across all accounts in an AWS Organization?
3How would you detect if someone disabled CloudTrail logging in your account?
4Walk me through how you would use CloudTrail to investigate a suspected credential compromise.
5What is log file validation and why is it important for compliance?
6What is the difference between using Athena vs CloudTrail Lake for querying CloudTrail logs?
7How would you use an SCP to prevent any account in your Organization from disabling CloudTrail?
8A security incident occurred 6 months ago. How do you ensure CloudTrail logs from that period are still available for investigation?