CloudWatch

CloudWatch is organized around four core data types, each with its own ingestion path, storage model, and retention behavior:

Metrics are the most critical concept. AWS services publish metrics automatically into namespaces like AWS/EC2 or AWS/RDS. You can also publish custom metrics using the PutMetricData API or the CloudWatch agent.

Metric retention tiers automatically aggregate over time: data points are available at the original resolution for 3 hours, at 1-minute resolution for 15 days, at 5-minute resolution for 63 days, and at 1-hour resolution for 455 days.

A CloudWatch Alarm watches a single metric or a math expression and transitions between three states: OK, ALARM, and INSUFFICIENT_DATA. When state changes occur, alarms can trigger SNS notifications, Auto Scaling actions, EC2 actions (reboot, stop, terminate, recover), or Systems Manager OpsItems.

Composite alarms are underused but powerful. They let you suppress noisy individual alarms by only triggering when multiple conditions are true simultaneously. This dramatically reduces false positives in production alerting.

# Create an alarm on CPU utilization
aws cloudwatch put-metric-alarm \
  --alarm-name "high-cpu-web-server" \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --statistic Average \
  --period 300 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alerts

CloudWatch Logs stores log events in Log Groups, each containing multiple Log Streams. A Log Group is typically one application or service. A Log Stream is one instance or source (one Lambda function invocation container, one EC2 instance).

CloudWatch Logs Insights provides a query language to search and analyze log data. It's far more powerful than the basic filter and can aggregate across log groups.

# Logs Insights: find the top 10 slowest Lambda invocations
fields @timestamp, @duration, @requestId
| filter @type = "REPORT"
| sort @duration desc
| limit 10

# Count errors per minute
filter @message like /ERROR/
| stats count(*) as error_count by bin(1m)
| sort @timestamp asc

The CloudWatch Agent must be installed on EC2 instances (and on-premises servers) to collect system-level metrics like memory utilization, disk usage, and swap - these are not available from the hypervisor. The agent also ships log files to CloudWatch Logs.

CloudWatch Dashboards provide customizable views across metrics and alarms. They can include graphs, number widgets, alarm status widgets, and Logs Insights query results. Dashboards can be shared publicly (read-only) or accessed via cross-account sharing.

Container Insights is a specialized feature for ECS and EKS that collects CPU, memory, network, and disk metrics at the cluster, service, task, and container level using an embedded agent. It also includes pre-built dashboards.

Application Insights automatically discovers application components (EC2, RDS, ELB, etc.), establishes baselines, and creates alarms for anomalies. It's particularly useful for .NET and SQL Server workloads on EC2.

CloudWatch pricing has multiple dimensions. Costs can grow unexpectedly if you're not careful about custom metrics, log volume, and API call frequency.

# Set retention policy on a log group (never let logs accumulate indefinitely)
aws logs put-retention-policy \
  --log-group-name /aws/lambda/my-function \
  --retention-in-days 30

# Find log groups with no retention policy set
aws logs describe-log-groups \
  --query "logGroups[?!retentionInDays].logGroupName" \
  --output text