Amazon Web Services services
8 / 24
Amazon Web Services

Observability — CloudWatch & X-Ray

Collect metrics, logs, and traces to monitor health, alert on problems, and debug distributed systems.

On AWS the three observability signals map onto three services, and an interviewer expects you to know which question each one answers.

A hospital monitoring station. Vital-sign monitors (metrics) beep an alarm when a reading crosses a threshold, the patient chart (logs) records the detail, and tracking a patient through admission, surgery, and recovery (traces) shows exactly where a delay occurred.

Key Concepts

1
    CloudWatch metrics   is something wrong?     cheap, aggregated, alarmable
    CloudWatch Logs      what happened exactly?  priced by GB ingested
    X-Ray traces         which hop was slow?     one request, end to end
2
CloudWatch metrics drive alarms, and alarms are what turn data into action:
3
    ALB 5xx rate > 1% for 5 minutes
      -> SNS -> on-call page
      -> or trigger an Auto Scaling policy
      -> or roll back a CodeDeploy deployment
4
Page on what customers experience. An alarm on CPUUtilization wakes someone for an instance that is merely busy; alarms on the load balancer's HTTPCode_Target_5XX_Count and TargetResponseTime fire when users are actually affected. Composite alarms then combine several conditions, so one incident produces one page.
CPUUtilizationHTTPCode_Target_5XX_CountTargetResponseTime
5
Use percentile statistics. CloudWatch can alarm on p99 directly, while the Average statistic blends a slow 1% of requests into a line that looks healthy.
6
Logs Insights queries structured logs:
    fields @timestamp, @message
    | filter status >= 500
    | stats count() by bin(5m)
7
Log in JSON so fields are queryable rather than regex-matched, and set a retention period — log data kept forever is a quiet, permanent cost.
8
X-Ray follows one request across services and builds a service map:
9
    API Gateway -> Lambda (120ms) -> DynamoDB (8ms)
                           \-> S3 (40ms)
10
That is how you answer "the checkout is slow" with a service rather than a guess. Each segment carries timing, so the slow hop is visible rather than inferred.
11
Correlate the three. A trace id in every log line lets you move from an alarm, to the traces that were slow, to the exact log lines for one of them.
12
The cost note. Custom metrics are charged per metric per month, and high-cardinality dimensions — a metric per user id — become expensive very quickly. That belongs in logs, not metrics.
13
What the interviewer is probing.1. "Which do you alert on: CPU or error rate?" Probing: symptoms versus causes. Stalls: "CPU, it shows load." Moves up: the error rate or p99 latency — what the user feels; high CPU may be perfectly healthy.
14
2. "One bad deploy produced twenty separate pages. How do you stop that?" Probing: alarm design. Stalls: "Raise the thresholds." Moves up: a composite alarm that pages only when the symptom alarms — 5xx rate and p99 latency — breach together, with the component alarms notifying nobody on their own.
15
3. "What does X-Ray give you that logs do not?" Probing: the third pillar. Stalls: "Prettier logs." Moves up: one request's path across services with timing per hop, so "checkout is slow" resolves to a service rather than a guess.
16
4. "How do you keep CloudWatch costs down?" Probing: the cost model. Stalls: "Log less." Moves up: custom metrics are billed per metric, so high-cardinality dimensions like a metric per user are expensive — that belongs in logs, with a retention period set.
Use cases:cloudwatchx-raymetricstracing