Google Cloud services
16 / 24
Google Cloud

Orchestration — Workflows & Cloud Composer

Coordinate multi-step processes with retries and compensation, at the right weight for the job.

Two orchestrators, and the gap between them is large.

Workflows is a checklist taped to one job. Composer is the whole production planning office, which is worth having when there are hundreds of jobs and overkill for one.

Key Concepts

1
    Workflows        serverless, YAML, per-step billing. Milliseconds
                     to start. For service orchestration.
    Cloud Composer   managed Apache Airflow. A real cluster, always
                     running. For data pipelines and DAGs.
2
Workflows is the lightweight one.
    - validateOrder:
        call: http.post
        args:
          url: https://validate-abc.run.app
          auth: {type: OIDC}
        result: validation
    - chargeCard:
        try:
          call: http.post
          args: {url: https://charge-abc.run.app}
        retry:
          predicate: ${http.default_retry_predicate}
          max_retries: 3
          backoff: {initial_delay: 2, multiplier: 2}
        except:
          as: e
          steps:
            - compensate:
                call: http.post
                args: {url: https://release-stock-abc.run.app}
3
Retries, backoff and the compensating call are declared rather than written, and auth: OIDC means it calls private Cloud Run services with no key.
auth: OIDC
4
It has no cluster, so it costs nothing when idle and starts instantly — the opposite of Composer.
5
Cloud Composer is Airflow, with everything that implies: Python DAGs, a scheduler, operators for every Google service, backfills, and a web UI showing each task's state and logs.
6
It runs a GKE cluster underneath, so it bills continuously whether a DAG runs or not. That is the single most important cost fact about it, and the reason it is wrong for occasional service orchestration.
7
Choosing between them.
    chaining a few API calls, event-driven   Workflows
    nightly ETL with dependencies and backfill  Composer
    needs operators for BigQuery, Dataproc, GCS Composer
    cost matters and usage is intermittent   Workflows
8
The saga pattern applies to both. There is no distributed transaction, so each step needs a compensating action and failure walks them backwards — explicit in a declared workflow, and buried in service code otherwise.
9
Workflows waits cheaply. A sleep step or a callback endpoint can pause an execution for up to a year without consuming anything, which is how approval flows are built.
sleep
10
What the interviewer is probing.1. "Workflows or Cloud Composer?" Probing: the weight difference. Stalls: "Composer, it is Airflow." Moves up: Workflows for chaining service calls — serverless, per-step billing, nothing when idle; Composer for scheduled data pipelines with real dependency graphs and backfill.
11
2. "What is the main cost characteristic of Composer?" Probing: the always-on cluster. Stalls: "Per DAG run." Moves up: it runs a GKE cluster continuously, so it bills whether DAGs run or not — which makes it wrong for occasional orchestration.
12
3. "How do you implement compensation in Workflows?" Probing: saga. Stalls: "Use a transaction." Moves up: try and except per step with a compensating call, so failure walks the completed steps backwards.
13
4. "How does a workflow wait days for approval cheaply?" Probing: the callback. Stalls: "It polls." Moves up: a callback endpoint or sleep step pauses the execution for up to a year while consuming nothing.
Use cases:workflowscloud-composerairflowsagaretry