GCP Conceptual

The reasoning

**Storage.** Managed tables are written to Colossus in Capacitor, a columnar format: each column is stored, compressed and encoded separately, with metadata such as min and max values per block. Nothing is attached to a particular machine, so storage grows without anyone provisioning disks, and it is billed on its own — active storage, and long-term storage at roughly half the price for any table or partition not modified for 90 consecutive days.

**Compute.** Dremel turns SQL into a multi-stage execution plan. Each stage is split into units of work that run on **slots** — BigQuery's unit of compute, roughly a virtual CPU with memory. Stages exchange data through a distributed shuffle tier rather than through the workers' local disks, which is what lets the scheduler add or remove slots mid-query. The network underneath (Jupiter) is fast enough that reading remote storage is the normal case, not a penalty.

**Why it matters in practice.** Because storage is columnar, the bytes a query is billed for are the bytes of the columns it touches — `SELECT *` on a wide table costs the whole table even with `LIMIT 10`, since `LIMIT` is applied after the scan. Because compute is separate, you choose how to pay for it independently: per TiB scanned (on-demand, US list price $6.25 per TiB, first 1 TiB a month free) or per slot-hour under an edition. And because there are no indexes, the levers are partitioning, clustering and table shape, not index design.

The edge worth naming: "serverless" does not mean unlimited. An on-demand project can use up to 2,000 concurrent slots, and a query that needs more simply runs longer. That is the first thing to check when a query is slow but cheap.

The reasoning

A query plan is a graph of stages, and each stage is split into many units of work. Slots pick those units up; when a stage finishes, freed slots move to whatever is queued. If a stage asks for 2,000 slots and 1,000 are available, it runs in more waves rather than failing. BigQuery shares slots between concurrent queries with **fair scheduling**, so one heavy query does not starve the rest indefinitely.

**On-demand** is the default. You are billed for bytes processed (US list price $6.25 per TiB, with a 10 MB minimum per table referenced), and the project draws on a shared pool capped at 2,000 concurrent slots. You never see slots on the bill, which is why teams on on-demand rarely think about them until a query is inexplicably slow.

**Capacity pricing** replaces bytes with slot-hours. You create a **reservation** in an edition — Standard, Enterprise or Enterprise Plus — and **assign** projects, folders or the organisation to it. A reservation has a baseline (always allocated) and an autoscaling maximum; the autoscaler adds slots in multiples of 50 and bills per second with a one-minute minimum. Standard edition supports autoscaling only, up to 1,600 slots per reservation; Enterprise and Enterprise Plus add baselines, one- and three-year commitments at lower rates, and features Standard lacks. US pay-as-you-go list prices are $0.04, $0.06 and $0.10 per slot-hour respectively.

The practical consequences: under a reservation, `SELECT *` no longer costs more money, it costs more time — queries compete for the same slots. Idle baseline slots can be shared with other reservations in the same admin project unless you turn that off. And you size a reservation from evidence: `total_slot_ms` in `INFORMATION_SCHEMA.JOBS`, divided by the hours in the window, gives average slots actually used.

The reasoning

In a schema, a column can be `RECORD` (a `STRUCT` in SQL) and can have mode `REPEATED` (an `ARRAY`). An `orders` table can then hold `line_items ARRAY<STRUCT<sku STRING, qty INT64, price NUMERIC>>`. Capacitor stores `line_items.sku`, `line_items.qty` and `line_items.price` as separate columns with repetition information, so reading only `line_items.price` scans only that leaf — nesting does not force you to read the whole record.

You query it with `UNNEST`: `SELECT o.order_id, li.sku FROM orders AS o, UNNEST(o.line_items) AS li`. That comma is a `CROSS JOIN`, and it **drops orders whose array is empty**. When every order must appear, write `LEFT JOIN UNNEST(o.line_items) AS li` — this is the bug interviewers most like to hear you anticipate. Aggregating inside the row without flattening, such as `(SELECT SUM(li.qty) FROM UNNEST(o.line_items) AS li)`, avoids multiplying rows at all.

When to nest: the child is always read with the parent, is written at the same time, and is bounded in size — line items on an order, attributes on an event. You remove a join, and a join between two large tables is the expensive part of most warehouse queries. When not to: the child is updated on its own (changing one element means rewriting the whole array in that row), grows without bound, or is shared by many parents, such as a customer dimension. Those stay as tables and join by key.

Limits worth knowing: nesting can go up to 15 levels, arrays cannot directly contain arrays (wrap the inner one in a `STRUCT`), and a whole row still has to fit BigQuery's row-size limit, so an unbounded array of clickstream events on a user row is a design error, not a clever one.

The reasoning

**Managed tables** hold data in BigQuery storage. You get everything the engine offers: partitioning and clustering, block pruning from Capacitor metadata, `INSERT` / `UPDATE` / `MERGE`, time travel (seven days by default), cached results, and accurate dry-run estimates. You pay BigQuery storage prices and the data is only readable through BigQuery's APIs.

**External tables** are definitions over data kept elsewhere — most often CSV, JSON, Avro, Parquet or ORC files in a Cloud Storage bucket, optionally with a Hive-style `dt=2026-09-14/` layout so BigQuery can prune by path. You pay no BigQuery storage, the files stay usable by Spark on Dataproc or anything else, and a new file is visible to the next query without a load. You give up DML and clustering, performance depends on the file format (Parquet with few large files is fine; thousands of small gzip CSV files is not), and cost estimates are less precise. Result caching still works for GoogleSQL queries over Cloud Storage external tables; queries over other external sources such as Bigtable or Drive are never cached, so every repeat is billed again.

**BigLake tables** sit between the two: still external data, but with metadata caching and fine-grained access control through a connection, so you can grant column-level access without granting access to the bucket. When an interviewer asks "managed or external", mentioning that BigLake exists shows you know the security gap in plain external tables.

The usual pattern: external tables over the raw landing bucket for exploration and one-off checks; a scheduled load or `INSERT … SELECT` into a partitioned managed table for everything a dashboard or model reads daily.

The reasoning

The views are scoped by region or dataset. The region-qualified `JOBS` view (written with the region quoted, as in `region-us`) covers every job in the project in that region for the last 180 days; a dataset's `PARTITIONS` view lists each partition with its row count and last-modified time. Other useful ones are `COLUMNS` (schema discovery, or finding every table with a `customer_email` column), `TABLE_STORAGE` (logical versus physical bytes, time-travel bytes) and the `RESERVATIONS` and `JOBS_TIMELINE` views for capacity.

The query people are expected to write is the cost breakdown below: bytes billed and slot-hours per user over the last 30 days. Filtering on `creation_time` matters because the view is partitioned on it; without that filter the metadata query itself scans more than it needs to. Swap `user_email` for `query_info.query_hashes.normalized_literals` to group the same query text run with different literals, which is how a runaway scheduled query shows up.

Things it answers that the console does not, at scale: which scheduled query started scanning ten times more after someone edited a view; which partitions have not been updated since yesterday (a freshness check a DAG can assert on); how many slot-hours a workload really uses before you size a reservation; and which tables have no partition filter requirement.

Two caveats. Querying `INFORMATION_SCHEMA` is itself a billed query, with the usual 10 MB minimum. And `JOBS_BY_ORGANIZATION` needs organisation-level permissions, so on a locked-down project you may only be able to see your own jobs through `JOBS_BY_USER`.

The reasoning

**Legacy streaming (`insertAll`).** You send JSON rows over REST; they are queryable within seconds. US list price is $0.01 per 200 MiB, and each row is counted at a minimum of 1 KB, so millions of tiny rows cost more than their bytes suggest. De-duplication is best effort: if you supply an `insertId`, BigQuery drops a repeat seen within roughly one minute, and nothing more. Recently streamed rows sit in a streaming buffer that DML statements and table copies cannot fully touch until it is flushed.

**Storage Write API.** A gRPC API with a binary protocol-buffer payload. US list price is $0.025 per GiB with the first 2 TiB a month free. It offers stream types: the **default stream** (committed immediately, at-least-once, no stream management — the recommended migration target for `insertAll` users); **committed type** streams you create, where supplying an **offset** with each append makes a retry idempotent, so the write is exactly-once; **pending type**, where rows stay invisible until you commit the stream, giving an atomic batch load; and buffered type, which is rarely needed.

What changes in a design: exactly-once becomes something the sink can enforce, provided the writer tracks offsets; batch-style atomic loads are possible from a streaming client; and Dataflow's BigQuery connector can use the Storage Write API directly. What does not change: if the upstream sends the same event twice with different payload framing, no stream type will notice, so a business key is still the real de-duplication boundary.

And the option people forget: if the data arrives in files and minute-level freshness is not needed, a **load job** from Cloud Storage uses the free shared slot pool and costs nothing to ingest.

The reasoning

Delivery is driven by acknowledgements. A subscriber receives a message, processes it and acks; if the ack does not arrive before the ack deadline (10 seconds by default, up to 600), or the subscriber nacks, Pub/Sub redelivers it with the **same message ID**. Google's docs also note that redelivery can happen before the deadline or after an ack, so a subscriber must tolerate repeats. Unacknowledged messages are kept for the subscription's retention period — seven days by default, up to 31.

**Exactly-once delivery** is a subscription setting. With it on, a successfully acknowledged message is not redelivered, and a message is not redelivered while it is outstanding within its deadline; the client gets an ack result it must check, because an ack can fail. The limits are the point of the question: it applies to **pull** subscriptions (StreamingPull) only — not push, and not BigQuery or Cloud Storage export subscriptions — and only when subscribers connect to the same region. Subscribers spread across regions can still see duplicates.

What it never covers: two different messages with the same content. A publisher that retries after a lost publish response creates a second message with a new ID, and exactly-once delivery treats it as new. So end-to-end once-ness still needs a business key and an idempotent sink. And if Dataflow is the subscriber, leave the setting off — Dataflow de-duplicates by message ID itself in exactly-once streaming mode, and its docs advise against combining the two.

The reasoning

Without ordering, messages can arrive in any order. With it, two things must be true: the subscription has message ordering enabled, and the publisher sets an **ordering key** — a customer id, an order id, a table's primary key — on related messages, publishing them in the same region. Messages with the same key are then delivered in the order Pub/Sub received them; messages with an empty key are not ordered, and messages with different keys have no order relative to each other.

The costs: publish throughput per ordering key is limited to 1 MBps, so a hot key is a bottleneck by design. On the subscriber side, a key's next message is not delivered until the previous one is acknowledged, so one slow or poisoned message stalls its key. And **redelivery of a message triggers redelivery of all subsequent messages for that key**, even ones already acknowledged — ordering and duplicates come together. On the publisher side, a failed publish pauses the key until the client resumes publishing, so ordering cannot silently skip a message.

Ordering keys are not partitions: they are expected to be high-cardinality, and there is no fixed count to size. The design question is the same one as in any broker, though — you get order exactly for the key you chose. If two services publish events for one order under different keys, those events are unordered.

The reasoning

A message that is never acked is redelivered indefinitely within the subscription's **message retention duration** — seven days by default, 31 at most — and then discarded. Meanwhile it costs redelivery traffic, and with ordering keys it blocks everything behind it for that key. The ack deadline defaults to 10 seconds (maximum 600); the client libraries extend it automatically while a message is being processed, by default up to an hour, which is why a stuck handler can hold a message far longer than the deadline suggests.

The controls: a **retry policy** with exponential backoff (minimum and maximum backoff) so failing messages are not hammered; a **dead-letter topic** with **maximum delivery attempts** between 5 and 100 (default 5) — after that many failed deliveries the message is forwarded to the dead-letter topic, which needs its own subscription or it is lost. The Pub/Sub service agent must be allowed to publish to the dead-letter topic and subscribe to the source subscription, which is the setup step people miss.

What to watch: `subscription/oldest_unacked_message_age` and `subscription/num_undelivered_messages` in Cloud Monitoring. A rising oldest-age with a flat backlog means one stuck message; both rising means consumers are too slow. For recovery, **seek** can replay a subscription to a snapshot or timestamp — for already-acked messages only if the subscription retains acked messages or the topic has retention enabled.

The reasoning

Each element has an event timestamp; with Pub/Sub it is the publish time unless you tell the connector to read it from a message attribute. The source estimates a **watermark** — for Pub/Sub, from the oldest unprocessed timestamps in the backlog. When the watermark passes the end of a window, Dataflow considers that window's data complete. Anything arriving later is **late data**.

Three knobs decide what happens. **Allowed lateness** (zero by default in Beam) is how long after the watermark a window keeps accepting late elements; after it, they are dropped and state is freed. **Triggers** decide when a window emits: the default fires once at the watermark; you can add early firings (for example every minute of processing time, for a live dashboard) and late firings (once per late element, or per batch of them). **Accumulation mode** decides whether each firing emits the full result so far (accumulating) or only the change since the last pane (discarding) — which must match how the sink applies updates.

The trade-off is freshness against completeness and cost. A tight lateness drops more real events; a long one holds more state per key and window for longer. Beyond hours of lateness, most teams stop stretching the stream and let a batch job restate the affected day from the raw events instead.

The reasoning

**Fixed windows** (Beam's name for tumbling) split time into equal, non-overlapping intervals: streams per track per minute, revenue per hour. Each element is in exactly one window, so totals across windows add up. **Sliding windows** (hopping) have a size and a period: a 10-minute window every 1 minute means each element sits in 10 windows — good for rolling averages and alerting, but it multiplies the work and output.

**Session windows** are per key and data-driven: a new session starts when a key has been inactive for longer than the gap duration, so a listener who pauses for more than 30 minutes starts a new session. They merge as events arrive, so a late event can join two sessions into one, which is why late firings and accumulation matter most here. There is also the **global window**, used with custom triggers when you want, say, a running total emitted every N elements.

The questions that follow: fixed windows align to the epoch in UTC, so a "daily" window is a UTC day unless you offset it; session windows hold state for every active key, so a key that never goes quiet (a bot) produces one huge session; and sliding windows with a small period over a long size are expensive for exactly that reason.

The reasoning

Beam is the programming model: a pipeline of `PCollection`s transformed by `PTransform`s such as `ParDo` (with a `DoFn`), `GroupByKey` and `Combine`, with windowing and triggers for streams. One model covers bounded (batch) and unbounded (streaming) input. Dataflow takes that graph, fuses steps, autoscales workers, handles shuffle and state (through Dataflow Shuffle for batch and Streaming Engine for streaming), and retries failed work.

**Choosing the SDK.** Java has the widest set of I/O connectors and the most mature performance; Python is the common choice for teams in data science and ML, and can use Java connectors through cross-language transforms; Go is smaller. Spotify built Scio, a Scala API over Beam, and runs it on Dataflow. For recurring jobs, **templates** (Flex Templates) package a pipeline so it can be launched with parameters from Composer or the console without a build environment.

The portability claim has limits worth saying: runner capability differs, some Dataflow features (Streaming Engine, at-least-once streaming mode, right-fitting) have no equivalent elsewhere, and a pipeline tuned for Dataflow's autoscaler may behave differently on Flink.

The reasoning

**Exactly-once mode** is the default for streaming jobs. Dataflow acks Pub/Sub messages only after the first fused stage has processed them and persisted the results, removes duplicate deliveries by message ID (or by an ID attribute you configure, if the publisher sets one consistently, within ten minutes), and commits state and output of each step atomically so a retried bundle does not double-count. Streaming Engine holds that state and the shuffle in Google's backend rather than on worker disks.

What it does not do: make a side effect in a `DoFn` happen once. If your code calls an HTTP API or writes to a database directly, a retried bundle calls it again. Exactly-once reaches external systems only through sinks that cooperate — for example BigQuery via the Storage Write API in exactly-once mode — or through writes you make idempotent on a key.

**At-least-once mode** (the `streaming_mode_at_least_once` service option) skips the de-duplication and some checkpointing, and Google's docs say it can significantly reduce cost and latency. Choose it when the sink de-duplicates anyway (a MERGE on a key, a Bigtable row keyed by entity), when the output is itself at-least-once so paying for exactly-once upstream buys nothing, or for approximate metrics. The duplicate rate depends on how often work is retried, so test under realistic load.

The reasoning

**Clusters.** You choose machine types, the number of primary workers and optional secondary workers (preemptible or Spot by default, so cheap but reclaimable), image version, optional components such as Jupyter or Trino, initialisation actions and an autoscaling policy. Data lives in Cloud Storage rather than HDFS, so clusters can be **ephemeral**: create per job, run, delete — Composer and workflow templates do this routinely. Clusters suit interactive notebooks shared by a team, jobs needing custom images, GPUs or Hadoop components, and steady 24-hour workloads.

**Serverless.** You submit a batch (`gcloud dataproc batches submit pyspark …`) and Google provisions and autoscales the executors. Billing is per second on Data Compute Units plus shuffle storage, with a one-minute minimum, in standard or premium tiers, and nothing is charged when nothing runs. You give up node-level control and some tuning knobs, and start-up takes longer than submitting to a warm cluster.

The names moved in 2026: Google now calls the service **Managed Service for Apache Spark**, with cluster and serverless deployment models, and the serverless side was previously "Google Cloud Serverless for Apache Spark" and before that "Dataproc Serverless". Using the old name in an interview is fine; knowing it changed is a signal you are current.

The reasoning

The operators come from `apache-airflow-providers-google`, which Composer environments include. For a cluster: a create task (cluster config, region, a name that includes the run date), a submit task with a job spec pointing at the PySpark file in Cloud Storage, and a delete task. For serverless: one `DataprocCreateBatchOperator` with a batch id and the same file, and no cluster to clean up at all.

The failure handling is the actual answer. Default trigger rules mean a failed submit task marks the delete task `upstream_failed`, and the cluster runs until someone notices the bill. Set `trigger_rule="all_done"` on the delete, or set the cluster's idle-delete TTL as a second guard. Give the batch or cluster a deterministic id per run so a retried task does not create a duplicate; Dataproc rejects a batch id that already exists. Long Spark jobs should use the operators' deferrable mode so they do not hold an Airflow worker slot while waiting.

And the naming note: Cloud Composer is now Managed Service for Apache Airflow (Composer 2 and 3 map to Gen 2 and Gen 3). The operators and DAG code are unchanged.

The reasoning

A task pushes an XCom explicitly or by returning a value (the TaskFlow API does this for you), and a downstream task pulls it by task id and key, or receives it as a function argument. Templated fields can read it too, as in `{{ ti.xcom_pull(task_ids="extract") }}`. In Composer the metadata database is a managed Cloud SQL instance, so every XCom is a row there.

Use it for: the Cloud Storage path a task wrote, the Dataproc batch id to poll, a row count to assert on, a watermark value. Do not use it for: rows of data, large JSON, anything measured in megabytes — it bloats the database, slows the scheduler and the UI, and has size limits that depend on the database. If values really must be large, a custom XCom backend can store them in Cloud Storage and keep only a reference in the database.

The subtler trap: XComs are per DAG run and task, so a task that reads "the latest" XCom from another DAG is coupling two schedules invisibly. Cross-DAG dependencies are better expressed through datasets (assets) or sensors.

The reasoning

**Batch.** A nightly export to Cloud Storage and a load job, a federated query from BigQuery straight into Cloud SQL, or a scheduled transfer. Simple, but it sees only the state at export time — deletes and intermediate updates are invisible, and a full dump of a large table every night is slow and hard on the source.

**CDC with Datastream.** Datastream is serverless. You create connection profiles (with private connectivity to the database) and a stream; it backfills existing rows, then reads the change log — the MySQL binary log in row format, PostgreSQL logical replication through a replication slot and publication, Oracle redo logs, SQL Server change tables — and delivers changes continuously. In BigQuery, tables with a primary key use **merge** mode, so the table mirrors the source; tables without one are **append-only** with a deletion flag. A `max_staleness` setting controls how often background merges run, trading freshness against the compute they consume.

Things to know before promising it: the source must keep enough change log for Datastream to catch up after an outage, adding or removing a primary key on a table already replicated to BigQuery is not supported by default, and merge jobs use slots, which Google suggests covering with a reservation for heavy CDC.

The reasoning

Unlike tape-style archives, even Archive objects are readable immediately through the same API; there is no restore step. What changes is cost shape. Colder classes charge less per GB stored and more per GB read, and each has a **minimum storage duration**: delete or rewrite a Coldline object after 20 days and you are billed as if it stayed 90. So a class is a bet on access frequency and lifetime, not only on price.

Choosing: raw landing data that pipelines read for the first weeks — Standard. Monthly reprocessing inputs — Nearline. Quarterly audits — Coldline. Seven-year regulatory retention — Archive, with a retention policy (and optionally a bucket lock) so it cannot be deleted early. Location type — region, dual-region or multi-region — is a separate choice about availability and where compute runs; keep data in the same region as the Dataproc or BigQuery jobs that read it.

**Lifecycle rules** move objects between classes by age and delete them at end of life, and **Autoclass** moves objects automatically based on access, which suits buckets with unpredictable access patterns. Class changes are operations billed per object, so a bucket of millions of tiny files costs more to tier than its bytes suggest.

The reasoning

Cloud Storage has two access systems. **IAM** grants roles on the project or bucket. **ACLs** are the legacy per-bucket and per-object grants. Mixing them makes access impossible to audit, because an object can be readable through an ACL nobody remembers. **Uniform bucket-level access** disables ACLs so IAM is the only path; it is Google's recommendation and can be reversed within 90 days of enabling it.

With IAM: one service account per pipeline, with `objectViewer` on the buckets it reads and `objectCreator` or `objectAdmin` on the ones it writes — not `storage.admin` on the project. IAM Conditions can restrict a grant to a prefix (`resource.name.startsWith(...)`) when a shared bucket cannot be split. **Public access prevention** blocks `allUsers` grants regardless of IAM mistakes. Signed URLs give time-limited access to one object without granting a role.

Around it: data is encrypted at rest by default, with customer-managed keys in Cloud KMS when policy demands control of the key; audit logs record data access if data-access logging is enabled; and VPC Service Controls stop a valid identity from copying data to a bucket outside the organisation.

The reasoning

**Cloud SQL** is the default for an application database: familiar engines, joins, transactions, read replicas. It scales vertically, so write throughput and size are bounded by one primary. **Spanner** keeps SQL, schemas and ACID transactions but splits data across nodes automatically and replicates synchronously, including across regions, with externally consistent reads. You choose it when one primary is not enough, or when you need multi-region writes with strong consistency — and you design keys to avoid hotspots, since a monotonically increasing primary key concentrates writes.

**Bigtable** has no SQL joins and no secondary indexes in the relational sense; every lookup is by row key or key range, and transactions are single-row. In exchange it sustains very high write rates with single-digit-millisecond reads when the key is designed well — for example `user_id#reverse_timestamp` for "latest events per user". A key starting with a timestamp sends all writes to one node.

And BigQuery is none of these: it is for analytics over large scans, not for serving a row to an application in milliseconds. The usual GCP architecture uses two: Bigtable or Spanner for the app, BigQuery for analysis, with Dataflow or Datastream in between.

The reasoning

A traditional database such as PostgreSQL stores rows, uses indexes to find a few of them quickly and handles many small transactions. BigQuery is built for the opposite workload: scanning billions of rows to aggregate a few columns. It stores each column separately (in Capacitor, on Google's Colossus file system) and runs each query across many slots on the Dremel engine, so a scan of terabytes takes seconds and you never provision servers.

What you get: SQL (GoogleSQL) with nested and repeated types, partitioning and clustering instead of indexes, streaming and batch ingestion, and pay-per-use pricing — on-demand at a US list price of $6.25 per TiB scanned, or capacity pricing per slot-hour. What you give up: fast single-row lookups, high-rate updates of individual rows and enforced constraints; primary and foreign keys can be declared but are not enforced.

So the honest one-liner is "OLAP, not OLTP": the application keeps its transactional database, and BigQuery is where that data is analysed.

The reasoning

**Partitioning** works on one column: a time-unit column (hourly, daily, monthly or yearly), ingestion time, or an integer range. A table can have up to 10,000 partitions. Pruning happens before the query runs, so a dry run shows the exact bytes, and you can set `require_partition_filter`, expire old partitions and overwrite a single partition. It suits low-cardinality, time-shaped filters.

**Clustering** sorts rows by up to four columns, in order, and BigQuery keeps min and max values per storage block. A filter on the first clustering column lets it skip blocks — block pruning. It handles high-cardinality columns like `customer_id` that partitioning cannot, and BigQuery reclusters automatically as data arrives. The trade-off is that the number of blocks read is known only during execution, so the pre-run estimate is an upper bound and the real bill is usually lower.

Most large tables use both: partition by date, cluster by the columns most queries filter on.

The reasoning

A **topic** is a named stream. A **subscription** is one consumer group's view of it: every subscription gets every message, and within a subscription each message goes to one of the subscribers. Delivery is **pull** (the subscriber asks, usually through StreamingPull), **push** (Pub/Sub calls an HTTPS endpoint) or an **export** subscription that writes straight to BigQuery or Cloud Storage.

The contract: at-least-once delivery by default, redelivery when an ack deadline (10 seconds by default) expires, unacknowledged messages retained for seven days by default, ordering only per ordering key when enabled, and optional exactly-once delivery on pull subscriptions in one region. There are no partitions to size and it scales without provisioning.

Use it for event ingestion from apps and devices, fan-out of one event to several consumers, and buffering spikes in front of a processor. Compared with Kafka, it trades partition-level control and long log retention for no operations.

The reasoning

Composer runs Airflow on Google-managed infrastructure: you create an environment, choose its size and Airflow version, and deploy DAG files by copying them to the environment's `dags/` folder in Cloud Storage. Composer 3 environments (Gen 3) offer Airflow 3 builds as well as Airflow 2; Composer 2 (Gen 2) runs Airflow 2. Google maintains the images and their support dates.

A **DAG** declares tasks (operators such as `BigQueryInsertJobOperator` or `DataprocCreateBatchOperator`), the dependencies between them (`extract >> transform >> load`), a schedule, and settings like retries and catchup. Acyclic means no loops: a task cannot depend on itself through others. Good DAG design keeps tasks idempotent per run date, small enough to retry cheaply, and free of heavy work in the file's top-level code, which the scheduler parses repeatedly.

Composer orchestrates; it should not process data itself. The heavy lifting happens in BigQuery, Dataflow or Dataproc, and the DAG waits for them.

The reasoning

**Dataflow**: you write Beam (Java, Python, Go); Google runs, scales and retries it. One model for batch and streaming, event-time windows, watermarks and triggers, Streaming Engine for state. No cluster to manage, and you pay for workers while the job runs.

**Dataproc** (now Managed Service for Apache Spark): Spark, PySpark, Hive and the Hadoop ecosystem on clusters you configure, or as serverless batches. Best when you are migrating existing Spark or Hadoop jobs, need Spark libraries or notebooks, or want control of the runtime.

The usual rule: new streaming pipelines on Dataflow; existing or Spark-native batch on Dataproc; and pure SQL transformations in BigQuery with neither.