GCP Theoretical

The reasoning

A read session is created for a table with the columns and an optional filter; it returns a set of streams that a client reads in parallel. Each stream yields Arrow record batches, which Spark converts to partitions without a serialisation hop. Snapshot time is fixed at session creation, so a read is consistent.

The alternative, exporting to Cloud Storage and reading files, is slower, costs an export job, and leaves copies. Paging through query results with the jobs API is limited in throughput and shape. The Storage Read API replaced both for bulk reads.

Pricing is separate from queries: bytes read through the API are billed at a per-TiB rate, with a free monthly allowance, and column projection and filters reduce it. Reading a view or an external table is not supported directly; the connector runs a query to materialise first, which costs query bytes.

The companion is the Storage Write API, the current high-throughput ingestion path with exactly-once semantics through committed streams, replacing the legacy streaming inserts.

The reasoning

Full names are `project.dataset.table`. The project that runs a query is the one billed for it. That is how a central data project can expose datasets to consumer projects that pay for their own queries. Quotas, such as concurrent query limits, are per project.

Datasets carry access control and defaults. IAM roles granted on a dataset apply to every table in it; table-level and column-level controls narrow further. A dataset can set default table expiration and default partition expiration, and it can be shared through Analytics Hub as a listing.

Location is the constraint people hit. A dataset in `europe-west2` cannot be joined with one in `US`, and a query that references both fails. Copying a dataset across regions is a Data Transfer Service job, not a `CREATE TABLE AS`. Multi-regions, `US` and `EU`, are the common choice for flexibility within a continent.

Tables are typed: native tables in BigQuery storage, external tables over Cloud Storage or other clouds, BigLake tables, views, materialized views, snapshots and clones. Time travel, partitioning and clustering are table properties.

The reasoning

Dashboards fire the same shape of query many times: filter by date and region, group by a few columns, sum. Running each as a full BigQuery job costs slots and seconds. BI Engine keeps the hot columns in RAM and answers those queries with vectorised execution, bypassing the distributed engine.

The reservation is the control. A project reserves, say, 50 GB in `US`. Queries are accelerated when the data they touch fits in that memory, and fall back to the normal engine when they do not. The query plan shows whether BI Engine was used and, if not, the reason, such as unsupported SQL or insufficient memory.

Acceleration is partial by design: BigQuery accelerates the stages it can and runs the rest normally, so a query may be fast in parts. Supported functions and join shapes are listed in the docs; most dashboard SQL qualifies. Materialized views and BI Engine combine well, the view shrinking what must be cached.

Cost is per gigabyte of reservation per hour, independent of queries. It is worth it when a dashboard audience is large and the queried data is hot and small. A 20 TB fact table is not what it caches; the aggregate over it is.

The reasoning

`CREATE MATERIALIZED VIEW ds.daily_sales AS SELECT dt, region, SUM(amount) FROM ds.sales GROUP BY 1, 2`. BigQuery refreshes it in the background, within `max_staleness` or at a default interval. At query time it merges the view with recent base-table changes, so results are always current even between refreshes.

Smart tuning is the distinctive part. A dashboard query `SELECT region, SUM(amount) FROM ds.sales WHERE dt = ...` is rewritten to read the view, cutting bytes billed and time, without anyone changing the dashboard. The query plan shows the rewrite.

Costs are the refresh, billed as query bytes or slots, plus the view's storage. Base tables that churn heavily make refresh expensive; setting `max_staleness` lets a view serve slightly stale results without a merge, trading freshness for cost.

Limits that shape use: inner joins are allowed only in defined patterns, and there is no `UNION`, no window function and no `HAVING` on non-aggregated columns. Partitioning aligns with the base table. A view over an external table requires BigLake with metadata caching. Clusters and partitions on the view itself are allowed.

The reasoning

You write `INSERT INTO ds.daily_agg SELECT ... WHERE dt = DATE(@run_date) - 1`, pick a schedule such as every day 06:00, choose a service account to run as, and optionally a destination table with a `{run_date}` template. Each run is a normal query job, billed as usual, visible in the transfer service's run history.

Backfills are built in: you can schedule a run for a range of past dates and `@run_date` takes each value, which is how a corrected transformation is replayed. Notifications on failure go to Pub/Sub or email.

The limits: a scheduled query knows nothing about other scheduled queries, so a chain of four must be timed by hand with gaps. A failure in step two does not stop step three. There is no lineage, and it is SQL only, with no external calls.

Dataform, built into BigQuery, adds the dependency graph with `ref()`, assertions and versioned SQLX. Cloud Composer adds orchestration across services. The usual progression is scheduled queries for one or two tables, Dataform once there is a graph.

The reasoning

`SELECT * FROM ds.orders FOR SYSTEM_TIME AS OF TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 2 HOUR)` reads the pre-mistake state. `bq cp ds.orders@-7200000 ds.orders_restored` copies it to a new table using a milliseconds-ago decorator. A dropped table can be restored the same way within the window.

The window is a dataset setting, `max_time_travel_hours`, between 48 and 168. The default is the maximum. The reason to shorten it is cost: with physical storage billing, every deleted or overwritten byte is stored for the whole window, so a table rewritten daily holds seven copies.

Fail-safe adds seven days you cannot reach yourself. It exists for disaster recovery: a support case can restore data up to 14 days after deletion with the default window. It is also billed under physical storage.

Time travel is per table state, not per row lineage, and it does not survive a dataset deletion beyond the window. For long-term point-in-time needs, table snapshots, which are cheap and keep a state indefinitely, are the tool.

The reasoning

`CREATE TABLE ds.events (...) PARTITION BY DATE(event_ts) OPTIONS (partition_expiration_days = 90, require_partition_filter = TRUE)`. Each day, partitions older than 90 days disappear without a job, and a query without `WHERE DATE(event_ts) ...` fails with an error instead of scanning everything.

Expiration is per partition, not per row: a partition's age is its boundary date or ingestion time. Deleted partitions enter time travel like any deletion, so storage drops after the window. Changing the setting applies to existing partitions immediately.

The filter requirement has a subtlety: the filter must be on the partition column itself, or on `_PARTITIONTIME` for ingestion-time tables. A filter on a function of the column that BigQuery cannot prune may not satisfy it, and a filter through a joined table does not count.

Dataset-level `default_partition_expiration_days` applies to new tables only. Table expiration, a different setting, deletes the whole table at a time. Clustering is independent and continues to work within surviving partitions.

The reasoning

An ingestion-time partitioned table has no user column for the partition; BigQuery assigns rows to the partition of the time they were loaded, by day, hour, month or year. `SELECT * FROM ds.raw WHERE _PARTITIONTIME = TIMESTAMP('2026-11-01')` reads one partition. `_PARTITIONDATE` is the DATE form for daily tables.

You can load into a specific partition with a decorator, `ds.raw$20261101`, which is how a backfill writes yesterday's data into yesterday's partition rather than today's. Querying with a decorator reads just that partition.

`__NULL__` appears on column-partitioned tables: a row whose partition column is null cannot be placed in a date partition, so it lands in `__NULL__`. Those rows are read by queries without a partition filter and invisible to queries with one, which is a classic missing-data bug.

`__UNPARTITIONED__` holds rows in the streaming buffer before they are partitioned, and rows whose partition value is outside the allowed range. Queries filtering on the partition column may not see them for a few minutes after streaming, which matters for near-real-time checks.

The reasoning

Columnar storage means a query pays only for the columns it references across the partitions it touches. `SELECT *` on a 100-column table reads everything; `SELECT user_id, amount` reads two columns. The dry-run flag, or the estimate in the console, gives bytes processed before you pay.

`LIMIT` does not reduce bytes billed on a native table, because the scan happens before the limit. Clustering does: when a filter hits clustered columns, BigQuery skips blocks, and the actual bytes billed are lower than the estimate, which cannot account for clustering. A partition filter reduces the estimate itself.

The 10 MB minimum per table means a query touching 50 tiny lookup tables bills at least 500 MB. Cached results are free when the same query repeats and the underlying tables have not changed. Storage is billed separately, per GiB-month, with a lower rate for data untouched for 90 days.

Editions, Standard, Enterprise and Enterprise Plus, price slots per hour with autoscaling between a baseline and a maximum, and one- or three-year commitments discount them. The switch makes sense when monthly on-demand spend exceeds what a steady slot reservation would cost, roughly when queries are frequent and predictable.

The reasoning

Pull with the client libraries opens a streaming connection, receives batches, and acks each message when processing is done. Flow control settings cap outstanding messages. The client library extends the ack deadline automatically for messages still in flight, up to a configured maximum.

Push removes the need for a running consumer. Pub/Sub POSTs to the endpoint, which must be HTTPS and reachable; a 2xx response acks, anything else or a timeout causes redelivery with exponential backoff. Push rate adapts to the endpoint's success rate. Authentication uses a service account token Pub/Sub attaches.

A message not acked within the deadline is redelivered, possibly to another subscriber, which is the at-least-once guarantee and the source of duplicates. Exactly-once delivery, available on pull subscriptions in a single region, suppresses redelivery of acked messages. Consumers should still be idempotent.

Retention keeps unacked messages for 7 days by default, up to 31, and acked messages can be retained too for seek and replay. Dead-letter topics receive messages redelivered more than a maximum number of times. Filters on attributes let a subscription receive a subset of a topic.

The reasoning

You create the subscription with a destination table and options: use topic schema, write metadata, drop unknown fields. With `use_topic_schema`, an Avro or Protocol Buffer schema on the topic defines the columns; a message that does not fit goes to a dead-letter topic if configured. With `use_table_schema`, the JSON message is matched to the table's columns instead.

Delivery is at-least-once into the table, so duplicates are possible on retries, and the metadata message id is the key for downstream deduplication. There is no ordering guarantee into the table and no transformation step, which is the point and the limit.

Cost is Pub/Sub's per-TiB charge for the subscription plus BigQuery storage; there is no Dataflow job to run and no streaming insert fee beyond the Storage Write API. For a raw landing zone, that is a large saving.

The comparable option is a Cloud Storage subscription, which writes batches of messages to files in a bucket. Both are the "land it first" pattern; Dataflow remains for anything that must happen to the data in flight.

The reasoning

Without Streaming Engine, each worker holds the state of its keys on its persistent disk, and rescaling means redistributing that state, which is slow. With it, state lives in the service, and adding a worker is just adding compute. Streaming Engine is on by default for new streaming jobs in most regions.

Scaling signals: a streaming job watches the backlog in seconds, how long it would take to drain unprocessed input at current throughput, and CPU. If backlog grows, workers are added up to `maxNumWorkers`; if workers idle, they are removed. Batch jobs estimate total work from input size and progress.

The controls are `--autoscalingAlgorithm`, `--maxNumWorkers`, `--numWorkers` for the start, and worker machine type. Setting the maximum too low leaves lag; too high lets a hot period spend. Streaming Engine is billed per unit of data processed in the service, offsetting lower VM cost.

Dataflow Prime extends this with vertical autoscaling, raising a worker's memory when a stage runs out, and right fitting, assigning different resources per stage. It is the answer to out-of-memory in a ParDo without resizing the whole fleet.

The reasoning

The pipeline reads a source into a PCollection, applies transforms, and writes a sink. `events = p | ReadFromPubSub(topic) | beam.Map(json.loads) | beam.ParDo(EnrichFn()) | beam.WindowInto(FixedWindows(60)) | beam.CombinePerKey(sum) | WriteToBigQuery(table)`. Each pipe produces a new PCollection; nothing is mutated.

`ParDo` is the general worker. The DoFn's `process(self, element)` can yield several outputs, output to tagged side outputs for routing, read side inputs for lookups, and use state and timers for per-key logic. `Map`, `FlatMap` and `Filter` are ParDos with a lambda.

Unbounded PCollections, from Pub/Sub or Kafka, are where windows and watermarks enter. A GroupByKey over an unbounded collection needs a window to know when a key's group is complete, and the watermark says when the window can fire. Bounded collections, from files, use the global window.

The same code runs on the DirectRunner for tests, Dataflow for production, and Flink or Spark through their runners, which is Beam's portability promise. Python, Java and Go SDKs exist; cross-language transforms let a Python pipeline use a Java connector.

The reasoning

The build step takes your pipeline code and a base image, produces a container, pushes it to Artifact Registry, and writes the spec file with metadata and parameter definitions. The launch step passes parameters such as input topic and output table, and Dataflow handles the rest.

Classic templates staged a serialized graph; anything that shaped the graph, such as the number of sources or a conditional branch, was frozen at staging. Flex Templates run the construction code at launch, so the same template can produce different graphs for different parameters, and runtime parameters need no special types.

Launch latency is higher, because the launcher VM boots and builds the graph before workers start, typically a minute or two. Google's own provided templates, such as Pub/Sub to BigQuery, are published as Flex Templates.

Operationally, Flex Templates are how Composer, Cloud Scheduler or a CI pipeline starts Dataflow jobs: `DataflowStartFlexTemplateOperator` in Airflow passes parameters and waits. Versioning the image tag gives reproducible deployments.

The reasoning

The policy says, for example, primary workers 2 to 2 and secondary preemptible workers 0 to 50. It scales up by 1.0 of needed capacity and down by 0.5, with a 2-minute cooldown and a 1-hour graceful decommission. Every cooldown period Dataproc checks YARN's pending memory and adds or removes workers accordingly.

Graceful decommission lets a worker finish its running YARN containers before leaving, which avoids task failures on scale-down. Secondary workers hold no HDFS data, so they can leave freely. Primary workers do, so scaling them down is slower and risky, which is why policies usually fix primaries and scale secondaries.

Ephemeral clusters come from Dataproc Workflow Templates or from an orchestrator: create cluster, submit job, delete cluster, in one template or three Airflow tasks. Cluster scheduled deletion, `--max-idle`, is the safety net that deletes a cluster idle for N minutes. Data lives in Cloud Storage, not HDFS, so a cluster holds nothing to lose.

Dataproc Serverless for Spark removes the cluster entirely: submit a batch, Dataproc provisions executors, bills per second of compute used, and tears down. It is the default suggestion for Spark jobs that need no cluster-level customisation. Billing for clusters is per second with a one-minute minimum, plus Compute Engine.

The reasoning

A stream has a source connection profile, a destination connection profile, and a list of objects to include or exclude. On start it can backfill every included table, then switch to CDC from the log position captured at the start, so no change is missed between the two.

The source must be prepared: binary logging in row format on MySQL, logical replication with a publication and replication slot on PostgreSQL, LogMiner or Binary Reader on Oracle. Datastream needs a user with replication privileges and network access through a private connection, allowlisted IPs or a proxy.

To Cloud Storage, events are written as Avro or JSON files with the source metadata, change type and timestamp, which a Dataflow template or Spark job merges into tables. To BigQuery, Datastream creates the tables and applies changes itself, using the Storage Write API's CDC support. `max_staleness` on the target controls how often the merge happens; a lower value costs more.

Backfill is heavy on the source, so it is often run per table during low traffic and for very large tables replaced by a Dataflow or export-based initial load. Monitoring shows freshness, throughput, and the backfill state per object.

The reasoning

A lake is a logical domain, such as sales. Zones inside it separate raw from curated, with different type rules. Assets attach Cloud Storage buckets and BigQuery datasets to zones. Discovery scans the buckets and registers tables with inferred schemas, including Hive-style partitions, into BigQuery and the catalog.

Data quality scans run rules, such as not null, uniqueness, range and SQL assertions, on a schedule and publish results to the catalog and to BigQuery. Data profiling scans compute column statistics. Both are serverless and are the GCP answer to "how do you monitor data quality" without a third-party tool.

Lineage is captured automatically from BigQuery jobs, Dataflow and Composer, and shown as a graph of which tables fed which. Aspects, the successor to Data Catalog tags, attach structured metadata such as owner, sensitivity and SLA to entries, and are searchable.

Policy integration: Dataplex can apply IAM to assets by zone, and sensitive-data classification from Sensitive Data Protection can land as aspects. It catalogs; enforcement of column-level security is still BigQuery policy tags and IAM.

The reasoning

`CREATE EXTERNAL TABLE ds.raw_events WITH CONNECTION proj.region.conn OPTIONS (format = 'PARQUET', uris = ['gs://lake/events/*'], max_staleness = INTERVAL 1 HOUR, metadata_cache_mode = 'AUTOMATIC')` creates a BigLake table. Without `WITH CONNECTION`, it is a plain external table.

The security difference is the point. A plain external table requires every querying user to have `storage.objects.get` on the bucket, so you cannot hide columns or rows. A BigLake table is read by the connection's service account; users get BigQuery grants, policy tags and row access policies as if it were native.

Metadata caching addresses the performance gap. Listing millions of objects on every query is slow; the cache keeps file metadata and lets BigQuery prune by Hive partition and file statistics. `max_staleness` says how old the cache may be; a manual or automatic refresh updates it.

BigLake also covers open table formats: BigLake Iceberg tables in BigQuery, with BigQuery as the catalog and Cloud Storage as storage, readable by Spark through the BigLake metastore. Object tables bring unstructured files into SQL with their metadata. Plain external tables remain for quick, trusted, ad hoc reads.

The reasoning

Attaching a service account to a resource is the normal path. A Dataflow job runs as its worker service account, and a Composer environment as its environment service account. Each needs only the roles its work requires: BigQuery Data Editor on a dataset, Storage Object Viewer on a bucket.

A downloaded key never expires on its own, is copied into CI systems and laptops, and shows up in leaked repositories. Organisation policies can disable key creation, and most teams do so. The alternative for humans is impersonation: a user with the Service Account Token Creator role mints a short-lived token for the service account, which is audited.

Workload Identity on GKE binds a Kubernetes service account to a Google service account. A pod's token is exchanged for a Google access token by the metadata server, with no secret in the cluster. Workload Identity Federation extends the same exchange to external identity providers through a workload identity pool, letting GitHub Actions deploy with no stored key.

Service accounts are also resources: `iam.serviceAccountUser` lets someone attach one, and `serviceAccountTokenCreator` lets them act as it, both of which are effectively grants of the account's power. Auditing those bindings is part of least privilege.

The reasoning

The threat is a credential that is valid but misused: a stolen service account key, or an insider copying a BigQuery table to a personal project. IAM allows it, because the identity has the role. A perimeter around the data projects denies the API call, because the destination or the caller is outside.

Protected services are listed per perimeter; a service not listed is not restricted. Access levels from Access Context Manager define trusted conditions, such as corporate IP ranges or device policy, and are referenced by ingress rules. Dry-run mode logs what would be blocked before enforcement, which is how perimeters are rolled out.

Egress rules allow specific flows out, such as writing to a partner's bucket. Ingress rules allow specific flows in, such as a Composer environment in a shared project calling BigQuery inside. Perimeter bridges connect two perimeters. Misconfigured rules are the commonest cause of mysterious 403s on otherwise correct IAM.

Private Google Access and Private Service Connect let VPC workloads reach Google APIs without public IPs. That pairs with perimeters so that the only path to the data is from inside. Logs for perimeter violations appear in Cloud Audit Logs with a unique id you can look up.

The reasoning

Objects are immutable blobs with metadata; an overwrite creates a new generation, and versioning keeps old generations. There are no directories: the slash is part of the name, and listing by prefix is how folders are simulated, as on S3. Hierarchical namespace buckets, a newer option, add real folder operations for analytics workloads.

Strong consistency for all operations means a file written by Spark is visible to the next job immediately, including in listings. That removed a class of eventual-consistency bugs that other object stores once had.

Classes matter for a lake: raw landing in Standard, older partitions moved to Nearline or Coldline by lifecycle rules, compliance archives in Archive. Retrieval from colder classes costs per GB, and deleting before the minimum duration still bills the minimum. Autoclass automates the moves for unpredictable patterns.

Access is through IAM on buckets, uniform bucket-level access recommended, with signed URLs for temporary access and VPC Service Controls for perimeters. The gsutil tool is superseded by `gcloud storage`; the Storage Transfer Service moves data in bulk from S3 or on premises.

The reasoning

A Dataflow job takes a Beam pipeline and runs it on worker VMs that Dataflow provisions, scales and removes. There is no cluster to create. Billing is per second of worker vCPU, memory and disk, plus Streaming Engine and shuffle service usage.

Its strengths are streaming: exactly-once processing within the pipeline, event-time windowing with watermarks and late data handling, and stateful per-key processing, all from Beam's model. Pub/Sub to BigQuery with deduplication and windowed aggregation is the canonical job.

Dataproc runs open-source Spark, Flink, Presto and Hadoop on clusters you define, or Spark on Serverless. Spark's ecosystem, Delta and Iceberg support, MLlib, notebooks and the ability to lift existing jobs unchanged are its reasons. It is batch-first in practice, though Spark Structured Streaming runs on it.

The decision usually follows the team: Beam skills and streaming requirements favour Dataflow; Spark skills and existing code favour Dataproc. Google's own guidance positions Dataflow as the default for new streaming pipelines and Dataproc for migrated Spark workloads.

The reasoning

A cluster has a master, primary workers and optional secondary workers that are often preemptible or Spot VMs for cost. Initialisation actions and custom images install dependencies. Component gateway exposes the Spark UI and Jupyter without SSH. The Cloud Storage connector makes `gs://` paths behave like HDFS.

Jobs are submitted through the API, `gcloud dataproc jobs submit`, Workflow Templates that create a cluster, run steps and delete it, or Airflow operators. The Dataproc Metastore service provides a managed Hive metastore shared by clusters, and BigLake metastore is the newer alternative for Iceberg.

Serverless for Spark takes a Spark application and runs it with autoscaling executors, no cluster lifecycle and no idle cost. It is the default for scheduled batch Spark; clusters remain for interactive work, non-Spark engines, and jobs needing specific cluster configuration.

Cost control follows from design: ephemeral clusters or Serverless, Spot secondary workers, autoscaling policies, and `--max-idle` deletion. Data in Cloud Storage means a cluster is disposable.

The reasoning

A Bigtable table has one index: the row key. Rows are sorted by key, and performance comes from designing the key so that reads are contiguous ranges and writes spread across nodes. A key such as `device#reversed_timestamp` avoids hotspotting on the newest data while keeping a device's history together.

Columns are grouped in column families, and each cell is versioned by timestamp. There are no joins, no secondary indexes except through separate tables, and no transactions beyond a single row. Garbage collection policies expire old versions.

Capacity is nodes per cluster; each node serves a throughput of reads and writes, and replication across clusters gives availability and locality. Autoscaling adjusts nodes on CPU and storage. Integration with Dataflow and BigQuery external queries connects it to analytics.

The choice: Bigtable for key-based access at huge scale and high write rates. Spanner when you need SQL, secondary indexes and strong consistency across regions with transactions. Cloud SQL when a single-region relational database of a few terabytes is enough and compatibility matters. Firestore is the document option for application data.

The reasoning

An instance has a machine type, storage that can auto-grow, a region and zone, and optional high availability with synchronous replication to a standby. Point-in-time recovery uses backups plus transaction logs. Connections go through private IP, the Cloud SQL Auth Proxy or the connectors, which handle TLS and IAM authentication.

For analytics, the rule is not to query the primary. Read replicas take reporting and extraction load. Datastream reads the log, so it adds little load and captures deletes; it requires logical replication on PostgreSQL or binary logging on MySQL, which are instance flags.

BigQuery federated queries, `EXTERNAL_QUERY('connection', 'SELECT ...')`, run SQL on the Cloud SQL instance from BigQuery and return results, suitable for small reference tables and not for bulk. Exports to Cloud Storage as CSV or SQL dump are the simple nightly path.

Limits that shape design: instance size caps storage at tens of terabytes, connection counts are finite, and long analytical queries hurt the application. The pattern is operational data in Cloud SQL, analytics in BigQuery, connected by CDC.

The reasoning

Granting `roles/bigquery.dataViewer` on a project gives read on every dataset in it; granting it on one dataset limits it there. Because inheritance is additive, a role granted at the folder level cannot be removed at the project level, which is why broad grants high in the hierarchy are dangerous.

Basic roles are too coarse for production: Editor includes thousands of permissions across all services. Predefined roles follow least privilege per service, and custom roles pick exact permissions when none fits. Conditions add attributes, such as resource name prefixes or time, to a binding.

Deny policies, newer, override allows for listed permissions and principals, giving the explicit-deny concept. Organisation policy constraints restrict what resources can be created or configured, such as disabling service account key creation, separate from IAM.

For data access specifically, BigQuery adds dataset, table, row and column controls on top of IAM; Cloud Storage uses IAM on buckets with uniform access recommended. Policy Analyzer and IAM Recommender show who has access and which grants are unused, which is how least privilege is maintained.

The reasoning

Every API call targets a project, and the project's enabled APIs, quotas and billing apply. BigQuery query quotas, Dataflow job limits and Pub/Sub throughput quotas are per project, so a busy shared project can throttle unrelated teams.

Separating storage from compute projects lets data live in a few governed projects while many teams run their own jobs and pay their own bills. BigQuery's model supports this directly: the querying project pays, the data project only grants access.

Folders group projects for policy inheritance and billing roll-up: a data platform folder with prod and non-prod subfolders is typical, with organisation policies applied at the folder. Shared VPC lets a host project own the network that service projects attach to.

Project ids are permanent and globally unique, and deleting a project schedules it for removal after 30 days. Labels on projects drive cost allocation reports in Billing export to BigQuery, which is itself a dataset in a project.

The reasoning

Audit logs, admin activity and data access, record who did what to BigQuery and Cloud Storage, and are the source for access reviews. Data access logs for BigQuery are on by default; for Cloud Storage they must be enabled. A sink of audit logs to BigQuery makes them queryable with SQL.

Log-based metrics turn a filter such as `severity=ERROR AND resource.type=dataflow_step` into a counter you can alert on. Dataflow emits its own metrics, system lag and backlog, which Monitoring charts; Pub/Sub exposes oldest unacked message age, the freshness alert for any streaming pipeline.

Alerting policies compare a metric against a threshold over a window and notify Slack, PagerDuty, email or Pub/Sub. Composer surfaces task failures as metrics and logs per DAG. Uptime checks probe endpoints. SLOs can be defined on custom metrics for data freshness.

Retention and cost are controlled with log buckets, exclusion filters for noisy logs, and sinks to cheaper storage. The Logging and Monitoring pair is what Google calls the Cloud Operations suite, formerly Stackdriver.

The reasoning

A SQLX file looks like `config { type: "table", schema: "marts" } SELECT ... FROM ${ref("stg_orders")}`. Types include table, view, incremental and assertion. Incremental tables use a `${when(incremental(), ...)}` guard, comparable to dbt's `is_incremental()`.

Repositories connect to GitHub, GitLab or Bitbucket, or use a Dataform-hosted repo. Development workspaces give each developer their own schema. Release configurations compile a branch; workflow configurations run the compiled graph on a cron or through the API, which Composer can also trigger.

Assertions are SQL that must return zero rows, plus built-in `uniqueKey`, `nonNull` and `rowConditions` in config. Dataform records run history and lineage into Dataplex. Pricing is just the BigQuery queries it runs.

Against dbt: dbt is multi-warehouse, has a vast package ecosystem, Jinja, unit tests, and dbt Cloud or your own orchestrator. Dataform is simpler to adopt on a pure BigQuery stack and needs no infrastructure. Teams with existing dbt projects keep dbt; BigQuery-only teams starting fresh often pick Dataform.

The reasoning

A service is a container image plus a revision configuration: CPU, memory up to 32 GB, concurrency per instance, minimum instances to avoid cold starts, and a service account. Traffic can split between revisions. It is the natural target for a Pub/Sub push subscription or an EventBridge-style Eventarc trigger on Cloud Storage object creation.

A job runs the same kind of image but exits when done. `--tasks 50 --parallelism 10` runs 50 tasks, 10 at a time, each receiving its index in an environment variable, which is a simple way to shard a backfill. Jobs have retries per task and a timeout per task.

Compared with Cloud Functions, now Cloud Run functions, you bring any runtime and system packages in the image. Compared with Dataflow or Dataproc, it is single-container compute with no distributed engine. It fits tasks that a single machine finishes: API extraction, small transforms, dbt or Dataform invocation, notifications.

Cloud Run connects to a VPC through Direct VPC egress or a connector to reach Cloud SQL and private services. It runs as a service account for IAM, and logs to Cloud Logging. Billing to zero when idle is what makes it the glue of choice over an always-on VM.