AWS Scenarios & Design

The reasoning

**The link first.** 10 TB is 80,000 gigabits; at a full 1 Gbps that is about **22 hours**, and real throughput with protocol overhead and a shared link is lower, so plan for well over a day. That is acceptable for an online full load as long as changes are captured during it. If the link cannot be given to the migration, AWS Snowball Edge moves the bulk offline and CDC covers the gap.

**Replication.** Use **AWS DMS** with a **full load plus CDC** task (or CDC starting from a recorded SCN after a Snowball restore). The Oracle source needs **supplemental logging** and archived redo logs kept long enough to cover the full load and any stall — if DMS falls behind the retained logs, the task cannot resume. Split large tables with parallel-load settings, and let DMS cache changes made during each table's load. For a target engine change (Oracle to Aurora PostgreSQL), convert schema and code with the Schema Conversion Tool first; for a lake target, DMS writes Parquet to S3.

**Catch-up and validation.** 40 GB of change a day is small beside the link, so CDC should run at near-zero latency after the load. Turn on **DMS data validation** or compare row counts and checksums per table, and watch the `CDCLatencySource` and `CDCLatencyTarget` metrics — AWS gives no latency SLA for DMS CDC, so you measure it.

**Cut-over within 30 minutes.** Stop application writes, wait for CDC latency to reach zero, run the final validation, repoint the application (DNS or connection strings), and keep reverse replication or the old database read-only as a rollback path for an agreed period.

The reasoning

**Source.** Set `binlog_format = ROW` and `binlog_row_image = FULL` in the parameter group and keep binary logs long enough to survive an outage — `call mysql.rds_set_configuration('binlog retention hours', 24);` — or a task that stalls finds its position purged. Automated backups must be on for RDS to keep binlogs at all.

**DMS to S3.** A **full load plus CDC** task writes the initial 40 million rows, then change files. Each change row carries an **operation column (`I`, `U`, `D`)**; add a commit-timestamp column so changes can be ordered. `cdcMaxBatchInterval` and `cdcMinFileSize` decide how often files appear — a 300-second interval uses a third of the 15-minute budget, and 2,000 changes a second at peak is about 600,000 rows per batch.

**Current state.** Change files are not a table. A Glue or EMR Serverless job every 5 minutes reads new files, keeps the **latest change per `order_id`** by commit timestamp, and runs `MERGE INTO` an **Iceberg** table — matched `D` deletes, matched `U` updates, unmatched `I` inserts. Athena reads Iceberg tables directly. 5 minutes of DMS batching plus a 5-minute merge cycle plus merge runtime fits 15 minutes with margin.

**What goes wrong.** Binlog retention too short; deletes lost by an append-only design; two changes to one order in one batch applied in the wrong order; a schema change on the source that the merge does not expect; and many small Parquet files from short batch intervals, which the Iceberg table needs compacting to avoid. Alarm on `CDCLatencySource` and `CDCLatencyTarget` so latency is measured, not assumed.

The reasoning

**Read and shape.** Read `s3://landing/sales/dt=2026-09-14/` as a DynamicFrame (or DataFrame), cast types, drop malformed rows to a quarantine prefix, and add a `load_date` column. 5 GB of gzip is not splittable, so if it arrives as one file, repartition before heavy transforms or ask the producer for several files.

**Write through the connector.** The Glue Redshift connection writes the frame to the job's **`redshiftTmpDir`** in S3 and issues a **`COPY`**, so the load is parallel across slices. Writing row-by-row over JDBC would take hours for tens of millions of rows.

**Make it idempotent.** Write to `stage.sales` with a **`preactions`** statement that truncates it, and a **`postactions`** block that, inside one transaction, deletes the target rows for the business keys (or the `load_date`) present in staging and inserts from staging — or uses `MERGE`. A rerun of 14 September then replaces 14 September. Redshift does not enforce primary keys, so without this a rerun silently doubles the day.

**Operational details.** The Glue role needs access to the temp directory and the connection; the Redshift role needs to read it. Clear the temporary directory with a lifecycle rule. And ask whether Glue is needed at all: if the transform is trivial, a Redshift `COPY` from S3 into staging and the same merge is simpler and cheaper.

The reasoning

**Shape of the dimension.** `customer_sk` (an `IDENTITY` surrogate key), `customer_id` (the business key), the tracked attributes, `row_hash`, `valid_from`, `valid_to` and `is_current`. Every fact row points at the `customer_sk` that was current when the fact happened.

**Load.** `COPY` the 50,000 rows into `stage.customer_changes` and compute `row_hash` over the tracked columns. Then, **in one transaction**: (1) `UPDATE` the current row of every customer whose hash differs — set `valid_to` to the change date and `is_current = false`; (2) `INSERT` a new current row for every changed customer and every customer not yet in the dimension. Doing it in two statements inside one transaction means a failure leaves nothing half-applied.

**Rerun safety.** Replaying the same file changes nothing: after the first run the current row's hash equals the staged hash, so neither statement matches. Files must be applied **in order of their extract date**; if a late file arrives, reprocess from that date forward.

**Why not Glue.** 50,000 rows a day against 2 million is small, well-suited to Redshift SQL. Glue (or an Iceberg `MERGE` in the lake) is the choice when the dimension lives in S3, or when the change set is too large to stage comfortably in the warehouse.

The reasoning

**Registry.** A DynamoDB table with one item per source: engine, endpoint, secret name, tables, load mode (full, incremental by column, or CDC), watermark, schedule and owner. Adding database 601 is a new item, not new code.

**Extraction.** Relational sources: Glue jobs parameterised by registry item, reading incrementally on an `updated_at` or key watermark, or **DMS tasks** where deletes must be captured. **DynamoDB**: table export to S3 — full or incremental — which reads from backups rather than consuming table throughput. Everything lands in `s3://raw/<source>/<table>/dt=.../` as Parquet.

**Orchestration.** A **Step Functions** state machine reads the registry and fans out with a **Map state** (distributed mode for hundreds of items), with a concurrency limit so 600 databases are not all hit at once. Each branch has `Retry` with exponential backoff and a `Catch` that records the failure and lets the other branches continue.

**Self-healing.** Loads overwrite a partition or merge on a key, so reruns are safe. The watermark moves **only after** the load commits, so the next cycle automatically picks up what a failed run missed. Sources that fail three cycles in a row are marked in the registry, alerted to their owner through SNS, and skipped until fixed. Schema drift is detected by comparing the landed schema with the catalog, and the warehouse load stops for that table rather than corrupting it. A freshness metric per source feeds CloudWatch alarms.

The reasoning

**Arithmetic.** 20,000 devices every 5 seconds is **4,000 records/s**, and at 1 KB each **4 MB/s**, about **345 GB a day** of raw JSON. A provisioned shard takes 1 MB/s or 1,000 records/s, so the floor is 4 shards; with headroom and uneven device clocks, 8, or **on-demand** mode, which starts at 4 MB/s and scales to double the previous peak.

**Ingest.** **AWS IoT Core** handles device identity and MQTT; a rule forwards messages to **Kinesis Data Streams** with `device_id` as the partition key, so each device's readings stay in order.

**Real-time path.** **Managed Service for Apache Flink** reads the stream, computes one-minute windows per device and site, and writes to a low-latency store the dashboard reads (DynamoDB or Timestream). The dashboard is behind by the window plus a few seconds, inside a minute.

**History.** **Amazon Data Firehose** reads the same stream, converts to Parquet, partitions by `dt` and `hour` with dynamic partitioning, and buffers up to 128 MiB or 300 seconds before writing to S3, which keeps files large. Athena queries history through the Glue Catalog. Parquet compresses sensor data heavily, so two years is far less than 2 × 365 × 345 GB. A lifecycle rule moves raw JSON (if kept) to Glacier and keeps curated Parquet in Intelligent-Tiering.

The reasoning

**Throughput.** 15,000 × 2 KB is **30 MB/s**. A provisioned Kinesis shard takes 1 MB/s of writes, so at least **30 shards**, more for headroom and key skew; **on-demand** mode avoids the planning, scaling to double its previous peak. On MSK, 30 MB/s is a modest cluster, sized by partitions and brokers.

**Fraud path.** The fraud service must read every event within seconds, so it needs a stream it consumes itself. With several consumers on one Kinesis stream, give the latency-sensitive one **enhanced fan-out** — its own 2 MB/s per shard, pushed — so it does not share the 2 MB/s per shard and five reads per second of standard consumers.

**Landing path.** **Firehose** reading from the stream converts to Parquet, partitions by date and hour, and buffers up to 128 MiB or up to 900 seconds; set the interval to about 60–120 seconds to meet 5 minutes with large files. No consumer code to run.

**When MSK.** Choose MSK when you need the Kafka API — existing Kafka producers and consumers, Kafka Connect connectors, Kafka Streams, transactional exactly-once processing — or a team that already operates Kafka. Choose Kinesis when you want AWS-native integration (Lambda, Firehose, IAM) with less to operate. Firehose alone is enough only when nothing needs to read the data before it lands.

The reasoning

**Request path.** Slack sends the slash command as an HTTPS POST to an **API Gateway** endpoint. API Gateway invokes the **Lambda** synchronously and returns its response to Slack. Lambda parses the command, fetches the KPI with `s3:GetObject` using its **execution role** — scoped to the KPI prefix — and formats the reply.

**Authentication.** Slack signs every request: an HMAC-SHA256 over the version, the request timestamp and the raw body, keyed with the app's **signing secret**, sent in the `X-Slack-Signature` header. Lambda recomputes it with the secret held in **Secrets Manager** and compares in constant time, and rejects requests whose timestamp is more than about five minutes old to stop replays.

**Why the KPIs are pre-computed.** Slack expects a response within **3 seconds**. A Lambda cold start plus an Athena query over the lake would regularly miss that, so the nightly ETL writes a small JSON of today's KPIs to S3 and the Lambda only reads it. For anything slower, acknowledge immediately and post the answer later to Slack's `response_url` from an asynchronously invoked function.

**Scaling limits.** Traffic is tiny, so throughput is not the issue — latency is. Cold starts, a large deployment package, or a VPC-attached function reaching S3 without an endpoint can push past 3 seconds. The data is only as fresh as the nightly ETL; if leadership wants intraday numbers, that is a different design.

The reasoning

**Size it.** At the us-east-1 list price of **$5 per TB** scanned (September 2026), $12,000 a month is **2,400 TB**, about 80 TB a day, or two full scans of the 40 TB table daily. The jump from 1,200 TB means something new is scanning roughly one full table a day.

**Find it.** Cost Explorer confirms it is Athena bytes scanned and not S3 requests. Athena query history (or CloudTrail and the workgroup's CloudWatch metrics) lists each query with its data scanned; sort by bytes. The usual culprits are a dashboard refreshing every few minutes with no date filter, a `SELECT *`, a filter on a function of a timestamp instead of on the `dt` partition, or a view that hides a full scan.

**Fix the files.** CSV is the most expensive format Athena can read. Convert to **Parquet with compression** — Athena `CTAS`/`INSERT INTO` in batches (each writes at most 100 partitions) or a Glue job — so queries read only the columns they name; AWS's pricing example shows a query falling from $15 to $1.25 that way. Add **partition projection** for the date column.

**Guardrails.** One workgroup per team, each with a **per-query data usage limit** (say 2 TB) that cancels runaway queries, and workgroup-wide alerts. Turn on query result reuse for dashboards, and point them at a small daily aggregate table instead of the raw events.

The reasoning

**Read the plan.** `DS_DIST_NONE` and `DS_DIST_ALL_NONE` are good — nothing moves. `DS_BCAST_INNER` and **`DS_DIST_BOTH`** are the expensive ones: here every sales row and every customer row is shuffled over the network to meet on the join key. `SVV_TABLE_INFO` shows each table's `diststyle`, `skew_rows`, `unsorted` and `stats_off` — check them before changing anything.

**Fix the distribution.** `ALTER TABLE sales ALTER DISTKEY customer_id;` and the same on `customers`. With both on the join column, matching rows share a slice and the join is `DS_DIST_NONE`. **Check skew first:** if 20% of sales rows have `customer_id = 0` (guest checkout), one slice does a fifth of the work and the query is still slow — handle the guest rows separately or pick another key. `DISTSTYLE ALL` on a 60-million-row dimension copies it to every node and slows every write.

**Why it got slower.** Something changed: the fact grew, `AUTO` distribution moved a table to `EVEN`, statistics went stale after a large load (`stats_off` high — run `ANALYZE`), or the table is heavily unsorted. A sort key on `sale_date` also lets the date filter skip blocks.

**Confirm.** Re-run `EXPLAIN` to see `DS_DIST_NONE`, compare runtime and bytes in `SYS_QUERY_HISTORY`, and check that other queries joining `sales` on a different key did not get worse — a table has only one distribution key.

The reasoning

**Why it is slow.** 100 GB a day in 2 million objects means every query lists and opens thousands of files per partition, paying request and open overhead that dwarfs the data; Parquet also compresses and skips poorly in tiny files. The target is files of **128–512 MB**: 100 GB a day at 256 MB is about **400 files**.

**Compaction job.** A Glue or EMR Serverless job runs every hour over partitions that closed at least two hours earlier (so late writers have finished). For each partition it reads the files, writes `ceil(size / 256 MB)` files to a **new prefix** named with the run id, checks that input and output row counts match, and only then updates the **Glue partition location** to the new prefix — one catalog update, so readers move from old files to new at once. Old files are deleted by a lifecycle rule days later, so any in-flight query still finishes.

**Fail-safe and monitoring.** State lives in a DynamoDB table per partition (pending, compacting, validated, switched), so a crashed run resumes and never switches an unvalidated partition. Log input and output file counts, bytes and row counts per partition; publish files-per-partition and compaction lag as CloudWatch metrics; alarm on a failed validation or a partition left uncompacted past its window.

**Better: fix the source and the format.** Buffer larger at the writer — Firehose up to 128 MiB, or a longer micro-batch trigger. And store the table as **Apache Iceberg**: its `rewrite_data_files` compaction commits atomically, and the Glue Data Catalog can run compaction for Iceberg tables automatically — the managed version of this design.

The reasoning

**50 MB every 10 minutes → Lambda.** An S3 event invokes a function that parses and writes Parquet in seconds. It is far inside Lambda's 15-minute timeout and 10,240 MB memory ceiling, costs nothing between files, and needs no cluster. It becomes the wrong tool if files grow into the gigabytes or need joining to other data.

**2 TB nightly join → Glue** (or EMR Serverless). Spark is needed for a join of that size. Glue gives serverless Spark with job bookmarks, catalog integration and auto scaling, billed per DPU-hour per second with a 1-minute minimum; choose the worker type (G.2X or G.4X for join-heavy work) and the maximum workers. EMR Serverless is the alternative when the team wants open-source Spark configuration without the Glue libraries, with pre-initialised capacity for fast starts.

**60 TB backfill with a custom Spark build → EMR on EC2.** Only EMR on EC2 gives full control of the Spark version, custom JARs and bootstrap actions, and **Spot capacity** through instance fleets for task nodes, with managed scaling. Keep core nodes On-Demand so HDFS and shuffle survive Spot reclamations, run the backfill as steps, and terminate the cluster when the steps finish.

**Cost principle.** Serverless (Lambda, Glue, EMR Serverless) wins for spiky or small work because idle costs nothing; EC2-based EMR wins for large, long, well-understood jobs where Spot and right-sized instances undercut per-DPU pricing. Measure a representative run on each before committing a recurring job.

The reasoning

**Read less.** Partition pruning on `dt` reads one day of clickstream; column pruning reads only the Parquet columns the aggregation needs, often a fraction of the 8 TB. Filter out bots and irrelevant event types at the scan.

**Shrink before the shuffle.** Aggregate clickstream to one row per `session_id` or `order_id` — pages viewed, time on site — **before** joining. Billions of events become tens of millions of rows, and the shuffle shrinks by orders of magnitude.

**Choose the join.** 400 GB is far too large to broadcast (Spark's default broadcast threshold is 10 MB), so this is a **sort-merge join** on the key. With **adaptive query execution**, on by default in current Spark, skewed partitions are split and small post-shuffle partitions coalesced. If the same join runs every day, store both sides bucketed or sorted on the key (an Iceberg table with a sort order) to cut repeated shuffle work.

**Run and write.** EMR Serverless or Glue sized for the shuffle, the output written partitioned by date in Parquet, the job idempotent by overwriting its date partition. Check the Spark UI for the largest stage and for stragglers before adding capacity.

The reasoning

**Tag and measure first.** Cost Explorer by service and by cost-allocation tag (team, pipeline), so each saving is attributed and verified. Set AWS Budgets alerts so the saving holds.

**Redshift ($36,000).** A cluster busy eight hours a day is paying for sixteen idle ones. Options, in order: pause it or resize down on a schedule if the workload is predictable; buy **reserved nodes** for the steady baseline; move spiky workloads to **Redshift Serverless** with a maximum RPU cap; use concurrency scaling (one free hour of credit per cluster per day) instead of sizing for the 9 a.m. peak; and fix the few queries responsible for most of the compute.

**Glue ($20,000).** Check the job metrics for over-provisioned workers, turn on auto scaling, use the **Flex** execution class for jobs that are not urgent, and stop full reprocessing by making loads incremental (bookmarks or watermarks). Very large steady jobs may be cheaper on EMR with Spot — measure one run.

**S3 ($12,000) and transfer ($12,000).** Lifecycle rules to IA and Glacier for aged raw data, Intelligent-Tiering for unpredictable data, expiry of noncurrent versions and incomplete multipart uploads, Parquet with compression instead of CSV. For transfer: an **S3 gateway endpoint** removes NAT gateway processing charges for S3 traffic from private subnets, often a surprising share of "other". Add up the measured savings against the $32,000 target before promising it.

The reasoning

**Raw zone.** Heavy reads in the first week and IA's **30-day minimum** make **day 30** the earliest sensible transition to **Standard-IA**. Occasional reads until day 90 stay cheap there. After that the data is only for audit, so move it to **Glacier Deep Archive** — its **180-day minimum** is irrelevant for data kept 7 years — at day 180 (or 90, to Glacier Flexible Retrieval first, if audits sometimes need data within hours). Expire at **2,555 days**.

**Where the bytes sit.** At 3 TB a day, the steady state at 7 years is about **7.7 PB** of raw data, and only about 90 TB of it is newer than 30 days. So well over 90% of the eventual lake is in Deep Archive, which is why the policy matters more every year.

**Curated zone.** Query patterns are unpredictable, so **Intelligent-Tiering** moves objects between access tiers automatically with no retrieval fees; objects under 128 KB are not monitored. Scratch and staging prefixes expire after 7 days.

**Checks.** Objects under 128 KB are not transitioned by default, and transitions are billed per object, so compact small files before archiving. Deep Archive restores take hours (standard retrieval within 12 hours), which must be acceptable to the auditors. For a regulatory hold, add **S3 Object Lock** in compliance mode so the data cannot be deleted early. Expire noncurrent versions too, or versioning keeps the old copies at full price.

The reasoning

**Tracking.** Land files under `s3://drops/orders/dt=YYYY-MM-DD/`. A Glue job with **bookmarks enabled** on the S3 source reads only objects whose last-modified time is after the previous committed run, and the job ends with `job.commit()`. 20,000 files a day are small (about 1.5 MB each), so group them on read (`groupFiles: inPartition`, which Glue does automatically above 50,000 files) to avoid 20,000 tiny tasks; keeping each day in its own prefix also bounds the listing the bookmark does.

**Loading Oracle.** Write through a Glue JDBC connection to a **staging table** with batched inserts and a sensible number of parallel writers, then run a **`MERGE`** into the target on the business key. Record each file key, row count and run id in a **load-audit table**, which is also how you answer "was this file loaded?" without trusting the bookmark.

**The reset.** AWS documents that resetting or rewinding a bookmark **does not clean the target**, because Glue does not track outputs. After a reset the next run reads every file in the prefix — 30 days is 600,000 files and 900 GB. With an append-only load that duplicates a month of data. With the staging-plus-merge load it only costs runtime, because merging the same rows again changes nothing. Another hazard: changing the source path while keeping the same `transformation_ctx` makes Glue skip files it believes it has seen.

**Alternatives.** An S3 event per file into SQS gives an explicit list of new files instead of a timestamp scan, and makes per-file retries simple; S3 Inventory can reconcile what landed against the audit table daily.

The reasoning

**Where the time went.** Turn on the **Spark UI** event logs and Glue job observability metrics. Find the stage whose duration grew. If one task in that stage takes most of the time while the rest finish in seconds, it is **skew**: a new dominant key (a default value, a bot, a big customer). If every task is slower, look at spill to disk, garbage collection and executor out-of-memory retries.

**Input shape.** Count the input files, not just the bytes. If a producer switched from 500 files to 400,000, the driver spends its time listing and planning, and the bookmark lists every file too. Group small files on read, or fix the producer. For a JDBC source, a read with no `hashfield`/`hashpartitions` runs on **one connection** however many workers exist.

**Capacity.** Check whether auto scaling is enabled and already at the maximum, and whether memory-heavy stages need G.4X rather than more G.2X workers. Ten workers that each spill heavily are slower than five that do not.

**Fixes, matched to cause.** Skew: adaptive query execution's skew join handling (on by default in Spark 3.5), salting the hot key, or handling the dominant value separately. Spill: fewer rows per partition, a larger worker type. Small files: grouping or compaction. Then compare the rerun's stage timings with the old baseline so the fix is shown, not assumed.

The reasoning

**Take control from IAM.** For backward compatibility, Lake Formation starts with **`IAMAllowedPrincipals`** holding permissions and "Use only IAM access control" enabled for new catalog resources, which means IAM policies still decide access. Register the table's S3 location with Lake Formation, and revoke `IAMAllowedPrincipals` on the database and table (or opt the table in through **hybrid access mode** to migrate without breaking other users).

**Filters.** Lake Formation implements **column-level, row-level and cell-level security** with **data filters**. Risk team: `SELECT` on the table with all columns. Other teams: a column filter that excludes `pan_number` and `date_of_birth`. EU team: a data filter combining the same column exclusion with the row expression `region = 'EU'` — cell-level security. Grant each filter to the team's IAM role or IAM Identity Center group. At scale, **LF-Tags** (`classification = pii` on the columns) let a policy follow the tag rather than each table.

**Enforcement.** Athena asks Lake Formation for permissions and receives **temporary credentials** for exactly the data allowed; the filters are applied at query time. Analysts' own roles must have **no `s3:GetObject` on the bucket** — otherwise they can read the Parquet files directly and skip every filter. Deny it explicitly in the bucket policy for everyone except the Lake Formation registration role and the pipeline roles.

**Other engines and audit.** Each engine must support Lake Formation fine-grained access for the filters to apply — Glue 5.0 and later use a newer mechanism than 4.0, so check versions before relying on it. CloudTrail records Lake Formation permission checks for audit.

The reasoning

**Encryption at rest.** S3 encrypts all new objects with SSE-S3 by default since January 2023; for card data use **SSE-KMS with a customer-managed key**, whose key policy limits who can decrypt, and enable **S3 Bucket Keys**, which AWS says cut KMS request costs by up to 99% — at 200 GB a day in many objects, KMS request charges and throttling are otherwise real. Attach a **Glue security configuration** that encrypts the job's S3 output, CloudWatch logs and job bookmarks with KMS. KMS uses envelope encryption: data keys encrypt the data, and the KMS key encrypts the data keys.

**In transit and network.** Connect to RDS over TLS (require it in the database parameters), deny non-TLS S3 access with a bucket policy on `aws:SecureTransport`, and run the job in **private subnets** with an S3 gateway endpoint and interface endpoints for other services — no internet path.

**Secrets and identity.** Database credentials in **Secrets Manager**, referenced by the Glue connection; a job role scoped to one source and one target prefix; no one else writes there.

**Minimise the data.** Tokenise or hash card numbers (with a secret key or salt from Secrets Manager) during the transform, so raw PANs never land in the lake; Glue's sensitive-data detection transform can find PII in columns you did not expect. Do not log row contents. Use Lake Formation for column-level access, CloudTrail for access audit, and **Amazon Macie** to scan buckets for sensitive data that slipped through.

The reasoning

**Failure events.** EventBridge rules on **Glue Job State Change** (`FAILED`, `TIMEOUT`) and **Step Functions execution status change** (`FAILED`, `TIMED_OUT`, `ABORTED`) route to SNS and the on-call channel, with the run id and a link to logs.

**Metrics and alarms.** CloudWatch alarms on Step Functions `ExecutionsFailed`, on DMS `CDCLatencySource` and `CDCLatencyTarget` above an agreed threshold, and on Glue job duration against a baseline so a job that is still running at 4 a.m. is noticed before it fails. Redshift load errors are in `STL_LOAD_ERRORS` (`SYS_LOAD_ERROR_DETAIL` on Serverless).

**Outcome, not only jobs.** At 5:30 a.m. a small check queries the target: is the maximum `load_date` today, are row counts within the expected band, did key metrics move implausibly? Glue Data Quality rules can run inside the jobs for the same purpose. This catches the failure that no job reports — a job that succeeded on an empty file.

**Troubleshooting and recovery.** Every alert links to a runbook: where the logs are, how to rerun one step (Step Functions redrive or a single-job rerun), and why that is safe (idempotent loads). Retries with backoff in Step Functions absorb transient errors so people are paged only for real ones.

The reasoning

**What it manages.** Permissions on Data Catalog databases and tables, and on the S3 locations registered with it. Grants can be limited to specific columns, to rows through filter expressions, or to both (cell-level), and can be assigned through **LF-Tags** so a policy follows a tag across many tables.

**How it enforces.** When an integrated engine queries a table, it asks Lake Formation for permission and receives temporary credentials for the data allowed. That only protects the data if principals do not also have direct S3 access.

**Defaults.** For backward compatibility with Glue, new catalog resources start with IAM-only access control through `IAMAllowedPrincipals`; you revoke that, or use hybrid access mode, to put Lake Formation in charge.

**Also.** Cross-account sharing of catalog tables without copying data, and audit of permission checks through CloudTrail.

The reasoning

**EMR on EC2** gives a cluster of a primary node, core nodes (which hold HDFS) and task nodes (compute only). You choose instance types, the release and applications, bootstrap actions and configuration; managed scaling adds and removes nodes; instance fleets mix On-Demand and **Spot**. Clusters can be long-running or transient — created, run steps, terminated.

**EMR Serverless** runs Spark and Hive jobs without clusters: it determines the resources a job needs, acquires them and releases them when the job finishes, with optional **pre-initialised capacity** for responses in seconds. **EMR on EKS** runs Spark on an existing Kubernetes cluster.

**When EMR rather than Glue:** a custom Spark build or libraries, Hadoop-ecosystem tools Glue lacks, very large steady workloads where Spot and tuned instances cost less, or teams that want plain open-source Spark. Glue is simpler for catalog-centred ETL with bookmarks.

The reasoning

**Model.** A state machine is a JSON (Amazon States Language) definition of states: `Task` (call a service), `Choice`, `Parallel`, `Map`, `Wait`, `Pass`, `Succeed` and `Fail`. Service integrations call Glue, EMR, Lambda, Athena, DynamoDB, the Redshift Data API and many more directly.

**Integration patterns.** Request-response (start and move on), **run a job** (`.sync`, wait until the Glue job or EMR step finishes) and wait for a callback with a task token. `Retry` and `Catch` blocks give each step its own backoff and failure path.

**Workflow types.** **Standard** — up to one year, exactly-once, full execution history (up to 25,000 events), billed per state transition. **Express** — up to five minutes, at-least-once, billed by executions and duration, for event-processing volume.

In data pipelines it is typically started by EventBridge on a schedule or an S3 event, and chains extraction, a Glue job, a data-quality check and a Redshift load.

The reasoning

**Athena** queries data where it lies — Parquet, ORC, JSON, CSV or Iceberg tables in S3 — through the Glue Data Catalog. There is nothing to provision; per-query billing charges for bytes scanned ($5 per TB at the us-east-1 list price as of September 2026, 10 MB minimum per query), or you can buy Capacity Reservations. Performance depends on how the files are laid out.

**Redshift** loads data into managed columnar storage and tunes it with sort keys, distribution, compression, materialised views and result caching. Provisioned clusters bill per node-hour; Serverless bills per RPU-hour while active. It gives consistently fast, high-concurrency queries on hot data and joins.

**Choose** Athena for ad hoc and infrequent queries over the lake, or where no warehouse exists; Redshift for dashboards and heavy repeated joins on curated data. They combine well: Redshift Spectrum and Athena read the same catalog tables.

The reasoning

**Metrics.** Services publish metrics automatically: Kinesis `IteratorAgeMilliseconds` and throughput-exceeded counts, Lambda errors and duration, DMS CDC latency, Step Functions failed executions, Glue job metrics. You can publish custom metrics — rows loaded, freshness in minutes — from your own jobs.

**Logs.** CloudWatch Logs collects Glue driver and executor logs, Lambda logs and more; Logs Insights queries them; metric filters turn log patterns into metrics.

**Alarms and dashboards.** Alarms trigger SNS notifications or automated actions when a metric crosses a threshold or an anomaly band; dashboards put a pipeline's signals on one page. Service state changes go to EventBridge, which is the better trigger for "job failed" alerts.

**Its limit.** CloudWatch sees systems, not data. A job that loads an empty file reports success; freshness and volume checks on the tables themselves have to be added as custom metrics or quality rules.