AWS Conceptual

The reasoning

S3 Standard is designed for **99.999999999% (eleven nines) durability** by storing objects redundantly across devices in several Availability Zones. On consistency, AWS documents **strong read-after-write consistency for PUT and DELETE**: once the write returns success, any subsequent read or list sees it. The old advice to wait, or to keep a consistency layer such as EMRFS consistent view, dates from before December 2020 and is no longer needed.

What S3 does not have is file-system semantics. The namespace is flat: `s3://lake/events/dt=2026-09-14/` is a **key prefix**, not a directory. There is no append — changing an object means rewriting it. There is **no rename**: moving a "folder" is a copy of every object followed by a delete of every object, and a thousand-object move can fail halfway. There are no locks across objects, and concurrent writers to the same key are resolved by the last write.

That is what bites Spark. Hadoop's classic output committer writes to a temporary directory and renames it into place, which on HDFS is one metadata operation. On S3 it is a slow copy, and a reader listing the prefix mid-commit can see half the files. EMR and Glue ship S3-optimised committers that avoid the rename for common formats, but they still do not make a multi-file write atomic to readers.

The durable fixes are to publish differently. Either write to a staging prefix and flip a pointer that readers follow (a manifest, a `_SUCCESS` marker readers check, or a Glue table location swap), or use a table format — **Apache Iceberg, Delta Lake or Hudi** — whose commit is a single atomic metadata write, so readers see all of a write or none of it.

The reasoning

AWS documents that an application can achieve **at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix**, and that there is no limit to the number of prefixes in a bucket. Scaling to a new, higher request rate is gradual, and while it happens clients can receive **503 Slow Down** errors, which the SDKs retry with backoff. The old advice to put random hashes at the front of keys dates from before AWS raised these rates in 2018; a date-partitioned layout spreads load well enough for most lakes.

Hive-style keys — `s3://lake/clean/orders/dt=2026-09-14/region=eu/part-0001.parquet` — do double duty. Each partition is its own prefix, which spreads request rate, and Athena, Glue, EMR and Redshift Spectrum all understand `key=value` paths as partition columns, so `WHERE dt = '2026-09-14'` reads one prefix instead of the table.

The failure is over-partitioning. Partitioning by `customer_id` or by minute creates millions of tiny prefixes and tiny files; every query then pays per-file open and list costs, and a crawler or `MSCK REPAIR TABLE` takes hours. Partition by the column queries actually filter on, with a cardinality in the hundreds or low thousands per year, and aim for files in the **hundreds of megabytes** in a columnar format such as Parquet with Snappy or ZSTD compression.

Split zones — raw, clean, curated — by bucket or by top-level prefix, because IAM policies, lifecycle rules and replication all attach to buckets and prefixes. Raw data you must keep for audit then gets its own lifecycle, and analysts can be granted `curated/` without ever seeing `raw/`.

The reasoning

**S3 Standard** is for hot data. **Intelligent-Tiering** charges a small monitoring fee per object and moves objects between access tiers on its own — to Infrequent Access after 30 days without access and to Archive Instant Access after 90 — with **no retrieval fees**; objects under 128 KB are not monitored. **Standard-IA** and **One Zone-IA** are cheaper to store and charge per GB retrieved; they bill a **30-day minimum** and a **128 KB minimum object size**, and One Zone-IA keeps data in a single Availability Zone.

The archive classes: **Glacier Instant Retrieval** (millisecond access, 90-day minimum), **Glacier Flexible Retrieval** (minutes to hours to restore, 90-day minimum) and **Glacier Deep Archive** (the cheapest, restores in hours, 180-day minimum). Deleting, overwriting or transitioning an object before its minimum is charged as if it had stayed.

A **lifecycle configuration** has transition actions and expiration actions, filtered by prefix, tag or object size. Transitions follow a waterfall — Standard to IA to Glacier, not back up. Since September 2024, objects **smaller than 128 KB are not transitioned by default**, because each transition is a billed request and a tiny object in IA is billed as 128 KB anyway. Lifecycle rules should also expire noncurrent versions in versioned buckets and abort incomplete multipart uploads, which are two quiet sources of storage cost.

For a lake the usual shape is: raw landing data to IA after 30 days and to Deep Archive after 180 if audit requires keeping it; curated Parquet stays in Standard or Intelligent-Tiering because query engines read it unpredictably; scratch and staging prefixes expire after 7 days.

The reasoning

A **crawler** is pointed at one or more include paths and runs a list of **classifiers** in order. Built-in classifiers cover Parquet, ORC, Avro, JSON, CSV and others; custom classifiers (grok, XML, JSON path, CSV with a declared header) run first when present. Self-describing formats are cheap: Parquet and ORC carry the schema in the file, so the crawler reads it. Text formats are sampled, which is why a CSV column that is empty in the sample comes out as `string`.

The crawler then decides **table boundaries**. Folders whose files share a compatible schema become one table, and the folders below the table root become partitions — named after the key when the path is Hive-style (`dt=2026-09-14`), or `partition_0`, `partition_1` otherwise. When schemas differ between folders it may create one table per folder instead; the crawler option to **create a single schema for each S3 path** stops that.

What it does on later runs is configurable: a **schema change policy** (update the table, or only log the change), a **deletion behaviour** for vanished objects, and a **recrawl policy** — crawl everything, or only new folders (incremental crawls that add new partitions). Crawls are billed by DPU time, so a full crawl over a million objects every hour is a real cost.

A crawler is not the only way to get metadata. You can declare the table with `CREATE EXTERNAL TABLE` in Athena, have the Glue job update the catalog as it writes (`enableUpdateCatalog`), or skip partition registration with Athena partition projection. Many teams crawl once to bootstrap, then own the DDL.

The reasoning

A bookmark is persisted **state per job, per `transformation_ctx`**. For **Amazon S3** input, AWS documents that bookmarks check the **last modified time** of objects to decide what needs processing; a file that is modified after it was processed is reprocessed in full on the next run. For **JDBC** sources the bookmark stores the last value of the **bookmark keys**: by default the primary key, which is expected to be sequential, or user-defined `jobBookmarkKeys` that must be **strictly monotonically increasing or decreasing**. Gaps are fine; updates to old rows are invisible, because their key does not change.

The script has to cooperate. Each source that should be tracked needs a `transformation_ctx` name, the job must be started with `--job-bookmark-option job-bookmark-enable`, and it must end with **`job.commit()`** — the state is only saved on commit, so a job that fails before it reprocesses the same input next time. The other options are `job-bookmark-disable` (always read everything) and `job-bookmark-pause` (read from the last bookmark without advancing it, with optional `job-bookmark-from` / `job-bookmark-to` run ids).

The sharp edges are documented. **Resetting or rewinding a bookmark does not clean the target** — Glue does not track outputs — so a reset job appends a second copy unless the write is an overwrite or a merge. **Changing a source path while keeping the same `transformation_ctx`** makes Glue apply the old state to the new path and skip files it thinks it has seen. And for a partition with a very large number of files, bookmark listing can run the driver out of memory; AWS points at the Glue S3 file lister for that case.

So a bookmark is a watermark, not exactly-once delivery. Make the write idempotent — partition overwrite or `MERGE` on a business key — and a rerun or reset becomes safe rather than a duplicate-cleanup exercise.

The reasoning

**Role and version.** The job runs as an IAM role that needs access to the script location, the data and the temporary directory. **Glue 5.1** (Spark 3.5.6, Python 3.11) is the default for jobs created without a version; older versions stay selectable, and features differ between them.

**Capacity.** Capacity is measured in DPUs — one **DPU is 4 vCPUs and 16 GB of memory**. Worker types map onto it: **G.1X** is 1 DPU, **G.2X** 2 DPU, **G.4X** 4 DPU, and **G.025X** a quarter DPU for low-volume streaming jobs. Spark jobs need a minimum of 2 DPUs, and AWS bills per DPU-hour, per second, with a **1-minute minimum** on Glue 2.0 and later. Auto scaling lets a job add and remove workers up to the configured maximum, and the **Flex** execution class runs non-urgent jobs on spare capacity at a lower rate.

**Behaviour.** The job bookmark option, timeout, maximum retries, maximum concurrent runs, job parameters (`--TempDir`, `--additional-python-modules`, `--conf`), and the Spark UI and continuous logging settings used for debugging.

**Connections and connectors.** AWS defines a **connection** as a Data Catalog object that stores login credentials, URI strings and VPC information for a particular data store, reusable across crawlers and jobs. A **connector** is what the connection is created from: the native JDBC connectors, AWS Marketplace connectors, or a custom one. For a database inside a VPC, Glue attaches network interfaces in the connection's subnet with its security group, so that group needs a self-referencing inbound rule and the subnet needs a route to S3 — usually an S3 gateway endpoint. Keep the password in Secrets Manager and reference it from the connection.

The reasoning

A connection holds a single JDBC URL (or a Redshift or other native endpoint), its credentials and its VPC settings. What one connection **can** cover is many tables in the same database: a crawler with the include path `sales/%` catalogues every table in the `sales` database, and one job can read ten tables through the same connection.

RDS and Redshift are two data stores, so the job gets **two connections**. Each carries its own subnet and security group, and the practical constraint is network reachability: the job's network interfaces must be able to reach both endpoints, so the simplest arrangement puts both in one VPC or routes between them. Credentials belong in Secrets Manager, one secret per store.

Two alternatives are often better than a Glue join across two databases. The Glue **Redshift connector** reads and writes Redshift through an S3 temporary directory (`UNLOAD` out, `COPY` in), which is much faster than row-by-row JDBC for large tables. And **Redshift federated query** lets Redshift query RDS or Aurora PostgreSQL and MySQL tables live, so the join can run inside the warehouse with no Glue job at all.

If the question is really "one connection to manage", the answer is the Data Catalog: crawl both stores once, and every downstream job refers to catalog tables by name rather than by endpoint.

The reasoning

**Glue triggers** come in four types. **Scheduled** runs on a cron expression. **Conditional** fires when watched jobs or crawlers reach a state — `SUCCEEDED`, `FAILED`, `TIMEOUT` — with ANY or ALL logic, which is how one Glue job chains after another. **On-demand** fires when started manually or by API. **Event** fires on EventBridge events, optionally batched by count or time window. Glue **workflows** group triggers, jobs and crawlers into a DAG you can monitor as a unit.

**Step Functions** starts executions from outside — an EventBridge rule or EventBridge Scheduler, a `StartExecution` API call, API Gateway, or another state machine. The value is inside: service integrations call Glue, EMR, EMR Serverless, Lambda, Athena and the Redshift Data API, and the **`.sync`** integration pattern waits for a Glue job to finish before the next state. `Retry` and `Catch` give per-step error handling. **Standard** workflows run up to **one year** with exactly-once state transitions; **Express** workflows run up to **five minutes** with at-least-once semantics, for high-volume short work.

The decision is usually made on scope. If every step is a Glue job or crawler, Glue workflows and conditional triggers are enough and cost nothing extra. As soon as the chain mixes services, needs branching, human approval or careful failure handling, **Step Functions** is the AWS-native choice. **MWAA** (managed Airflow, with Airflow 2.10, 2.11 and 3.x versions supported) wins when the team already writes Airflow DAGs, needs backfills over logical dates, or orchestrates systems outside AWS.

The reasoning

**Architecture.** A leader node plans queries and compute nodes execute them; each compute node is divided into **slices** that each own part of every table. Storage is **columnar and compressed**, in 1 MB blocks with min/max metadata (zone maps), so a query reads only the columns it names and skips blocks outside its filter. That is why `SUM(amount) GROUP BY region` over two billion rows is seconds in Redshift and minutes in a row store.

**Pros.** Massively parallel scans and joins; **RA3** and the newer **RG** node types separate compute from managed storage, so storage grows without adding nodes; querying S3 in place (Spectrum, or the engine built into RG nodes); concurrency scaling for bursts; Serverless for spiky workloads; and it is a managed service, with automatic vacuum, analyse and table optimisation.

**Cons.** It is not an OLTP database. Single-row `INSERT`s and `UPDATE`s are slow and are meant to be batched through `COPY` and staging tables. **Primary, unique and foreign keys are informational and not enforced**, so duplicates are your problem. There are no indexes; performance depends on distribution and sort keys, which `AUTO` helps with but does not replace. Concurrency is far lower than an OLTP database's, and a provisioned cluster costs money while idle.

MySQL is a row store built for transactions and scales up, not out; Oracle can do analytics but licensing and scale-up hardware are the price. The honest framing: keep OLTP in RDS or Aurora, replicate into Redshift, and do analytics there.

The reasoning

Count the scans from the plan, not the SQL. `SELECT product_id, SUM(qty) FROM orders WHERE order_date >= '2020-01-01' AND order_date < '2021-01-01' GROUP BY 1 ORDER BY 2 DESC LIMIT 1` scans `orders` **once**. Written as "compute totals, then find the max, then join back to find which product has it", the plan can show two scans of the same table — which is what the interviewer wants you to spot. `EXPLAIN` lists each `Seq Scan` step.

**Redshift local tables have no `PARTITION BY`.** The mechanism that plays the same role is the **sort key**. Redshift stores each column in 1 MB blocks and records each block's minimum and maximum value — the **zone map**. With `order_date` as the leading sort-key column, the filter above lets the scan skip every block whose range lies outside 2020, which on ten years of orders removes about 90% of the I/O. Without a sort key, those blocks are read and discarded.

The **distribution key** matters too once the query joins `orders` to `products`: if both are distributed on `product_id`, or `products` is small enough for `DISTSTYLE ALL`, the join needs no redistribution.

Partitions do exist for **external tables** — Spectrum tables over S3 declare partition columns in the Glue Catalog, and a filter on them prunes whole S3 prefixes. So the full answer is: sort keys for local data, partitions for data left in S3.

The reasoning

**Table design.** Put a **timestamp as the leading sort-key column** when queries filter on time ranges, so zone maps skip blocks; if a table is frequently joined, a sort key on the join column lets the planner use a merge join instead of a hash join. Distribute large tables with **`DISTKEY`** on their most frequent large join column so joins are co-located, give small dimensions **`DISTSTYLE ALL`**, and let **`AUTO`** decide where there is no clear choice. Leave column compression to `ENCODE AUTO`.

**Loading and maintenance.** Load with `COPY` from multiple files, not row-by-row `INSERT`. Make sure statistics are current (`ANALYZE`, which runs automatically but can lag after a large load) and that deleted rows are reclaimed and data re-sorted (automatic `VACUUM`). Use materialised views with auto refresh for expensive repeated aggregations.

**Workload.** Automatic WLM with query priorities, short query acceleration for quick queries, and concurrency scaling for bursts of queued queries. Result caching serves repeated identical queries without re-running them.

**Diagnose before tuning.** `EXPLAIN` shows join redistribution: `DS_DIST_NONE` and `DS_DIST_ALL_NONE` are good, while `DS_BCAST_INNER` and `DS_DIST_BOTH` mean data moves between nodes at query time. `SYS_QUERY_HISTORY` finds the slow queries, `STL_ALERT_EVENT_LOG` flags missing statistics and nested loops, and Redshift Advisor recommends sort and distribution keys from the actual workload.

The reasoning

**Stage.** Put the data in S3 split into multiple files of similar size — AWS recommends files between **1 MB and 1 GB after compression**, and a file count that is a **multiple of the number of slices**, so every slice loads in parallel. One 20 GB CSV loads on one slice.

**Permissions.** Associate an IAM role with the cluster or workgroup that allows `s3:GetObject` and `s3:ListBucket` on the prefix, plus `kms:Decrypt` if the objects use SSE-KMS.

**Load.** Create the target table, then run `COPY` with the format options — `FORMAT AS CSV`, `IGNOREHEADER 1`, `DATEFORMAT`, `GZIP` — and `IAM_ROLE`. A **manifest** file pins the exact set of files to load, which makes reruns deterministic. `COPY` is parallel across slices; `INSERT` statements go one by one and are far slower.

**Verify and merge.** Failures land in `STL_LOAD_ERRORS` (`SYS_LOAD_ERROR_DETAIL` on Serverless); `MAXERROR` decides how many bad rows are tolerated. Because keys are not enforced, a load that must update existing rows goes into a **staging table** first, followed by `MERGE` (or delete-then-insert) into the target inside one transaction.

The reasoning

**Spectrum** defines an external schema over a Glue Data Catalog database. A query that touches an external table sends the S3 part to the Spectrum layer, which scans the files, applies filters and partial aggregates, and returns only what the cluster needs; most of the data stays in S3. Spectrum is billed by bytes scanned, on top of the cluster. The newer **RG** node type includes a built-in data lake query engine for which AWS notes Spectrum is not required.

**Athena** is serverless SQL over the same catalog — no cluster, billed per byte scanned (**$5 per TB** at the us-east-1 list price as of September 2026, which can change) with a 10 MB minimum per query. It is the natural tool for exploration, lake-only reporting and teams without a warehouse.

**COPY** moves the data into Redshift managed storage, where it gets sort keys, distribution, compression and result caching. Pay the load once, then query cheaply and fast.

The rule: hot and joined every hour → COPY. Five years of clickstream joined monthly to warehouse dimensions → Spectrum, partitioned in Parquet. Analysts poking at raw lake files → Athena. All three read less, and cost less, when the files are Parquet and partitioned by the filter column.

The reasoning

**Workload management (WLM)** routes queries into queues. Automatic WLM decides memory and concurrency itself and honours **query priorities**; **short query acceleration** runs quick queries in a dedicated space so they do not wait behind hour-long ETL.

**Concurrency scaling** is turned on per WLM queue. When queries queue, Redshift sends eligible ones to transient scaling clusters. It supports read queries and commonly used write operations — ETL statements such as `COPY`, `INSERT`, `UPDATE` and `DELETE` — but **not most DDL** such as `CREATE TABLE`, and not writes to a table with `DISTSTYLE ALL`.

**Cost.** AWS pricing states that each cluster earns **up to one hour of free concurrency scaling credits per day**, and usage beyond the credits is charged at a per-second on-demand rate. A usage limit can cap it, so a runaway dashboard cannot run up an unbounded bill.

**Serverless** handles concurrency by scaling Redshift Processing Units. The default **base capacity is 128 RPUs**, settable from 4 to 512; a **maximum RPU** setting caps how far a burst can scale, and so how much it can cost.

The reasoning

With per-query billing you pay for **bytes scanned per query, rounded up to the nearest megabyte, with a 10 MB minimum**; DDL such as `CREATE`, `ALTER` and `DROP TABLE` is free. The list price in us-east-1 is **$5 per TB** as of September 2026 and may change. **Capacity Reservations** are the alternative: dedicated DPUs billed hourly, for predictable heavy use.

AWS's own pricing example makes the levers concrete: a query over 3 TB of uncompressed text costs **$15**; GZIP at 3:1 brings the same query to **$5**; converted to a columnar format and reading one of four columns, it scans 0.25 TB and costs **$1.25**.

**Partition pruning** happens when the `WHERE` clause filters on a partition column: Athena reads only the matching prefixes. It only works on the partition column as stored — `WHERE dt = '2026-09-14'` prunes, while wrapping a timestamp column in a function scans everything. **Predicate pushdown** goes further inside the files: Parquet and ORC keep min/max statistics per row group, and row groups that cannot match are skipped. It works best when files are sorted or clustered on the filter column.

Two cheap guards: a workgroup **per-query data usage control** cancels any query that would scan more than a set amount, and large files (hundreds of MB) keep S3 request and file-open overheads down. `LIMIT` does not reduce what is scanned on an unsorted table.

The reasoning

Normally Athena looks up partitions in the Glue Data Catalog, which means someone has to add them — a crawler, `MSCK REPAIR TABLE`, or `ALTER TABLE ADD PARTITION`. On a table with tens of thousands of partitions that lookup is slow, and a missing registration makes new data invisible.

With **partition projection** you declare in table properties how partitions are formed: a type for each partition column — **integer, date, enum or injected** — its range and format, and a storage location template. Athena then **projects** the partitions in memory. AWS documents that enabling projection makes Athena **ignore any partition metadata registered in the Glue Catalog** for that table.

`MSCK REPAIR TABLE` works only for Hive-style `key=value` paths and lists the whole table location, which is slow on large tables. Crawlers cost DPU time and can mis-infer. Projection needs neither.

The limits: projection is an **Athena** feature, so Spark on EMR or Glue reading the same catalog table still sees only registered partitions, and a projected partition with no data returns nothing rather than an error — a missing day looks like a quiet day unless something checks.

The reasoning

**Per shard, provisioned:** writes up to **1 MB/s or 1,000 records/s**; reads up to **2 MB/s**, shared by every standard consumer, with at most **five read transactions per second**, each returning up to 10,000 records or 10 MB. **Enhanced fan-out** gives each registered consumer its own **2 MB/s per shard**, pushed over HTTP/2 — up to 20 consumers per stream, or 50 with On-demand Advantage. Exceeding a limit returns `ProvisionedThroughputExceededException`.

**The partition key** is hashed (MD5) into the shard hash-key space. All records with one key go to one shard, which is what gives **ordering per key**; it also means one hot key can never go faster than one shard, whatever the shard count. Resharding splits or merges shards (`UpdateShardCount` does it uniformly).

**On-demand mode** removes shard planning. AWS documents that a new on-demand stream starts with **4 MB/s** of write capacity and accommodates up to **double its previous peak** write throughput, scaling within about 15 minutes; the ceiling is 10 GB/s of writes in the largest Regions and 200 MB/s elsewhere. A stream can switch between modes **twice in 24 hours**. The account-level **On-demand Advantage** mode changes pricing and allows pre-warming.

**Retention** defaults to 24 hours and can be raised to **8,760 hours (365 days)**. Records can now be up to **10 MiB**, but every stream, new or existing, defaults to a **1 MiB** maximum until you raise it with `UpdateMaxRecordSize`; AWS positions large records as occasional, alongside a baseline of records of 1 MiB or less.

The reasoning

**The ceilings.** A standard invocation can run for up to **900 seconds (15 minutes)**; the exception is Lambda Managed Instances, where asynchronous and event-source-mapping invocations can run up to 90 minutes. Memory is **128 MB to 10,240 MB**, with CPU allocated in proportion. Synchronous request and response payloads are capped at **6 MB** each. Concurrency is limited per account and Region, and new accounts start with reduced quotas. Cold starts add latency, and state does not survive between invocations.

**Where Lambda fits.** Event-driven work where each unit is small and independent: an S3 `ObjectCreated` event that validates or converts one modest file, a Kinesis batch that is enriched and written on, an API that returns a metric, a small file-arrival hook that starts a Glue job or a Step Functions execution. It scales per event and costs nothing when idle.

**Where Glue fits.** Anything that needs Spark: joins between large datasets, files in the gigabytes, aggregations across partitions, job bookmarks, catalog integration. A Lambda that loads a 3 GB file into memory, or loops through S3 to stay under 15 minutes, is a Glue job written the hard way.

The two work well together: Lambda as the trigger and light transform, Glue (or EMR) as the engine.

The reasoning

DMS runs **tasks** on a replication instance (or DMS Serverless) between a source and a target endpoint. A task is **full load**, **CDC only**, or **full load plus CDC**. In the combined mode, changes made during each table's full load are **cached** and applied once that table's load completes, then the task moves to **ongoing replication**, reading the source's log through engine-specific APIs.

**Source prerequisites** are where tasks fail. On MySQL, CDC needs `binlog_format = ROW` and `binlog_row_image = FULL`, and on RDS the binary logs must be kept long enough — for example `call mysql.rds_set_configuration('binlog retention hours', 24);` — or a task that falls behind finds its position purged. PostgreSQL needs logical replication (on RDS, `rds.logical_replication = 1`), which creates a replication slot. Oracle needs supplemental logging. Updates and deletes need a primary key to be applied correctly.

AWS is explicit that **DMS CDC is not real-time** and has **no latency SLA**; latency depends on source load, instance size and the target. Watch the `CDCLatencySource` and `CDCLatencyTarget` CloudWatch metrics.

Targets shape the rest. Writing to S3, DMS produces CSV or Parquet files with an **operation column (`I`, `U`, `D`)**, batched by settings such as `cdcMaxBatchInterval` and `cdcMinFileSize`; something downstream must merge those into a current-state table. Large objects need a LOB mode chosen deliberately, and data validation can compare source and target row by row.

The reasoning

**Users** are long-lived identities for people or legacy tools; **groups** attach policies to many users; **roles** have no long-term credentials — a principal **assumes** the role and receives temporary credentials from STS. A role's **trust policy** says who may assume it: `glue.amazonaws.com` for a Glue job role, `redshift.amazonaws.com` for a Redshift role.

**Policies** are identity-based (attached to a user, group or role) or resource-based (attached to the resource, such as an S3 bucket policy or KMS key policy). Evaluation: an **explicit deny always wins**; otherwise the request needs an allow from an identity or a resource policy (within one account); organisation SCPs and permission boundaries can only narrow further.

**In practice:** the Glue job role gets `s3:GetObject` on `arn:aws:s3:::lake/raw/*` and `s3:ListBucket` on `arn:aws:s3:::lake` limited to that prefix, plus Glue Catalog permissions. Redshift `COPY` names the role with `IAM_ROLE`. If the bucket uses **SSE-KMS**, the role also needs `kms:Decrypt` in the key policy — the most common cause of "Access Denied" on a bucket whose policy looks right. Cross-account access needs both the bucket policy and the role policy to allow it.

Least privilege means prefix-scoped resources, no `s3:*`, and no access keys in job code. For table- and column-level control over the lake, Lake Formation adds a grant layer on top of IAM.

The reasoning

**Data Catalog** — the central metastore of databases, tables, schemas, locations and partitions. Athena, Redshift Spectrum, EMR and Glue jobs all read it, which is why it is the part of Glue used most widely.

**Crawlers** scan S3 and JDBC sources, infer schemas with classifiers and create or update catalog tables. **Connections** hold endpoints, credentials and VPC settings for data stores.

**Jobs** — Spark (Python or Scala, via `GlueContext` and DynamicFrames or plain DataFrames), Spark Streaming, and Python shell for small tasks; Glue 5.1 is the default version. Capacity is set in workers of a chosen type, measured in DPUs (4 vCPUs and 16 GB each). **Job bookmarks** track what has been processed. **Glue Studio** builds jobs visually.

**Triggers and workflows** start and chain jobs and crawlers. Around them sit Data Quality rules, the Schema Registry for streaming schemas, and DataBrew for no-code preparation.

The reasoning

**KEY** hashes one column to choose the slice. Two large tables distributed on the column they are joined by are **co-located**, so the join needs no redistribution. A table can have only **one** distribution key, so pick the join that matters most and a high-cardinality column — a key with a few dominant values puts most rows on a few slices (skew).

**ALL** keeps a full copy on every node. Right for small, slowly changing dimensions joined to everything; wrong for large or frequently updated tables, because every write happens once per node.

**EVEN** spreads rows round-robin. Right when a table is not joined, or when no key is clearly better.

**AUTO** is the default when no style is specified. AWS documents that Redshift starts a small table as ALL, may switch it to KEY (choosing a primary-key column) as it grows, and to EVEN if no column suits; `SVV_ALTER_TABLE_RECOMMENDATIONS` shows what it proposes. `EXPLAIN` confirms the effect: `DS_DIST_NONE` means co-located, `DS_BCAST_INNER` and `DS_DIST_BOTH` mean data moved at query time.

The reasoning

**Kinesis Data Streams** stores records in shards for 24 hours by default (up to 365 days). Producers write with a partition key, and any number of consumers — Lambda, the Kinesis Client Library, Flink, Firehose — read independently and can replay. Use it when something must process the data, in order per key, possibly in several ways.

**Amazon Data Firehose** has no read API. It takes records from producers or from a Kinesis stream, optionally transforms them with Lambda, converts JSON to Parquet or ORC, partitions them dynamically by fields in the data, buffers them, and delivers them. For S3 the buffer is configurable from **1 to 128 MiB** (default 5) and **0 to 900 seconds** (default 300); whichever fills first triggers a delivery.

The family also includes **Managed Service for Apache Flink** (formerly Kinesis Data Analytics) for stateful stream processing, and Kinesis Video Streams.

The decision rule: Streams when data has to be **processed**, Firehose when it only has to **land**. Many designs use both — Streams for the real-time consumers, Firehose reading the same stream to archive to S3.

The reasoning

**ADF pipelines with Copy activities and Mapping Data Flows** map to **Glue jobs** — Spark jobs written in code or built visually in Glue Studio. ADF **linked services** map to **Glue connections**, and ADF **datasets** roughly to **Glue Data Catalog tables**.

ADF's **self-hosted integration runtime**, which reaches on-premises systems, has no single AWS equivalent: a Glue connection in a VPC reaches on-premises databases over Direct Connect or VPN, and DMS or DataSync handle bulk movement.

ADF **triggers** (schedule, tumbling window, event) map to Glue triggers or EventBridge rules. ADF's role as an orchestrator of many services maps to **Step Functions**, or to **MWAA** when the team works in Airflow. The older service literally called AWS Data Pipeline is in maintenance mode and is not the answer for new work.

The reasoning

With asynchronous invocation — how **S3 event notifications, SNS and EventBridge** call Lambda — the service places the event on an **internal queue** and returns success to the caller straight away. The caller never sees the function's result or its errors.

**Retries.** If the function returns an error, Lambda retries **up to two more times** by default, with waits between attempts. If it is throttled, Lambda keeps retrying for up to the **maximum event age** (six hours by default). Both the retry count (0–2) and the maximum event age are configurable.

**Failures.** Configure an **on-failure destination** (SQS, SNS, EventBridge, another Lambda, or S3) or a dead-letter queue to capture events that exhaust their retries; without one, they are discarded, and a file-triggered pipeline silently skips a file. On-success destinations can record completions.

**Consequence.** Because of retries, a function can see the same event more than once, so the handler must be idempotent — for example, write to a deterministic output key.