AWS Theoretical

The reasoning

IAM policies travel with the identity. A role with `s3:GetObject` on `arn:aws:s3:::lake/*` can read that bucket from any service it runs in. They cannot name a principal, because the principal is whoever holds the policy.

Bucket policies live on the bucket and have a `Principal` element. That is what makes them the tool for cross-account grants, for forcing TLS with `aws:SecureTransport`, and for restricting to a VPC endpoint. They also govern public access when Block Public Access allows it.

Evaluation follows the IAM logic: an explicit deny in either policy wins over any allow. Same account: the union of allows. Cross account: the resource owner's bucket policy must allow the external principal, and the external account must also allow its identity, so a deny on either side blocks.

Access Points add a third layer with their own policies, and S3 Block Public Access overrides any policy that would grant public reads. The practical rule: identity policies for your own roles, bucket policies for other accounts and for guardrails.

The reasoning

With versioning on, `PUT` on an existing key creates a new version id and the old one stays. `DELETE` without a version id inserts a delete marker, which makes the key look gone while every version remains and can be restored by deleting the marker. A `DELETE` with an explicit version id removes that version for good.

Storage cost grows with every version, so versioning is paired with a lifecycle rule: expire noncurrent versions after N days and clean up expired delete markers. Without that rule a daily-overwritten file costs 365 copies a year.

Object Lock is write-once-read-many. A retention period is set per object or as a bucket default, in days or years. A legal hold is an on-off flag with no end date. Both block deletes of that version until lifted.

Governance mode lets principals with `s3:BypassGovernanceRetention` override, for operational mistakes. Compliance mode is absolute: the version cannot be deleted until retention expires, and retention can only be extended. Object Lock once had to be enabled at bucket creation; since late 2023 it can be turned on for an existing versioned bucket.

The reasoning

There is one catalog per account per region. A table entry records the S3 location, the input and output format, the SerDe, the columns, and the partition keys. A partition entry records one partition value set and its location. A query engine reads these to know which files to scan.

A crawler walks an S3 prefix, infers the schema and partitions, and creates or updates the table. Crawlers are convenient and the usual source of schema drift surprises, because an inferred type can change between runs. Explicit DDL or a Glue job's `getSink` with `enableUpdateCatalog` gives control.

Partition management is the operational cost. Every new daily folder needs a partition entry before Athena can see it, hence `MSCK REPAIR TABLE` or crawler schedules. Partition projection is the alternative: it computes partitions from a pattern and skips the catalog entirely.

Pricing is per object stored above the first million and per request above the first million per month. Lake Formation layers permissions over the catalog, and Iceberg tables register in it with their metadata location, which is what lets Athena and EMR share one table.

The reasoning

The design target is semi-structured data. JSON where `price` is a number in some files and a string in others fails DataFrame inference or becomes a string. A DynamicFrame reads it and marks the column `choice<double, string>`.

`resolveChoice(specs=[('price', 'cast:double')])` casts; `make_cols` splits into `price_double` and `price_string`; `project:double` keeps one type and nulls the rest; `make_struct` nests both. `applyMapping` renames and casts in one pass, and `relationalize` flattens nested structures into several frames with keys.

DynamicFrames integrate with the catalog and the Glue sinks: `glueContext.create_dynamic_frame.from_catalog` reads with partition pushdown; `write_dynamic_frame.from_options` writes and can update the catalog. Job bookmarks work on DynamicFrames read from the catalog.

For joins, window functions, SQL and most transformations, a DataFrame is the better tool and usually faster. The common pattern is to read as a DynamicFrame, resolve choices, call `toDF()`, transform, and `fromDF()` back for the sink.

The reasoning

Each worker runs one Spark executor, and the number of workers sets the parallelism. A job of 10 G.1X workers is 10 DPUs. The same 10 workers as G.2X is 20 DPUs with twice the memory per executor, which is the fix for executor out-of-memory on skewed joins.

G.4X and G.8X exist for memory-heavy workloads and are available from Glue 3.0 onward. The driver is also a worker of the chosen type. Auto scaling, from Glue 3.0, lets you set a maximum and have Glue add and remove workers per stage, which is the main cost lever.

The Glue version is a compatibility decision. Glue 5.0 brought Spark 3.5.4, Python 3.11, native Iceberg, Delta and Hudi support, and Lake Formation fine-grained access. Upgrading changes Spark behaviour, so it is a tested migration, not a flag.

Billing is DPU-hours times the regional rate, per second after the first minute. A 10-DPU job running 6 minutes is 1 DPU-hour. Flex execution, which runs on spare capacity, is cheaper for jobs that tolerate delay. Python shell jobs bill at a fraction of a DPU.

The reasoning

On EC2 you pick instance types, as the dump's exercise does with four r5.12xlarge nodes, and you tune executors to them. Capacity is yours whether a job uses it or not, and Spot instances cut cost at the risk of interruption. It supports every framework EMR packages: Spark, Hive, Presto, Trino, Flink, HBase.

Serverless removes the cluster. An application fixes the release label and the maximum capacity. Each job request runs with its own driver and executors, sized from Spark configs such as `spark.executor.memory`, and scales down to zero at the end. There is no SSH, no YARN UI, and only Spark and Hive.

Pre-initialised capacity keeps workers warm so a job starts in seconds instead of a minute or more. It is billed while idle, so it is for latency-sensitive or frequent jobs. Without it, every job pays start-up time.

The fit: Serverless for bursty, scheduled Spark jobs where paying only for the job beats managing a cluster. EC2 for long-running clusters with high utilisation, Spot-heavy cost strategies, or frameworks Serverless lacks. EMR on EKS is the third option for teams already on Kubernetes.

The reasoning

`CREATE TABLE curated.orders WITH (format = 'PARQUET', external_location = 's3://lake/curated/orders/', partitioned_by = ARRAY['order_date']) AS SELECT ... FROM raw.orders`. Athena writes the files, creates the table, and adds the partitions. The partition columns must be last in the select.

The 100-partition limit per query is enforced; a CTAS over a year of daily data fails. The workaround is `INSERT INTO` in batches of up to 100 partitions, or a loop from Step Functions or Airflow over date ranges.

Costs follow Athena's model: the query scans the source and is billed per terabyte scanned, with a 10 MB minimum per query. Converting CSV to Parquet once typically cuts every later query's scan by an order of magnitude, which is why CTAS is often the first optimisation.

CTAS cannot update in place, has no job bookmarks, and runs with Athena's query limits on time and concurrency. For incremental daily loads the pattern is CTAS once for history, then daily `INSERT INTO` for the new partition, or a Glue job when the logic grows beyond SQL.

The reasoning

The namespace is the database side: schemas, tables, snapshots, the KMS key, IAM roles for COPY. The workgroup is the compute side: base RPUs, max RPUs, network, and the endpoint your tools connect to. One namespace can be served by one workgroup at a time.

Base capacity can be set from as low as 4 RPUs up to 1,024, and Serverless adds RPUs beyond the base when concurrency or query size demands it. A max capacity setting caps the spend. More base RPUs means more memory and faster cold queries, at a higher rate per second.

Idle costs nothing for compute. A warehouse that is queried for two hours a day pays two hours. A provisioned RA3 cluster of the same power pays 24, which is why Serverless suits spiky BI and development, and provisioned suits steady 24-hour load where reserved instances get cheaper.

Features are nearly equal: Spectrum, data sharing, materialized views, federated queries. Serverless lacks a few knobs, such as manual WLM queues. It also surprises people with the 60-second minimum per burst of activity: a monitoring query every 30 seconds keeps it billing continuously.

The reasoning

With `COMPOUND SORTKEY (order_date, customer_id)`, blocks are ordered by date, then customer within date. A filter on date skips most blocks; a filter on customer alone skips almost none, because every date range contains every customer. Range scans, merge joins and `GROUP BY` on the leading columns all benefit.

`INTERLEAVED SORTKEY (order_date, customer_id, region)` interleaves the bits of each column into the sort order, so a filter on any one column prunes about equally. It suits ad hoc query patterns with no dominant column. It costs more to load, degrades as data skews, and needs `VACUUM REINDEX` to recompute.

The docs' guidance: use compound unless you specifically need equal selectivity across columns, and prefer `SORTKEY AUTO` on new tables, which observes queries and picks a key. Keep the sort key to a few columns, since more columns dilute the leading-column benefit.

Sort keys interact with distribution keys. The distribution key decides which node holds a row; the sort key decides the order within a node. A timestamp sort key with an `EVEN` or key distribution is the typical fact-table design, and new rows appended in date order stay sorted without a vacuum.

The reasoning

Redshift's storage is append-only: an `UPDATE` writes a new row and marks the old one deleted, and a `DELETE` just marks. Those ghost rows still occupy blocks and are still scanned. `VACUUM DELETE ONLY` removes them. A table that is rewritten daily by delete-and-insert without vacuum doubles in scanned size.

Rows loaded out of sort order land in an unsorted region at the end of the table, and zone maps stop pruning for it. `VACUUM SORT ONLY` merges it into the sorted part. `VACUUM FULL` does both. `VACUUM RECLUSTER` sorts only the unsorted portion and is the cheaper choice on very large tables.

`ANALYZE` updates statistics the planner uses for join order and distribution decisions. Stale statistics after a big load produce bad plans, so `ANALYZE` after a load, or `COPY` with `STATUPDATE ON`, is standard. `svv_table_info` shows `stats_off` and `unsorted` percentages to tell you which table needs attention.

Automatic vacuum delete, automatic sort and automatic analyze run during low activity and handle the steady state. They do not keep up with a 100 million row delete at 9 a.m., which is when a manual, scheduled `VACUUM` still matters. VACUUM is heavy and single-threaded per table; running it in a maintenance window is the norm.

The reasoning

Data Streams is the Kafka-like primitive: shards, partition keys, ordering within a shard, multiple independent consumers, replay within retention. You build the consumer with the KCL, Lambda, Flink or Spark, and you own the delivery logic.

Firehose is the pipe to storage. You configure a destination, a buffer size and interval, an optional Lambda transformation, and an optional conversion to Parquet or ORC using a Glue table schema. It scales without shards in direct-put mode, and it can read from a Data Stream as its source.

Buffering decides latency. The default 5 MB or 300 seconds means a file lands on S3 every five minutes at low volume. The 2023 zero-buffering option delivers within seconds for destinations that support it, at the cost of many small files. Dynamic partitioning writes to S3 prefixes from record fields, such as `year=/month=/day=`.

The choice: Firehose when the job is "land this in S3 or Redshift with minimal code". Data Streams when you need custom consumers, replay, ordering guarantees or fan-out to several processors. Both together, Streams as source and Firehose as archive, is the common pattern.

The reasoning

Sizing is arithmetic: 200,000 events per second at 2 KB is 400 MB per second, so at least 400 shards in provisioned mode. The dump's note is the right one: total throughput is not enough, because one hot partition key saturates one shard while the others idle. Partition key cardinality must be high and evenly spread.

Standard consumers share a shard's 2 MB per second and five reads per second, so three consumers polling the same shard each get a fraction and add latency. Enhanced fan-out registers each consumer, and Kinesis pushes records to each at a full 2 MB per second per shard. It supports up to 20 consumers per stream, with roughly 70 millisecond propagation. It costs per consumer-shard-hour and per GB.

Provisioned mode means you set the shard count and reshard by splitting and merging; the KCL handles shard changes in consumers. On-demand mode scales capacity with traffic, doubling peak within limits, and bills per GB written and read. It is the default suggestion for unknown or spiky loads.

Ordering holds only within a shard, so a key's events are ordered but the stream as a whole is not. Retention is 24 hours by default, extendable to 365 days. A record is at most 1 MB.

The reasoning

Without reserved concurrency, one function under a burst can consume the whole account pool and throttle every other function. Setting reserved concurrency of 50 on it limits the damage. Setting it to zero is the emergency stop that disables a function.

A cap also protects downstream systems: a Lambda writing to a database with 100 connections gets reserved concurrency of 80. With Kinesis or SQS sources, the effective parallelism is also bounded by shard count or the event source's maximum concurrency setting.

Provisioned concurrency targets latency, not throughput. The runtime, dependencies and initialisation code are already loaded, so the first request is as fast as the hundredth. It is for synchronous APIs with latency budgets, and it costs per GB-second whether invoked or not. Batch pipelines rarely need it.

The 15-minute ceiling is the design boundary: a transformation that cannot finish in 15 minutes belongs in Glue, EMR or Fargate, or must be chunked by Step Functions. Memory from 128 MB to 10 GB also sets CPU proportionally, so a slow function is often fixed by more memory.

The reasoning

A Standard workflow suits a nightly pipeline: start a Glue job, wait, run a crawler, run Athena CTAS, notify. The `.sync` integration pattern waits for a job to finish without a polling loop. Every step's input and output is recorded and visible for a year. Cost is about 2.5 cents per thousand transitions, so a long-running wait is cheap.

Express suits high-volume, short event processing: thousands of executions per second transforming records, each done in seconds. History goes to CloudWatch Logs rather than the console. Asynchronous Express may run a step twice, so steps must be idempotent.

Standard's exactly-once applies to the state machine's bookkeeping; a task it invokes can still fail halfway and be retried by your `Retry` block, so idempotency in the task still matters. Both types support `Retry` and `Catch` per state, error matching, and backoff.

Distributed Map, from late 2022, reads an S3 inventory, a JSON array or CSV and launches child workflows per item or batch. Concurrency goes up to 10,000, with tolerance for a failure percentage. It is how Step Functions processes a million files without a Spark cluster.

The reasoning

Deployment is a file copy: `aws s3 cp dags/ s3://my-mwaa-bucket/dags/ --recursive`, and the environment picks up changes within a short sync interval. requirements.txt and plugins.zip are versioned objects, and changing them triggers an environment update that restarts components, which takes tens of minutes.

Scaling is by workers. Each environment class fixes the vCPU and memory per worker and the default tasks per worker. You set min and max workers, and MWAA adds workers when queued tasks exceed capacity. Scheduler count can be raised for parse-heavy environments.

Networking is yours: the environment lives in your VPC's private subnets, with public or private web server access. Connections and variables can come from Secrets Manager via the built-in secrets backend. Logs go to CloudWatch per component, which is where debugging happens without shell access.

Limits that shape design: the Celery executor only, with no custom executor, and Airflow versions limited to those MWAA supports, with upgrades as environment updates. Packages install only through requirements.txt, with a constraints file to keep Airflow itself compatible. Heavy compute belongs in EMR, Glue or ECS called from DAGs, not in the workers.

The reasoning

The replication instance reads from the source's transaction log for CDC, which requires the source to be configured for it. That means binlog in row format on MySQL, and logical replication or supplemental logging on PostgreSQL and Oracle. Instance size sets memory for the change cache and the number of concurrent tables.

Endpoints are reusable across tasks and tested from the console. Extra connection attributes tune behaviour: on an S3 target, `dataFormat=parquet`, `cdcInsertsAndUpdates`, `includeOpForFullLoad`, and `timestampColumnName` are the ones data engineers set, because they decide what the lake receives.

Full load copies tables as they are, in parallel, with no ordering guarantee between tables. CDC only starts from a position in the log and streams changes. Full load plus CDC does the first and then switches to the second, caching changes that occur during the load, which is the migration and lake-ingestion default.

On S3 the CDC files carry an `Op` column with I, U or D, and a timestamp. Downstream, a merge job applies them to a Delta or Iceberg table, deduplicating by primary key and latest timestamp. Task monitoring through CloudWatch shows latency between source and target, which is the health signal.

The reasoning

When a Glue job runs with a role, Glue's service principal assumes it, so the trust policy must name `glue.amazonaws.com`. The permissions policy then grants S3 and catalog access. A trust policy that names the wrong service gives "not authorized to perform sts:AssumeRole", the commonest Glue setup error.

Cross-account access is the same mechanism: account A's role trusts account B's principal, and B's principal has `sts:AssumeRole` on A's role ARN. B calls STS, receives A-scoped temporary credentials, and acts in A. An external id in the trust policy guards against the confused-deputy problem.

Credentials are temporary by design. `DurationSeconds` ranges from 900 seconds to the role's maximum session duration, which an administrator sets up to 12 hours. Role chaining, assuming a role from an assumed role, is limited to one hour. Long-running jobs rely on the SDK refreshing automatically.

The practical distinction: the trust policy answers who, the permissions policy answers what, and a permission boundary or SCP can cap both. Reading an access denied means asking which of the three said no.

The reasoning

Both use envelope encryption: a unique data key per object encrypts the bytes, and the data key is itself encrypted by the master key. With SSE-S3 the master is S3's. With SSE-KMS it is a KMS key, either AWS managed, `aws/s3`, or customer managed, which is the one whose policy you write.

The control SSE-KMS adds is the second gate. A role with `s3:GetObject` but no `kms:Decrypt` on the key gets access denied, and the key policy can span accounts. Disabling the key makes every object unreadable at once, which is the crypto-shredding pattern for deletion requests.

Audit is the other gain. Every decrypt appears in CloudTrail with the principal and the object context, which SSE-S3 cannot provide. Compliance regimes that ask "who accessed this data" are usually answered with SSE-KMS plus CloudTrail data events.

Costs: a customer managed key is about one dollar per month, and requests are billed per 10,000 beyond a free tier. A pipeline reading millions of small objects can generate millions of decrypt calls and hit the KMS request quota. S3 Bucket Keys derive a bucket-level key so that most objects stop calling KMS. SSE-C, where you supply the key per request, and DSSE-KMS, dual-layer, are the other two options.

The reasoning

Lake Formation sits between the Glue catalog and the engines. When Athena, Redshift Spectrum, EMR or Glue reads a table registered with Lake Formation, the engine asks Lake Formation for permissions and receives temporary, scoped S3 credentials. IAM policies on S3 are replaced by Lake Formation grants for registered locations.

Named-resource grants are `GRANT SELECT ON TABLE sales.orders TO role analyst`, one statement per table and principal. They are precise and do not scale: a hundred tables and twenty roles is two thousand grants, and each new table needs new ones.

With LF-tags, the administrator tags once and grants once: analysts get `SELECT` where `classification=public`; the finance role gets `classification=pii` as well. A new table tagged at creation, or inheriting from its database, is immediately governed. Column tags let `email` carry `pii` while the rest of the table is `public`, which gives column-level security without views.

Row-level security comes from data filters: a filter expression such as `region = 'EU'` attached to a grant. Hybrid access mode lets a table be governed by both IAM and Lake Formation during a migration. The trade is a steeper setup and engine support that varies by feature.

The reasoning

CloudWatch Metrics are time series emitted by services and by your code; Alarms watch them and notify through SNS. CloudWatch Logs collects log streams from Lambda, Glue, MWAA and agents, with Logs Insights for querying. A pipeline's health dashboard and failure alerts live here.

CloudTrail records management events, API calls that change configuration, free for the last 90 days in the event history and delivered to S3 or CloudWatch Logs for a trail. Data events, such as individual S3 `GetObject` calls or Lambda invocations, are opt-in and billed, because of their volume.

A `DeleteTable` call to the Glue catalog, a `DeleteObject` on S3 with data events on, a change to an IAM policy: each is one CloudTrail record. The record carries the IAM identity, including an assumed role's session name. Athena over the trail bucket is the standard way to query it.

They connect: a CloudTrail trail can deliver to CloudWatch Logs, where a metric filter and alarm notify on a sensitive call such as a bucket policy change. CloudWatch cannot tell you who; CloudTrail cannot tell you how slow.

The reasoning

An object is bytes plus metadata under a key such as `lake/orders/dt=2026-11-01/part-0001.parquet`. There are no directories; the slashes are part of the key, and prefixes are what listing and the request limits operate on. Renames are copies, which is why Spark's commit protocols matter on S3.

Strong consistency means a `GET` or `LIST` after a `PUT` or `DELETE` returns the new state immediately, which removed the old need for consistent-view layers like EMRFS. It does not make S3 transactional across objects, which is the gap table formats fill.

The per-prefix request rate scales with more prefixes, so a layout with many partitions parallelises. Small files are the enemy: each object is a request, so millions of small files make listing and reading slow and expensive, which the dump notes as the HDFS comparison. Compaction to 128 MB to 1 GB files is the standard fix.

Storage classes trade retrieval time and cost: Standard, Intelligent-Tiering, Standard-IA, One Zone-IA, Glacier Instant, Glacier Flexible, Deep Archive. Lifecycle rules move objects between them by age. Pricing is per GB-month, per request and per GB transferred out.

The reasoning

The leader node receives SQL, compiles it, and distributes execution steps to the compute nodes. Each compute node has slices, and each slice owns a portion of every table according to the distribution style. `KEY` co-locates rows by a column, `EVEN` round-robins, `ALL` copies small tables to every node, and `AUTO` chooses.

Columnar storage with compression encodings per column, zone maps per 1 MB block, and sort keys give the scan efficiency. Queries that touch few columns of a wide table read a fraction of the bytes. The planner relies on statistics from `ANALYZE`.

RA3 node types, the current generation, keep hot data in local SSD cache and the full dataset in managed storage, billed separately. That decoupling is what allows concurrency scaling clusters and data sharing to read the same storage. DC2 nodes with local storage remain for small, fixed workloads.

Loading is `COPY` from S3 in parallel, one file per slice ideally; single-row inserts are slow. Spectrum extends queries to Parquet on S3 through the Glue catalog. Redshift is the AWS answer when a SQL warehouse with predictable performance is wanted next to a lake.

The reasoning

Triggers include S3 events, Kinesis and DynamoDB streams, SQS queues, EventBridge rules and schedules, API Gateway and direct invocation. Each delivers an event payload to the handler. Stream and queue sources are polled by Lambda on your behalf in batches.

Memory sets CPU and network proportionally, so a 1,769 MB function gets one full vCPU. Cold starts add initialisation time on new instances; provisioned concurrency removes them where latency matters. Ephemeral storage up to 10 GB and container images up to 10 GB are available.

In data engineering the fit is small and event-driven. Validate a landed file and move it, fan out notifications, call an API, enrich records in flight, run light Python over a batch. Heavy or long transformations exceed the 15-minute limit and belong in Glue, EMR or a container.

Failure handling is per source. Asynchronous invocations retry twice and go to a dead-letter queue or destination. Stream sources retry the batch until success or expiry, with bisect-on-error and a failure destination to avoid a poison record blocking a shard.

The reasoning

A policy statement has Effect, Action, Resource and optional Condition. `{"Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::lake/curated/*"}` grants one read. Identity-based policies attach to users, groups and roles; resource-based policies attach to buckets, keys and roles' trust.

Evaluation is deny by default, explicit deny wins, and allows from any applicable policy are combined, within the ceilings of service control policies and permission boundaries. Conditions add context: source IP, VPC endpoint, MFA, tags, time.

Roles are the operational unit. A Glue job, an EC2 instance, a Lambda and an EKS pod each assume a role and receive rotating temporary credentials. Humans federate in through an identity provider and assume roles too. Access keys on users are the legacy path and the main source of leaked credentials.

Data engineers meet IAM in every service call: a job's role needs the S3, catalog, KMS and Lake Formation permissions its work touches, and nothing more. IAM Access Analyzer and CloudTrail show what a role actually uses, which is how least privilege is reached.

The reasoning

Access is by key: `GetItem` and `PutItem` on the primary key, and `Query` on a partition key with a sort key range. `Scan` reads the whole table, which is slow and expensive. Design starts from the queries, because there are no joins and secondary indexes are the only other access paths.

Global secondary indexes give alternative keys with their own capacity; local secondary indexes share the partition key. Hot partitions throttle, so the partition key must spread load, the same lesson as Kinesis.

DynamoDB Streams, or Kinesis Data Streams for DynamoDB, emit every item change, which is how a table feeds a lake or a Lambda. Export to S3 writes a point-in-time snapshot without consuming capacity, and the zero-ETL integration to Redshift and OpenSearch is the analytics path.

Pipelines use it as a control plane: a table of processed file names for idempotent S3 ingestion, or a watermark per source. It also serves as a lock for a singleton job, or a cache of lookups a Lambda enriches with. TTL expires rows automatically. Conditional writes give the compare-and-set that locks need.

The reasoning

A consumer calls `ReceiveMessage`, processes, then `DeleteMessage`. Between those, the message is invisible to others for the visibility timeout. If processing takes longer, another consumer receives it again, which is the usual cause of duplicate processing on standard queues. Set the timeout above the processing time or extend it.

A dead-letter queue receives messages that were received more than `maxReceiveCount` times without deletion, isolating poison messages. Long polling, `WaitTimeSeconds` up to 20, reduces empty receives and cost.

FIFO queues order by message group id and deduplicate by content hash or an explicit id for five minutes. The group id also bounds parallelism: one group is consumed in order by one consumer at a time. High throughput mode raises FIFO limits by partitioning per group.

In pipelines SQS buffers S3 event notifications for Lambda or a Spark structured streaming source. It smooths bursts to a rate a database can absorb, and is the retry layer in front of an API. It is not a stream: once deleted, a message is gone, and there is no replay.

The reasoning

A topic is the address. Publishing is one API call; delivery is SNS's job, with retries per protocol. Standard topics are at-least-once and unordered; FIFO topics, which deliver only to FIFO queues, preserve order and deduplicate.

Subscription filter policies let each subscriber receive only messages whose attributes match, so one topic can route by event type without a consumer discarding most of what it gets.

The fan-out pattern solves the problem that an S3 event notification can go to only one target per event type. Send it to SNS and subscribe three queues. Three pipelines then each process the file independently with their own retries and dead-letter queues.

SNS is also the alerting channel: CloudWatch alarms publish to a topic, and the topic emails, pages and invokes a Lambda. It stores nothing; a subscriber that is down misses the message unless it is a queue.

The reasoning

Every account has a default bus that receives AWS service events. Examples: S3 object created, with EventBridge notifications enabled on the bucket; Glue job state changes; EMR step completions; DMS task events. Custom buses hold your own domain events. Partner buses receive from SaaS vendors.

A rule is an event pattern plus targets. `{"source": ["aws.s3"], "detail-type": ["Object Created"], "detail": {"bucket": {"name": ["landing"]}}}` fires on each landed object and can start a Step Functions execution with the key as input. Input transformers reshape the event for the target.

Scheduler, split out in 2022, is the cron replacement: one-off or recurring schedules with time zones and flexible windows, targeting any API. It is how a nightly Glue job starts when there is no Airflow.

Archive and replay store events on a bus and replay them to rules, which is the recovery tool when a consumer was broken. Pipes connect a source such as SQS or Kinesis to a target with filtering and enrichment, point to point, without a rule.

The reasoning

Each event is JSON with `eventName`, `eventSource`, `userIdentity` including the assumed role session, `sourceIPAddress`, `requestParameters` and `responseElements`. That is enough to reconstruct a deletion or a permission change.

Management events cover every service's configuration calls. Data events are opt-in per resource type: S3 object-level operations on chosen buckets, Lambda `Invoke`, DynamoDB item operations. Enabling S3 data events on a busy lake produces huge volumes, so teams scope them to sensitive buckets.

A trail writes to an S3 bucket, where Athena with a partitioned table over the trail prefix is the standard query interface, and CloudTrail Lake offers a managed SQL store. Organisation trails collect every account in an organisation into one bucket.

For a data platform, CloudTrail is the evidence for access reviews, the source for alerts on dangerous calls, and the first stop when something disappeared. It records calls, not query text; what someone selected inside Redshift is in Redshift's own audit logs.

The reasoning

Without an endpoint, a Glue job or EMR cluster in a private subnet reaches S3 through a NAT gateway, which bills per GB processed. A Spark job that shuffles terabytes through S3 makes that the largest line item. The gateway endpoint removes it.

The gateway endpoint works through routing: a prefix list for S3 in the region points at the endpoint. It is regional and reachable only from inside that VPC. An endpoint policy can restrict which buckets are reachable, and a bucket policy can require `aws:sourceVpce` so the bucket is reachable only from that endpoint.

The interface endpoint gives S3 a DNS name and private IP inside your subnets. A data centre connected by Direct Connect or VPN can then reach S3 privately, and security groups apply. It is the choice when the gateway's routing model does not fit.

Glue connections to a VPC, EMR in private subnets, Redshift enhanced VPC routing and MWAA environments all expect an S3 endpoint. Their failures without one show up as timeouts rather than access denied, which is the diagnostic clue.