Data Pipeline Conceptual

The reasoning

Start from why at-least-once is unavoidable. A sender transmits, the receiver processes, the acknowledgement is lost on the way back. The sender now cannot distinguish between a message that never arrived and one that arrived and was processed, so it has two options: send again, or do not. Sending again gives at-least-once. Not sending gives at-most-once. There is no third option, and this is a property of unreliable networks, not of any particular product.

What a system like Kafka or Flink can genuinely offer is exactly-once *within a closed world*: offsets and output are committed in one transaction, so a restart cannot double-apply anything the system itself controls. That is real and it is valuable. It stops applying the moment the output leaves — a warehouse, an API, an email — because the platform can no longer make the write and the offset atomic.

So the once-ness you can actually rely on end to end is built at the write. Give every unit of work an identity the sink can recognise, and let the sink discard a repeat. The run below is the whole idea in miniature: the broker delivers the same batch twice, which is normal and not a fault, and the write keyed on event_id leaves the table identical. The industry name for this is effectively-once, and it is what every production system you have used is actually doing.

The reasoning

The two orderings are exhaustive, and each has a window in which the process can die. Commit-then-write dies between the two and the batch is gone — the offset has moved past data nobody wrote, and there is no record that it existed. Write-then-commit dies between the two and the batch is written but not accounted for, so the restart reads it again and writes it again.

The program below runs both against the same four messages and the same single crash. Committing first loses a and b entirely. Writing first duplicates them. Neither is a bug in the consumer; both are the direct consequence of a choice someone made in two lines.

The right default is write-then-commit, because duplication is recoverable and loss is not — you can always deduplicate a row you have, and you can never reconstruct one you dropped. Then the third line of the output is the actual design: the same at-least-once consumer, writing through a sink that recognises a key it has already seen, produces exactly the four messages. The guarantee was built at the sink, not at the offset.

The reasoning

The trade is between two failure costs and one running cost. At-least-once requires the sink to deduplicate, which means holding keys, doing lookups, and paying for both at your busiest moment. At-most-once requires nothing and occasionally drops something. Which is correct depends entirely on what one record is worth.

For a payment, an order or an audit record, a lost row is unrecoverable and possibly reportable, so you pay for at-least-once without discussion. For per-request latency metrics arriving at a million a second, one lost sample changes no percentile anyone will ever look at, and the deduplication infrastructure would cost more than the entire signal is worth. Dropping is the engineering decision, not the accident.

The rule worth stating out loud is that the guarantee should be chosen per stream and written down, not inherited from whatever the default was. And the uncomfortable version of the same point: if you cannot say what one lost record costs, you cannot justify the money you are spending to prevent it — which is a conversation many teams have never had about their most expensive pipeline.

The reasoning

A keyed merge assumes the batch is a set of *changes* and that the key you merge on is the grain. It cannot delete, because a key absent from the batch means nothing to it. Partition overwrite assumes the batch is a *complete* window, and in exchange it can delete by absence — which is the only way a re-extract removes a row that vanished upstream.

Dedupe-on-insert assumes each record carries a stable unique identity and that records are never revised. It is the cheapest of the four and the right choice for event streams, where an event happened and does not change afterwards. Full refresh assumes the batch is the entire table, which makes it trivially correct and trivially expensive, and it remains the right answer for small dimensions far longer than most people expect.

The failure mode is always the same: applying a pattern whose assumption the batch does not satisfy. A keyed merge on a partial extract leaves deleted rows alive. A partition overwrite on a change-feed batch destroys everything in the partition the batch did not happen to mention. So the useful question in a design review is not “is this idempotent” but “what does this assume the batch is, and is that what the source sends?”

The reasoning

Retries have two independent dimensions and people usually set only one. The first is the *budget*: attempts multiplied by backoff is a total tolerance, and it should be chosen against how long the thing you depend on typically takes to come back. The program below makes the arithmetic concrete — three attempts with exponential backoff buys seven seconds of tolerance, which loses to any upstream that restarts, while eight attempts buys enough to survive it.

The second dimension is *safety*, and no retry setting anywhere addresses it. A retry re-executes the task, so if the task is not idempotent the retry policy is a duplication policy — and the more generous your budget, the more copies you make. This is the connection worth stating explicitly, because the two settings are usually owned by different people: the platform team sets retries, the data team writes the load, and nobody checks that one is safe under the other.

Two more things to raise unprompted. Retry only what is retryable — a 503 is worth retrying and a 400 never will be, so blanket retries on a malformed request just burn the budget and delay the alert. And add jitter: without it, everything that failed at the same moment retries at the same moment, which is how a brief upstream wobble turns into a sustained stampede that keeps it down.

The reasoning

Boundedness is the root difference and everything else follows from it. A batch job is handed a set that is complete by definition — a day, a file, a partition — so it can sort, join, aggregate, and know that its answer is final. It can also fail and start again from the beginning, because the input has not moved.

A stream has no end, so the job can never say “that is all of them”. Every aggregate is a running answer over what has arrived so far, which is why streaming brings in windows to impose artificial boundaries, watermarks to decide when a window can be answered, and state that has to survive restarts because there is nothing to replay from the beginning of.

The second-order consequences are what actually shape the work. Correctness in batch is a property you can check after the fact by recomputing; in streaming it is a property of the running system and its state. Failure in batch means rerun; in streaming it means restore state from a checkpoint and resume from an offset. Late data in batch is next run's problem; in streaming it is a design parameter you have to choose a number for. Latency differs too — but it is the symptom, and every one of these is a cost you take on.

The reasoning

A micro-batch engine slices the stream into bounded intervals and runs a normal batch computation over each. Because each slice is bounded, you get everything batch gives you: an input you can reason about, a job you can rerun, an answer you can recompute and compare. Spark Structured Streaming in its default mode works this way, and so does a five-minute scheduled job reading a change feed — which is worth saying because the second one needs no streaming platform at all.

What you give up is genuine per-event processing. If a requirement is stated in single-event terms — block this transaction before it settles, alert within two seconds of the reading — a micro-batch cannot meet it, because the interval is the floor and no amount of tuning goes below it. That is a hard boundary, not a tuning problem.

The reason it matters commercially is that the interval buys back operational simplicity that continuous streaming charges a lot for: no long-lived state to manage, no checkpoint compatibility to worry about across deploys, failure recovery that is just “run that interval again”. For the large majority of pipelines described as needing streaming, a short interval meets the requirement at a fraction of the operational cost — and being able to say which requirements it cannot meet is what makes that a judgement rather than a preference.

The reasoning

The productive question is never “do you really need it”, which sounds like pushback. It is “what happens differently if this arrives at 09:00 rather than 09:15?” Genuine streaming requirements answer that instantly and specifically: the transaction is blocked before it settles, the alert reaches an on-call engineer before the tank overflows, the recommendation changes while the session is still open. Those are real and they justify the cost.

The other answers reveal the real requirement. “The morning meeting is at 09:30” is a deadline, not a latency — a batch that finishes by 09:00 meets it completely. “Last time the data was a day stale and we made a bad call” is a freshness *floor*, usually satisfied by hourly. “It feels slow” is often about the dashboard query, not the pipeline, and no amount of streaming will fix it.

State the cost honestly rather than as a veto, because that is what makes you credible. Streaming means always-on infrastructure, on-call for a system that has no natural quiet period, state and checkpoints to manage across deploys, harder correctness testing, and reprocessing that is a project rather than a rerun. If the requirement is real, all of that is worth it. The job is to find out which it is before building, not after.

The reasoning

A window over an unbounded stream needs a rule for when to stop waiting, because there is no end of input to wait for. The watermark is that rule — an assertion that no event older than this will still arrive. When it passes a window's end the window is answered and its state is released, which is the part that matters operationally: state cannot be held open forever, so closing is not optional.

Allowed lateness is the deliberate slack you add. In the program below the event at second 6 arrives at second 14, after its window would otherwise have closed; with zero lateness it is dropped, and with five seconds of lateness it is counted, moving the first window from 2 to 3. The event at second 4 arriving at second 21 is beyond both and is dropped either way. Every extra second of lateness is state held longer and an answer published later — that is the actual trade, and it is why the number should come from measured arrival lag rather than from a guess.

The part people miss is that dropped events are dropped *silently*. The job stays healthy, the counts look plausible, and the loss is invisible unless you asked for it — which is what a side output or dead-letter stream is for. If you cannot say how many events your streaming job discarded yesterday, you do not know whether your lateness setting is right, and the number is usually the first thing an interviewer follows up on.

The reasoning

Lambda's premise was that the fast path was approximate and the slow path was correct. You served the streaming result for freshness and let a nightly batch overwrite it with the authoritative number. Given the state of stream processing when it was proposed, that was a sound engineering response to a real limitation.

Its cost was the thing that eventually killed it: the same business logic implemented twice, in two engines, by people under different deadlines. They drift. Every discrepancy investigation starts with “which layer is this from”, and every change has to be made and tested twice. That maintenance burden is the standard argument against it and it is a fair one.

Kappa's premise is that if the stream processor is accurate and the log is replayable, the batch layer is redundant — to reprocess, you replay from the beginning with new code and swap the output. That works when the log genuinely retains enough history, which is a retention bill, and when replaying years of events is actually feasible, which at large volumes it often is not. So the honest modern position is that most platforms are neither: a replayable raw layer with batch reprocessing over it, and streaming only where the latency requirement is real. The lesson that survived from Lambda is *one* implementation of the logic, and the lesson from Kappa is that reprocessing must be a first-class operation.

The reasoning

In ETL, data is transformed in flight and only the finished shape lands. That was the right design when warehouse storage and compute were expensive and fixed: you did not load anything you were not going to use. Its defining property is that the raw form is discarded, so a transformation bug loses the data that would have let you fix it, and a new requirement means going back to the source.

In ELT, raw data lands first and is transformed inside the warehouse. Cheap storage and elastic compute made this viable, and the consequence that actually matters is recoverability: the raw layer is still there, so a bug is fixed by rerunning the transform rather than by re-extracting, and a new question can be answered from data you already have. That single property is why ELT won for most workloads.

The rest of the consequences follow. Compute moves onto the warehouse bill, which is elastic and therefore easier to scale and easier to waste. Logic moves into SQL, which widens who can own it — analytics engineers rather than only data engineers — and that is a genuine organisational change, not just a technical one. And governance moves earlier: raw now contains everything the source had, including fields you did not want, which is the point the next question is about.

The reasoning

Raw exists to be a faithful record of what arrived, which means it is append-only and never edited. That buys replay: if a transform was wrong, you fix it and recompute, because the input is still there exactly as received. Deduplicating raw would destroy the property you built it for — you would no longer know what the source actually sent, only what you decided to keep.

So duplication is pushed up rather than prevented, and the projection has to absorb it. That is a stronger requirement than it sounds. In the run below the projection picks the latest row per customer, and it is only idempotent because it breaks ties deterministically — with the same record present twice at the same timestamp, a projection written as “where ingested_at is the maximum” emits both copies, and curated inherits exactly the duplication raw was allowed to have.

The general rule is worth stating: each layer publishes a contract and the layer above is written against it. Raw promises completeness and fidelity, not uniqueness. Curated promises one row per entity and must therefore be a function of the set of raw rows, insensitive to how many copies of any of them exist. When curated breaks because raw had a duplicate, the bug is always in the projection — raw did exactly what it promised.

The reasoning

The strongest forcing function is legal. If you have no lawful basis to store a field, landing it and deleting it later is still storing it, and 'the transform drops it' is not a control that survives an audit. Card numbers, health identifiers and biometric data are commonly in this category, and the answer is to tokenise or mask *in flight*, so the raw value never reaches your storage at all. Right-to-erasure obligations point the same way: a value you never landed is a value you never have to prove you deleted.

The second is economic and simpler. Landing a payload you will discard 99% of — verbose nested API responses, high-volume debug telemetry — means paying storage, scan and often ingestion cost for data with no consumer. Filtering and flattening before the load is straightforwardly cheaper, and the trade you are making is explicit: you lose the ability to answer questions about what you dropped.

The third is about coupling, and it is the one people argue over. Landing a source system's raw shape and modelling directly on it makes every downstream model depend on that system's internals, so a vendor's schema change becomes your incident. A light normalising step at the boundary — stable names and types, a declared contract — costs little and contains the blast radius. In practice most good platforms are ELT with exactly this thin T in front, which is worth saying because it stops the question being a binary.

The reasoning

A full refresh has properties that are very hard to buy back. It cannot drift, because it recomputes from the source every time. It self-heals — any bug you fixed is corrected everywhere on the next run, with no backfill. It handles deletes without being told, because a row absent from the source is simply absent from the result, as the run below shows when Germany disappears. And it needs no watermark, no state and no reasoning about boundaries.

Every incremental load gives those up. It carries state, and the state can be wrong. It has a boundary, and rows can fall through it. It cannot see a delete unless the source explicitly tells it. It does not self-heal, so a fixed bug needs a backfill for every affected partition. Look at the Failure & Recovery bank in this section: essentially every question there is a failure that only exists because a load was incremental.

So the argument should run from full refresh outward. Refresh until it stops fitting — until the scan cost is material, until it cannot finish inside the SLA, until it puts unacceptable load on the source. Then go incremental deliberately, and build the reconciliation that tells you it is still right, because you have just taken on the responsibility the refresh was carrying for you. A daily full refresh of a hundred-thousand-row dimension is not naive; it is the correct answer, and it will still be correct in three years.

The reasoning

Two forces pull in opposite directions. Pushing logic downstream keeps it flexible and cheap to change, because fewer things depend on it. Pushing it upstream makes it consistent, because everyone downstream inherits one definition rather than writing their own. The right position is where those balance for that specific piece of logic, which is why the answer differs within one platform.

The test that resolves most cases is: how many consumers need this to mean the same thing? A metric definition — active customer, net revenue, churn — needs to be identical everywhere, so it belongs in the warehouse where there is one implementation and one place to change it. A currency format or a chart's date grouping matters to one dashboard, so it belongs in that dashboard. Putting the first in BI gives you six definitions of revenue; putting the second in the warehouse gives you a modelling change request every time someone wants a different axis.

Two positions deserve specific warnings. Logic in the *source system* is the most consistent and the least changeable — you now need another team's release cycle to alter a definition, and you have coupled their operational system to your reporting. Logic in the *ingestion job* is the worst of the options for anything business-meaningful, because it is applied before raw is retained, so it is the one place you cannot change your mind about later.

The reasoning

Unpack the claim, because everything follows from it. “Everything up to here is processed” entails that the pipeline will never look below the watermark again — which means any row that appears below it later is lost permanently and silently. That is why a watermark is a much stronger statement than a bookmark, and why putting it on the wrong column is so damaging.

It is therefore only valid on a column with two properties. It must be *monotonic from your point of view*, so that “above the line” reliably means “not yet seen” — which is true of an arrival-stamped ingest time and false of an event time that arrives out of order. And its ordering must match the order you actually receive things in, which is why a commit-ordered LSN works and a statement-time timestamp on a long transaction does not.

There is a second, quieter assumption: that the watermark advances only after the data below it is durably written. Advancing it first — or in a separate commit — makes the claim false at exactly the moment a crash occurs, and the loss is the same permanent silent kind. So a watermark is not one decision but three: which column, whether the comparison is inclusive, and when the value is allowed to move. Each of the three has its own failure, and each is demonstrated in the Failure & Recovery bank.

The reasoning

A high-water mark on a modified timestamp is the cheapest and needs only a column you can trust. That trust is the whole risk: it must be set on every update including the ones done by triggers, batch jobs and the occasional manual fix, and it must never be backdated. It also cannot see hard deletes, which is the gap that gets discovered late.

CDC reads the database's own log, so it catches everything including deletes and needs no cooperation from application code — but it needs privileged access, a connector to operate, and a DBA willing to have replication slots on a production system. Snapshot diff — extract everything, compare with what you have — requires no source support at all, which is why it is the fallback for systems that offer nothing, at the cost of extracting the whole table every run and storing two copies to compare.

Source-emitted events are the strongest option and the least available: the application publishes a domain event whenever something meaningful happens, so you get intent rather than inferred row changes. It requires another team to build and maintain something for you, which makes it an organisational decision rather than a technical one. The senior move in an interview is to name what each option needs and then say which you would ask for first, knowing you may be refused.

The reasoning

Take them one at a time. *Set on every change* fails the moment a batch job or a manual fix updates rows without touching the column — the change is real, the pipeline never sees it, and the row is stale forever. *Never backdated* fails when a bulk correction writes a historical timestamp; those rows land below your watermark and are skipped.

*Commit time rather than statement time* is the subtle one. If the column is set when the statement runs but the transaction commits thirty seconds later, then a row can become visible with a timestamp already below your watermark — you read at 10:00:15, the row says 10:00:00 and appeared at 10:00:30, and it is gone. This is the most common cause of unexplained missing rows in an otherwise correct incremental load, and it argues for reading with a safety margin behind the watermark rather than exactly at it.

*A clock you share* fails on replicas and across regions, where lag means the max timestamp you can see is behind what has been committed elsewhere. And *deletes are visible* usually fails outright, because a hard delete leaves no row to carry a timestamp. The productive framing is that these are contract terms, not implementation details — they should be written down, agreed with whoever owns the source, and where possible asserted on rather than assumed.

The reasoning

The decision has a data-driven answer that almost nobody computes. Measure the gap between event time and ingest time across a representative period, and you get a distribution: what fraction of yesterday's events had arrived by midnight, by 02:00, by noon. That converts an argument into a choice — waiting until 02:00 gets you 99.4% and costs two hours, and the business can decide whether the last 0.6% is worth it. Without the measurement, the cutoff is someone's guess defended as experience.

Whichever point you choose, the obligation is the same: the consumer must know. A number that is 94% complete is useful when it is labelled 94% complete and dangerous when it is presented as final, because someone will compare it with the restated version next week and conclude the pipeline is broken. Marking recent partitions provisional, or publishing the completeness estimate beside the figure, is what turns restatement from an incident into an expected behaviour.

Then design for the restatement rather than against it. Recent event-time partitions get recomputed on a rolling window sized from that same distribution; downstream aggregates over that window are recomputed too, not written once. And the arrival-lag distribution is itself monitored, because it drifts — a new mobile client or a new region shifts the tail, and a window that was right last quarter silently stops being.

The reasoning

Start by being clear about what a green run proves: that the code executed without raising. Silent duplication, a watermark skipping a row, a dropped delete and a dedupe key above the grain all execute perfectly. The signals teams usually rely on are equally weak — row counts are stable in most of those failures, freshness checks pass because rows did arrive, and null checks pass because the values that are there are fine.

What actually detects them is reconciliation: compute the same answer by a second, independent route and compare. Against the source where you can — row counts and sums per partition, straight from the system of record. Or a periodic full recompute over the raw layer, compared against what the incremental pipeline built, which is exactly the comparison the runs throughout this section publish. Two different computations agreeing is real evidence; one computation succeeding is not.

Make it cheap enough to run often and specific enough to act on. Reconcile recent partitions nightly and older ones on a rolling sample, so cost stays bounded. Alert on the *difference* rather than on a threshold nobody tuned. And treat a reconciliation gap as a real incident with a root cause, because in every case in this section the gap was the only signal the system ever produced.

The reasoning

Each partition is an append-only log read strictly in sequence. Between partitions there is no relationship at all: two records written a millisecond apart to different partitions can be consumed in either order, minutes apart, by different consumers.

That turns ordering into a design decision rather than a platform property. Because the partition is chosen by hashing the key, **you get ordering exactly for the entity you keyed on**. Key by `order_id` and every order's lifecycle is ordered. Key by `user_id` and each user's actions are ordered — but the events of a single order placed by two different services are not, if those services key differently.

The practical consequence is that anything requiring global ordering has to be bought elsewhere, and it is expensive: a single partition, which caps throughput at one consumer, or a sequence number assigned by a central authority, which reintroduces the bottleneck you used a stream to remove. In practice you almost never need global ordering — you need per-entity ordering, which is free once the key is right.

And the guarantee is only as strong as delivery. Most streams are at-least-once, so a consumer may see a record twice. Ordered and deduplicated are different properties: ordering says the second copy arrives after the first, not that there is only one. That is why the event id and the version field exist alongside the key.

The reasoning

**Append** is right when rows are immutable facts that only ever accumulate — sensor readings, page views, an event log. It is the cheapest write and it has one fatal property: **it is not idempotent**. A rerun after a partial failure appends the same rows again, so append is only safe when something upstream guarantees each batch is delivered once, or when a downstream dedup step exists.

**Merge** (upsert) is right when a business entity can be updated or resent — an order whose status changes, a customer record, a CDC stream. Keyed on the business key, a merge is naturally idempotent: applying the same row twice leaves the same result. It costs more per row and it needs a genuine key, which is the part that goes wrong — a merge on a key that is not unique in the source silently updates the wrong row or fails.

**Overwrite**, scoped to a partition, is right when you can recompute a bounded slice from scratch — a day, a region, a batch window. Replacing `date=2026-04-20` with a fresh computation is idempotent by construction and it is the simplest thing to reason about during a backfill. The danger is scope: an unscoped overwrite replaces the whole table, and the difference between `overwrite` and dynamic partition overwrite is a configuration setting people discover after losing history.

**The test that decides it: run the job twice on the same input. Append duplicates, merge converges, partition overwrite converges.** If your load is append and the answer to "what if this runs twice" is "we would have to clean up", you have chosen the write mode for its cost rather than its semantics.

The reasoning

The mechanism is a consumer that cannot keep up refusing to accept more, and that refusal travelling back up the chain until whatever is producing slows down. The point is not speed, it is **bounded memory**: without it, a slow stage buffers in RAM until the process dies, and it dies at the moment of peak load, which is the worst possible time.

Inside one framework this is usually built in. Flink propagates backpressure between operators; a slow sink slows the operator before it, and so on back to the source. That is why a Flink job under load gets *slower* rather than falling over, and why its backpressure metric is the first thing to look at when latency climbs.

Across a **durable bus**, it works differently and this is the part that gets answered badly. Kafka and Kinesis do not push back on producers when consumers are slow: the producer keeps writing, the log keeps growing, and the consumer simply falls further behind. That is not a failure of the design, it is the design — the bus is a shock absorber, and consumer lag is the signal that replaces backpressure. The bound is retention, so the honest number to monitor is not lag in records but **headroom in hours**: how long you can stay behind before the oldest unread record expires.

What you do about it depends on which of the two you are in. Inside a job: give the slow operator more parallelism, or find the one key that is doing all the work, because a backpressured job is very often one hot key rather than a global capacity problem. Across a bus: add consumer capacity, and if you cannot, decide explicitly which consumer is allowed to fall behind. A pipeline where "whoever is slowest loses" was never decided by anyone is an outage waiting for a busy morning.

The answer that fails is "we autoscale". Scaling reacts to a load that has already arrived, and the interesting window is the minutes before it takes effect — which is exactly the window backpressure and buffering exist to survive.

The reasoning

Tumbling and sliding windows are cut by a clock — every five minutes, or every minute over the last five. You know in advance when each one closes, and therefore how long state has to be held. A **session** window is cut by behaviour: it opens on the first event for a key and closes when that key has been silent for longer than a stated gap, typically thirty minutes. Two sessions for two users, on the same day, can be forty seconds and four hours long.

The consequence that matters operationally is that **the state is not bounded by the clock**. A tumbling window releases its memory when it closes, on schedule, whatever the data does. A session releases its memory only when the key goes quiet — so a key that never goes quiet never releases it. A bot, a device stuck in a retry loop, a screen left on: one such key holds a growing session for days. Any production session pipeline needs a maximum session duration alongside the gap, forcing a close and starting a new one, and that cap is a small, deliberate loss of correctness bought to bound the memory.

The second consequence is about lateness. With a tumbling window, a late event either falls in a window you have closed or it does not. With sessions, a late event can **merge two sessions that had already been emitted as separate** — a gap of thirty-five minutes with an event later arriving in the middle of it was never two sessions at all. Frameworks that support session windows handle this by retracting and re-emitting, and a downstream sink that cannot express a retraction will end up holding both the two old sessions and the merged one.

And the gap itself is not an engineering number. Thirty minutes is a product decision about what counts as one visit, and changing it is a reprocessing job over however much history anybody cares about — which is the argument for keeping the raw events rather than only the sessions.

The reasoning

**Raw keeps the source as it arrived**, with only ingestion metadata added — when it landed, where it came from, which batch. The reason is not purity, it is recovery: if raw has been cleaned, then a bug in the cleaning logic is unrecoverable, because the evidence of what the source actually sent is gone. Keeping raw untouched is what makes every downstream transform replayable.

**The curated layer standardises**: date formats, trimmed strings, consistent casing, unit conversion, decoded enumerations. Doing it there rather than per-consumer means the rules live in one place and every consumer sees the same values — which is the same argument as the semantic layer, one level down.

**Nulls need classifying before they can be handled**, and this is where most pipelines are sloppy. A null in a mandatory field is a broken record and belongs in quarantine. A null in an optional field is information — "not provided" — and should be preserved as null rather than defaulted, because replacing it with zero or "Unknown" destroys the distinction between *absent* and *actually zero*. Defaulting is only safe when the business meaning is explicit and documented.

The overall shape: **raw is append-only and unopinionated; curated is where opinions live; gold is where business definitions live.** A rule anywhere upstream of curated is a rule you cannot change without re-fetching from the source.