Azure Theoretical

The reasoning

Enabling the namespace on a storage account is what makes it ADLS Gen2. The account then exposes the DFS endpoint, used through the `abfss://container@account.dfs.core.windows.net/path` URI, alongside the Blob endpoint. Both endpoints address the same data.

On flat Blob Storage, `folder/a.parquet` and `folder/b.parquet` are two independent blobs; "renaming the folder" means copying both and deleting the originals, with a window where readers see half a folder. With the namespace, the directory is an object, and the rename is a metadata change.

Access control gains a second model. Azure RBAC still works at the container level. ACLs add read, write and execute bits per directory and file for users, groups and service principals, including default ACLs inherited by new children. Lake layouts with per-domain folders rely on them.

Costs: the namespace is free, but the account's transaction pricing differs slightly, and some Blob features such as soft delete for blobs arrived later for namespace-enabled accounts. Once enabled it cannot be turned off, though a flat account can be upgraded in place.

The reasoning

The tiers trade two prices against each other. Storage per gigabyte falls from Hot to Cool to Cold to Archive; read and write transactions and data retrieval rise. A partition read daily belongs in Hot; a partition read once a quarter belongs in Cool or Cold; a compliance copy nobody reads belongs in Archive.

Archive is different in kind: blobs are offline. Reading one means a rehydration to Hot or Cool, with a charge for the rehydrate. It takes up to 15 hours at standard priority, or under an hour at high priority for objects under 10 GB. A query engine cannot read Archive directly.

Lifecycle policies are rules on the account: move blobs under a prefix to Cool 30 days after modification or last access, to Archive after 365, delete after 2,555 days. Last-access tracking must be enabled for access-based rules. Versions and snapshots can be tiered separately.

The minimum durations are enforced as early deletion fees, prorated for the remaining days. The account's default tier applies to new blobs without an explicit tier. Tiers are per blob; a Delta table's files can span tiers, with the risk that a reader hits an Archive file and fails.

The reasoning

Microsoft quotes durability of at least eleven nines for LRS, twelve for ZRS and sixteen for GRS and GZRS over a year. The numbers describe object loss, not availability: LRS data is intact but unreachable while its data centre is down.

ZRS is the usual choice for a production lake: it survives a zone loss with no failover and no secondary to manage, at a modest premium over LRS. GRS adds a copy a few hundred miles away, replicated asynchronously, so a regional disaster can lose the last minutes of writes, the recovery point objective.

Failover to the secondary is customer-initiated for most accounts and converts the account to LRS in the new primary until you re-enable geo-redundancy. Read access with RA variants lets analytics continue reading in the secondary during a primary outage, but writes still require failover.

Not every feature supports every option: ADLS Gen2 with the hierarchical namespace supports LRS, ZRS, GRS and GZRS, with RA variants, in most regions. Premium block blob accounts offer only LRS and ZRS. Redundancy can be changed on a live account, which triggers a background copy.

The reasoning

Copy connects over a hundred connectors to a dozen sink types. It can read a query, write Parquet, map columns, and it parallelises with DIUs and parallel copies. It is the right tool for landing raw data: cheap, fast, no cluster.

A data flow is authored on a canvas and debugged against a sample. At run time ADF spins up a Spark cluster of the configured compute size on the Azure integration runtime, runs the flow, and tears it down. Start-up is around three to five minutes per run unless the runtime's TTL is set, which keeps the cluster for a period after each run and is billed while warm.

Billing is the trap: a data flow that runs every ten minutes on a general-purpose eight-core cluster pays for the cluster time plus start-up each time. A TTL shrinks start-up but bills idle time. Teams with many small flows often consolidate them or move to Databricks jobs they can tune.

Choosing: Copy for extraction and loading. Data flows for transformations the team wants low-code and inside the factory, such as a merge into a dimension. Databricks or Synapse Spark notebooks, called with the Databricks activity, when the logic is large, needs libraries, or must be unit tested as code.

The reasoning

Lift and shift is the use case. A company with hundreds of SSIS packages provisions an Azure-SSIS runtime with a node size and count and deploys the packages to SSISDB in Azure SQL. It then runs them from a Data Factory pipeline with the Execute SSIS Package activity. The packages are unchanged.

It is billed per node per hour while running, so it is started before a batch and stopped after, typically with Web activities or a schedule. Custom setups install third-party components on the nodes. Joining it to a virtual network lets packages reach on-premises data over a gateway.

The Azure runtime is the default: no infrastructure, auto-resolving region, billed per DIU-hour for copies and per vCore-hour for data flows. A managed virtual network version keeps traffic private with managed private endpoints.

The self-hosted runtime is an agent on a Windows machine, often a cluster of them for high availability, registered to the factory with a key. It reads from SQL Server on premises or behind a firewall and streams to the cloud sink. It cannot run data flows, only copies and lookups.

The reasoning

You pick the storage account and container, set `blobPathBeginsWith` to a folder such as `landing/orders/` and `blobPathEndsWith` to `.csv`, and choose the created event. ADF creates an Event Grid subscription behind the scenes; the identity creating the trigger needs permission to do so.

The pipeline declares parameters, and the trigger maps `@triggerBody().fileName` and `@triggerBody().folderPath` into them. Inside, a Copy activity uses those parameters in a dataset's path, so the pipeline processes exactly the file that landed.

Each blob is an event, so a Spark job writing 200 part files fires 200 runs. The usual mitigations are to trigger on a marker file such as `_SUCCESS` or to filter by extension. The alternative is a tumbling window trigger on a schedule that processes whatever has arrived.

The other trigger types are schedule, which fires on a clock with no state, and custom event, which listens to any Event Grid topic. Tumbling window has fixed non-overlapping intervals, retries and dependencies on other windows, and supports backfill by running past windows.

The reasoning

A parameter such as `sourceFolder` is declared on the pipeline, referenced as `@pipeline().parameters.sourceFolder`, and supplied by the trigger definition, a parent's Execute Pipeline activity, or the manual run dialogue. The same pipeline serves many sources by parameter alone.

A variable such as `fileCount` is declared with a default, read as `@variables('fileCount')`, and updated with Set Variable, often from an activity's output: `@activity('Get Metadata').output.childItems`. Append Variable adds to an array across a ForEach loop, with the caveat that parallel iterations appending to one array are not safe.

Scope is the key difference. Parameters are the pipeline's contract; variables are its working memory. A child pipeline is a separate run with its own variables, so passing state down means a parameter. Passing results up means the Set Variable activity's pipeline return value in the child, read from the Execute Pipeline activity's output in the parent.

Global parameters, set at the factory level, are shared read-only values such as an environment name, overridable per environment in CI/CD. System variables like `@pipeline().RunId` and `@trigger().startTime` are built in.

The reasoning

Web is for quick calls: start a Databricks job through its REST API, fetch a token, call a Function that validates a file, post to Teams. The response body is available as `@activity('Web1').output` for the next activity. Authentication can use a managed identity, a service principal or basic credentials.

Webhook is for the long-running case. ADF sends `callBackUri` in the JSON body; the service must store it, do its work, and `POST` to that URI. The callback may carry a body that becomes the activity's output and a status code that marks success or failure. If no callback arrives within the timeout, the activity fails.

The failure mode is a service that ignores the callback: the Webhook activity sits until timeout, then fails even though the work succeeded. Logic Apps have a built-in response action for this; custom Functions must implement the callback explicitly.

Both can be the pattern for starting and awaiting external work; the alternative for Databricks specifically is the native Databricks activity, which waits for the job itself. Azure Function activity is the dedicated form of Web for Functions, with the function key handled for you.

The reasoning

A policy entry such as `"autotermination_minutes": {"type": "range", "maxValue": 60}` or `"node_type_id": {"type": "allowlist", "values": [...]}` constrains the create-cluster form. Users are granted the right to use a policy, and the UI shows only what the policy allows. Default policies ship for personal compute, shared compute and job compute.

Policies also enforce governance: a required `cost_center` tag, Unity Catalog access mode, a specific Databricks runtime range, or Photon on. Without them, an all-purpose cluster of 50 large nodes with no auto-termination is one click away.

A pool holds a configured instance type with a minimum idle count and an idle timeout. A job cluster attached to the pool takes its nodes from the pool, runs, and returns them. Preloaded Databricks runtime versions on pool instances save further start time. Driver and worker pools can be separate.

Economics: a job that runs every ten minutes on a fresh cluster spends five of them starting. A pool with two idle nodes removes that at the cost of two idle VMs. Serverless compute for jobs is the newer alternative that removes both the cluster and the pool from the user's concern.

The reasoning

The read is `spark.readStream.format('cloudFiles').option('cloudFiles.format', 'json').option('cloudFiles.schemaLocation', path).load('abfss://.../landing/')`. The write is a normal streaming sink to a Delta table with a checkpoint. New files appear in the next micro-batch; files already recorded in the checkpoint are never re-read.

Directory listing scans the path each batch, and Databricks optimises it with incremental listing for lexically ordered file names. File notification mode sets up an Event Grid subscription and a storage queue automatically. It scales to millions of files per directory and lowers latency, at the cost of extra Azure resources and permissions.

Schema inference samples the first files and writes the schema to the schema location. By default, a new column triggers a stream failure that the next start resolves by evolving the schema, `addNewColumns`; other modes rescue, fail or ignore. The `_rescued_data` column holds values that did not match the schema, so nothing is silently dropped.

`trigger(availableNow=True)` is the batch pattern: a scheduled job starts the stream, it consumes all unprocessed files in micro-batches, and exits. It gives incremental loading with exactly-once bookkeeping and no always-on cluster. Lakeflow Declarative Pipelines use Auto Loader under their streaming tables.

The reasoning

In SQL: `CREATE OR REFRESH STREAMING TABLE bronze_orders AS SELECT * FROM STREAM read_files('abfss://.../landing', format => 'json')` then `CREATE OR REFRESH MATERIALIZED VIEW silver_orders AS SELECT ... FROM bronze_orders`. In Python, `@dp.table` decorated functions return DataFrames. No orchestration code names the order; the framework reads the references.

A streaming table processes each input row once and is the ingestion and append layer. A materialized view is the transformation layer; the framework chooses full recompute or incremental refresh based on the query. `AUTO CDC` or `APPLY CHANGES INTO` applies change data with SCD Type 1 or 2 semantics to a target.

Expectations: `CONSTRAINT valid_amount EXPECT (amount > 0) ON VIOLATION DROP ROW`. Metrics on passed and failed rows land in the event log, which is a queryable Delta table, so quality is observable without a separate tool.

The pipeline runs triggered, once per schedule, or continuous, for low latency, on serverless or classic compute it manages, with automatic retries and schema evolution handling. The cost is the Lakeflow pricing tier per DBU and less control over the exact Spark plan; the gain is far less code and built-in lineage and quality.

The reasoning

Tasks run in dependency order with conditional paths, `if else` conditions, and a for-each task that fans out over a list with a concurrency limit. Task values pass small results between tasks. A failed run can be repaired to rerun only the failed tasks, and a job can be run as a service principal rather than a user.

Triggers: cron schedules with a time zone, and file arrival on a Unity Catalog volume or external location, polling for new files. Continuous restarts the job whenever it ends, for streaming. The API or a Databricks Asset Bundles deploy serves CI/CD. Queueing holds a run when the concurrency limit is reached.

Compute per task: a job cluster defined in the job, shared across tasks or per task, serverless compute for jobs, or an existing all-purpose cluster. Job clusters start fresh each run, so a pool or serverless is how start latency is managed. All-purpose clusters cost roughly twice the DBU rate and keep state between runs, which makes them unreliable for production.

Observability: run history, task-level logs, Spark UI per run, and duration and failure alerts to email, Slack or webhooks. System tables in Unity Catalog record job runs and billing for cost analysis. Asset Bundles define jobs as YAML in git so the same job deploys to dev and prod.

The reasoning

The pool is metadata: small, medium, large and larger node sizes, a minimum and maximum node count for autoscale, dynamic executor allocation, and a session-level timeout. Each user or job gets a session on the pool; sessions share the pool's capacity limits. Billing starts when a session starts and ends at auto-pause, so the pause setting is the cost knob.

Integration is its strength inside Synapse. Notebooks read the lake through linked services and the workspace's primary ADLS account, and Spark tables in the lake database are visible to the serverless SQL pool. Synapse pipelines, Data Factory inside the workspace, run notebooks as activities.

Databricks runs a newer Spark with Photon, Delta features first, Unity Catalog for governance, Lakeflow for pipelines and jobs, and serverless compute. Most Azure data teams choose Databricks for Spark workloads and use Synapse for its SQL pools, and Microsoft's own direction has moved Spark investment to Fabric Data Engineering.

Fabric's Spark, in Fabric Data Engineering, is the successor pattern: workspace-level Spark settings, starter pools that begin in seconds, and the same notebook experience over OneLake. A Synapse Spark pool question in 2026 is often really a migration question.

The reasoning

The Cosmos DB form is the most used. Enabling the analytical store on a container creates a fully managed column store that auto-syncs within a couple of minutes, with its own time to live. Queries from Synapse read it directly; the transactional request units are untouched. Schema is inferred, with well-defined and full-fidelity modes for nested or varying documents.

The SQL form runs a link connection in the Synapse workspace: pick the source database, select tables, choose the target dedicated SQL pool, and start. An initial snapshot loads the tables, then changes stream continuously. The source needs change feed enabled and a managed identity with access.

Compared with Data Factory CDC or Debezium into Event Hubs, Synapse Link needs no code and no infrastructure. The price is fixed targets: Cosmos data to Synapse engines, SQL data to a dedicated pool. Teams on Databricks or Fabric use mirroring in Fabric or CDC pipelines instead.

Fabric mirroring is the successor idea: Azure SQL, Cosmos DB, Snowflake and others replicate continuously into OneLake as Delta with no pipeline. The Synapse Link name persists for Synapse workspaces and Cosmos DB.

The reasoning

A lakehouse has a Files section for raw data in any format and a Tables section for managed Delta tables. Notebooks, Spark job definitions, Data Factory pipelines and dataflows write it. Its SQL analytics endpoint exposes the Delta tables as a read-only database for T-SQL queries and views, with a sync delay after Spark writes.

A warehouse is a database created and governed by the SQL engine: `CREATE TABLE`, `INSERT`, `UPDATE`, `MERGE` and cross-table transactions. It adds T-SQL security features such as row-level security and dynamic data masking. The data is still Delta in OneLake, so Spark and other engines read it through OneLake shortcuts.

Direct Lake lets a Power BI semantic model read the Delta files straight from OneLake with no import and no DirectQuery to an engine, for either item. V-Order, Fabric's Parquet write optimisation, matters for Direct Lake performance and is on by default for warehouse writes and configurable for Spark.

A common layout is a lakehouse for bronze and silver, written by Spark, and a warehouse for gold where analysts own T-SQL logic. Shortcuts let a warehouse query lakehouse tables without copying. Both are items in a workspace billed against the same capacity.

The reasoning

Every operation consumes CU-seconds: a Spark job, a SQL query, a pipeline run, a Power BI refresh or report render. The capacity metrics app shows consumption by item and operation. One capacity can serve many workspaces; a workspace belongs to one capacity, so teams isolate noisy workloads by assigning them their own capacity.

Smoothing is what makes a small SKU viable. A ten-minute Spark job that would exceed the capacity's instantaneous CUs is spread over a 24-hour window for background operations and shorter windows for interactive ones. If the smoothed usage stays above 100 percent, throttling policies kick in: interactive delays after ten minutes of overage, rejections after sixty, and background rejection beyond.

Pay-as-you-go F SKUs can be paused and resumed, and scaled up or down, which is how dev capacities are kept cheap. Reservations for a year discount about 40 percent. OneLake storage is billed separately per gigabyte, and BCDR storage and mirroring have their own meters.

Licensing: Fabric items need a capacity; Power BI items can live on Pro or Premium per user licences or on a capacity. At F64 and above, report viewers need no Pro licence, which is why F64 is the breakpoint organisations compare against the retired P1 SKU.

The reasoning

An event hub is the stream, partitioned into 1 to 32 partitions on Standard, more on Premium and Dedicated. Events with the same partition key land in the same partition in order. Retention is 1 to 7 days on Standard and up to 90 on Premium. A namespace holds many hubs that share its throughput units.

Throughput arithmetic: 50 MB per second of ingress needs 50 throughput units, above the Standard limit of 40, which points at Premium or Dedicated. Egress is twice ingress per unit because several consumer groups usually read each event. Exceeding a unit returns a server-busy error, which is why auto-inflate exists.

Consumer groups separate readers: a Databricks stream, a Stream Analytics job and an archiving function each use their own group and keep their own checkpoints in their own storage. One group should have at most one active reader per partition at a time, the same rule as Kafka's consumer groups.

The Kafka endpoint speaks the Kafka protocol on port 9093; a hub is a topic, a consumer group is a consumer group, and the namespace is the bootstrap server. It lets Debezium, Kafka Connect and Spark's Kafka source work unchanged, with SASL authentication using the connection string or Entra ID.

The reasoning

Enabling Capture on a hub asks for a destination container, the time and size windows, and whether to write empty files for windows with no events. The path format is a template such as `{Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}`. Files land in Avro with the event body as bytes plus metadata columns.

The Avro output means a downstream job must read Avro and decode the body, often JSON bytes, before it is useful. Spark reads it directly with the Avro source; Synapse serverless SQL and Databricks both handle it. Many teams run a nightly job that converts the day's Capture files to Delta.

Capture is billed per throughput unit per hour on Standard and is included in Premium. It uses no consumer group and does not count against egress. It is the cheapest replay source: a bug in a consumer is fixed by reprocessing the Capture files rather than depending on the hub's 7-day retention.

Compared with a Databricks stream writing bronze directly, Capture has no transformation, no Delta, and up to 15-minute latency. The two are often used together, Capture as the cold archive and the stream as the live path.

The reasoning

`SELECT deviceId, COUNT(*) FROM input TIMESTAMP BY eventTime GROUP BY deviceId, TumblingWindow(minute, 5)` counts by when the sensor measured, not when the message landed. A device offline for an hour that flushes its buffer contributes to the hour it measured, which is what analytics usually wants.

Arrival time is monotonic and simple, so without TIMESTAMP BY there is no lateness problem and also no correctness for delayed events: the buffered hour lands in one five-minute window. Event time is correct but requires the job to decide how long to wait for stragglers.

Late arrival tolerance ranges from 0 seconds to 20 days. An event whose event time is more than that behind its arrival time is either adjusted to the tolerance boundary or dropped. Out-of-order tolerance, from 0 seconds to an hour, buffers events for that long so they can be sorted by event time before windows close, adding latency.

`TIMESTAMP BY ... OVER partitionKey` makes each key its own substream with its own clock, so a slow device does not hold back the others. The System.Timestamp() function returns the effective time after these policies, which is what you output to understand what the job did.

The reasoning

Registering a source and scheduling scans is the first step; a scan uses a managed identity or credential from Key Vault, walks the source, and writes assets with technical metadata. Classifications are applied by built-in and custom rules, and sensitivity labels from Microsoft Information Protection can be applied to columns.

The catalog answers "what data do we have, who owns it, what does this column mean". Glossary terms link business vocabulary to assets. Collections group assets for access control inside Purview, so a domain team sees its own sources.

Lineage arrives from integrated services automatically. A Data Factory Copy or data flow records source and sink assets, Synapse and Fabric pipelines do the same, and the Databricks connector pulls Unity Catalog's table and column lineage. Custom lineage can be pushed through the Apache Atlas API, which Purview implements.

Unity Catalog is the metastore and access layer for Databricks: catalogs, schemas, tables, grants, row filters and column masks, with lineage for notebooks and jobs. Purview sits above it for the enterprise view and policy. The 2024 and 2025 product changes merged Purview governance with the compliance portal, so the name now covers both.

The reasoning

A pipeline is a logical group of activities. Activities are the steps: Copy, data flow, Lookup, Get Metadata, ForEach, If, Execute Pipeline, Web, Databricks notebook, stored procedure and more. A dataset describes data in a store, such as a folder of Parquet or a table. A linked service is the connection string and credentials to a store or compute.

Integration runtimes provide the compute: Azure for cloud, self-hosted for private networks, Azure-SSIS for packages. Triggers start pipelines on a schedule, a tumbling window, a storage event or a custom event. Parameters make pipelines reusable, and expressions in a JSON-based language glue values between activities.

Development happens in a browser studio over a git repository, with publish producing ARM templates for deployment; CI/CD promotes the factory across environments with parameterised linked services. Monitoring shows runs, activity outputs and durations, with diagnostic settings to Azure Monitor.

Its role in most Azure stacks is extraction and orchestration: copy raw data into ADLS, call Databricks for transformation, load a warehouse. The same engine appears inside Synapse as pipelines and inside Fabric as Data Factory, with dataflows added for low-code transformation.

The reasoning

Blob Storage stores objects in containers with a flat key space. ADLS Gen2 adds a namespace layer over the same storage, so paths like `raw/orders/2026/11/01/` are directories you can list, rename, delete and secure as units. There is no separate service or price; it is an account option.

Access is through Entra ID with RBAC roles such as Storage Blob Data Contributor at the container or account level, plus ACLs at directory and file level. Shared access signatures and account keys remain where legacy tools need them. Databricks and Synapse authenticate with managed identities or service principals.

Performance features for analytics: hierarchical listing, parallel reads, premium tier on SSD for latency-sensitive workloads, and integration with Spark's committers through the ABFS driver. Redundancy, tiers, lifecycle management, soft delete and versioning all apply.

Layouts follow the medallion convention: containers or top-level folders for bronze, silver and gold, Delta tables under them, and ACLs per domain. OneLake in Fabric can mount an ADLS Gen2 path as a shortcut, so the lake remains the system of record while Fabric engines read it.

The reasoning

A workspace is deployed into your subscription with a managed resource group holding its storage and network. Compute is all-purpose clusters for interactive work, job clusters for scheduled work, SQL warehouses for BI, and serverless options for all three. The control plane is Databricks-managed; data stays in your ADLS accounts.

Unity Catalog provides a metastore per region with catalogs, schemas and tables, external locations over ADLS secured by the access connector's managed identity, fine-grained grants, lineage and audit. Workspaces attach to the metastore so the same tables are governed across them.

Tiers on Azure are Standard and Premium; most governance features, including Unity Catalog and role-based access, need Premium. Pricing is DBUs per hour by compute type plus the Azure VM cost, with pre-purchase plans for discounts.

Typical Azure architecture: Data Factory or Lakeflow Jobs orchestrates, Auto Loader ingests from ADLS, and Delta tables sit in medallion layers. Databricks SQL or Power BI sit on top, with Unity Catalog governing and Purview cataloging. Fabric mirroring of Databricks catalogs connects the two platforms.

The reasoning

Workloads are items in a workspace: lakehouse, warehouse, pipeline, dataflow, notebook, eventstream, KQL database, semantic model, report. Each uses the workspace's capacity, and all persist to OneLake. There is no storage account, cluster or network to configure; Microsoft runs it.

OneLake is provisioned automatically per tenant and organised by workspace and item. Its open format, Delta Parquet, means Databricks, Synapse or any Parquet reader can access it through its ADLS-compatible API, and OneLake security applies table and column level rules across engines. Mirroring replicates external databases into it continuously.

Direct Lake mode in Power BI reads OneLake Delta files directly into the semantic model engine, avoiding both import refreshes and DirectQuery round trips. V-Order write optimisation is how Fabric engines lay out Parquet for that path.

Positioning: Fabric is the successor direction for Synapse Analytics and for Power BI Premium, aimed at organisations wanting a managed end-to-end platform. Azure Databricks remains the choice for code-heavy engineering, and the two interoperate through OneLake shortcuts and mirroring.

The reasoning

Core concepts mirror Kafka: namespace as the cluster, hub as the topic, partitions for ordering and parallelism, consumer groups for independent readers, offsets stored by the consumer, retention of days. Throughput is bought in throughput units or processing units, and Capture archives to storage.

Differences from self-run Kafka: no brokers, ZooKeeper or KRaft to operate, and limits per tier on partitions, retention and consumer groups. Schema Registry is built in. The protocol surface covers producers and consumers but not every Kafka admin feature. Kafka Connect and Debezium work against the Kafka endpoint.

Service Bus is about individual messages: a queue delivers each message to one receiver with a lock, supports sessions for ordering, scheduled and deferred messages, duplicate detection and dead-letter queues. Use it for orders, commands and workflows; use Event Hubs for telemetry, logs and change streams at scale.

Consumers on Azure: Databricks Structured Streaming through the Kafka source or the Event Hubs connector, Stream Analytics, Functions with the Event Hubs trigger, and Fabric eventstreams. Checkpointing lives with the consumer, which is why each consumer group has its own storage account container or Delta checkpoint.

The reasoning

A vault holds named secrets with versions. Applications authenticate to it with a managed identity or service principal and need a role such as Key Vault Secrets User. Firewall rules and private endpoints restrict network access. Diagnostic logs record each get, which is the audit trail.

In Data Factory, you create a Key Vault linked service authenticated by the factory's managed identity. Then in another linked service you pick the password field, choose Azure Key Vault, and name the secret. At run time the factory fetches the current version. Rotating the secret needs no factory change.

In Databricks, `databricks secrets create-scope --scope kv --scope-backend-type AZURE_KEYVAULT --resource-id ... --dns-name ...` creates a scope mirroring the vault. Code reads `dbutils.secrets.get('kv', 'storage-key')`, and any attempt to print the value shows `[REDACTED]`. Unity Catalog's external locations reduce the need for secrets at all.

Keys in Key Vault also serve customer-managed encryption for storage accounts and Databricks workspaces. Soft delete keeps deleted secrets recoverable for a retention period, and purge protection prevents permanent deletion during it, which guards against a deleted encryption key making data unreadable.

The reasoning

Access to Azure resources is granted by Azure RBAC: a role, such as Storage Blob Data Reader, assigned to a principal at a scope. The principal is a user, group, service principal or managed identity; the scope is a subscription, resource group or resource. Entra ID answers who you are; RBAC answers what you may do.

A system-assigned managed identity is tied to one resource and deleted with it. A user-assigned managed identity is a standalone resource that several resources can share, which suits a fleet of Functions or several factories needing the same access. Code obtains tokens from the local metadata endpoint with no secret.

Service principals remain for things outside Azure, such as a CI runner or a Databricks job running as a non-human, and for Databricks service principals in Unity Catalog. Their secrets expire and must be rotated and stored, which is why managed identities are preferred wherever the caller runs on Azure.

Data platform touchpoints: Data Factory's managed identity reading ADLS and Key Vault, Databricks' access connector identity for Unity Catalog storage, and the Synapse workspace identity. Entra groups map to Unity Catalog groups and Fabric workspace roles, and conditional access policies decide who can sign in.

The reasoning

A storage account also provides files, queues and tables, but blobs are the data engineering surface. Account kinds and performance tiers, Standard general-purpose v2 on HDD and Premium block blobs on SSD, set the price and latency class. Redundancy and default access tier are account settings.

Shared access signatures delegate access without sharing keys. A SAS token grants listed permissions on a container or blob until an expiry, and can be bound to a stored access policy for revocation. User delegation SAS tokens are signed with Entra credentials and are the recommended form. Account keys grant everything and should be disabled where possible.

Features that matter for pipelines: soft delete and versioning for recovery, immutability policies for compliance, and lifecycle management for tiering. Change feed and Event Grid events serve triggering, and object replication copies across accounts. Static website hosting and CDN integration exist for serving content.

Blob Storage with the hierarchical namespace is ADLS Gen2. Without it, it is the right choice for backups, archives, media and application files, and as a landing zone where directory semantics are unimportant.

The reasoning

Metrics are numeric time series with near-real-time alerting: failed pipeline runs, integration runtime CPU, Event Hubs incoming messages, storage transactions. Logs are records with properties, queried with KQL in Log Analytics: `ADFActivityRun | where Status == 'Failed' | summarize count() by PipelineName`.

Diagnostic settings on a resource choose which log categories and metrics to send and where. The destinations are a Log Analytics workspace, a storage account for archive, or an Event Hub for streaming to a SIEM. Without a diagnostic setting, Data Factory keeps only 45 days of run history in its own monitor.

Alert rules combine a signal, metric or log query, a condition, and an action group. A log alert on failed activity runs in the last 15 minutes, or a metric alert on Event Hubs throttled requests, posts to Teams or PagerDuty. Workbooks build dashboards over the same queries.

Databricks integration: workspace diagnostic settings send audit logs for clusters, jobs, notebooks and Unity Catalog to a workspace. Spark metrics go through the Log Analytics agent or the Databricks Monitoring library. Lakeflow Jobs and pipelines also expose their own notifications and system tables, which many teams query directly from Databricks SQL.

The reasoning

Each private endpoint targets one sub-resource: for a storage account, blob and dfs are separate endpoints. That trips people up when Spark uses the DFS endpoint and only the Blob one was created. A private DNS zone such as `privatelink.dfs.core.windows.net` maps the account's hostname to the private IP for resolvers in the network.

Service firewalls then deny public traffic: the storage account's networking set to disabled public access, Key Vault's firewall allowing only private endpoints and trusted services. Anything outside the network, including a developer laptop, must come through a VPN, ExpressRoute or a bastion.

Data Factory's Azure integration runtime in a managed virtual network provisions managed private endpoints that you approve on the target resource, so the serverless copy reaches a locked storage account. Databricks offers front-end Private Link for user access and back-end for cluster-to-control-plane traffic, plus VNet injection so clusters sit in your network.

Service endpoints are the older alternative: they keep traffic on the backbone and let the service firewall allow a subnet. But the service keeps its public IP and is not reachable from on premises. Private endpoints replaced them for locked-down platforms, at a small cost per endpoint hour and per gigabyte processed.