Mastercard · mastercard.wd1.myworkdayjobs.com · checked today
Senior Data Engineer
Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible.
Skills, with evidence
- Data modelling
Strong SQL and data modeling skills.
must have - Failure handling
Implement solutions for data replay, recovery, checkpoint management, and failure handling.
must have - Python
Proficiency in Scala, pyspark or Python.
must have - SQL
Strong SQL and data modeling skills.
must have - Spark
Strong expertise in Apache Spark (Batch and Structured Streaming).
must have - Streaming
Create and implement validation suite for Spark batch and streaming applications to process high-volume datasets efficiently.
must have - Data quality
Experience building observability solutions using monitoring and logging platforms.
- Airflow / orchestration
Experience with containerization and orchestration technologies (Docker, Kubernetes).
- Cost & performance
Analyze system bottlenecks and optimize application performance, throughput, and resource utilization.
- Idempotency & backfills
Implement solutions for data replay, recovery, checkpoint management, and failure handling.
- AWS
Experience operating production-grade distributed systems in cloud or hybrid-cloud environments.
not practised here - Governance & security
Knowledge of data governance, metadata management, and data platform best practices.
not practised here - Kubernetes
Experience with containerization and orchestration technologies (Docker, Kubernetes).
not practised here - Scala
Proficiency in Scala, pyspark or Python.
not practised here
Your plan
- The SQL screen: correct, then fast≈ 2 h
SQL · Data quality
- Median delivery time per cityIntermediate
- Bucket deliveries into quartilesIntermediate
- Median order value without a median functionIntermediate
- New and repeat orders by monthIntermediate
- Every order against its customer's averageIntermediate
- Python: the data-wrangling round≈ 3 h
Python
- Diff two snapshots of a tableIntermediate
- Explode an array column into rowsIntermediate
- Flatten nested event payloadsIntermediate
- Pivot a long metrics table to wideIntermediate
- Choose what an incremental run should readIntermediate
- Data modelling: the round most people fail≈ 3 h
Data modelling
- Addresses that stay true to the pastIntermediate
- Seat holds and the release-night raceIntermediate
- Subscription warehouse grainIntermediate
- Campaign efficiencyIntermediate
- Catalogue: products, variants and sellersIntermediate
- Pipeline design: safe to run twice≈ 4 h
Failure handling · Streaming · Airflow / orchestration · Idempotency & backfills
- Five minutes behind the sourceIntermediate
- The source will not let youIntermediate
- The fact arrived firstIntermediate
- Parcel tracking pipelineIntermediate
- Is this change safe?Intermediate
- Spark: read the plan Spark actually ran≈ 2 h
Spark · Cost & performance
- broadcast() with auto-broadcast off, and the case where Spark ignores itIntermediate
- autoBroadcastJoinThreshold compares an estimate: flip a join with select() and one settingIntermediate
- Does Spark really run your EXISTS subquery once per row?Intermediate
- left_semi and left_anti: "customers who did / never did" without a full joinIntermediate
- A self-join on a real key that still multiplies rowsIntermediate
- Say it out loud≈ 1 h
25 drills · Intermediate + Advanced≈ 14 hours
Not covered by the plan: AWS, Governance & security, Kubernetes, Scala.
Readiness
Counted from drills you have completed anywhere on D8LooP.
leaves in 6 dremoved the moment Mastercard closes it