Mastercard · mastercard.wd1.myworkdayjobs.com · checked today
Senior AI Data Engineer
Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible.
Skills, with evidence
- Airflow / orchestration
Strong knowledge of workflow orchestration tools such as Apache Airflow, Azure Data Factory, or similar platforms
must have - Data modelling
Solid understanding of data modeling, metadata management, data lineage, and enterprise data governance
must have - Data quality
Ensure high standards of data quality through automated validation, profiling, lineage, observability, and monitoring
must have - Python
Strong proficiency in Python, SQL, Spark, and distributed data processing frameworks
must have - SQL
Strong proficiency in Python, SQL, Spark, and distributed data processing frameworks
must have - Spark
Strong proficiency in Python, SQL, Spark, and distributed data processing frameworks
must have - Streaming
Experience with streaming technologies such as Kafka, Kinesis, or Azure Event Hubs.
must have - AWS
Experience building cloud-native data platforms using AWS, Azure, or Google Cloud Platform
must have · not practised here - Azure
Experience building cloud-native data platforms using AWS, Azure, or Google Cloud Platform
must have · not practised here - GCP
Experience building cloud-native data platforms using AWS, Azure, or Google Cloud Platform
must have · not practised here - Failure handling
- Ensure high standards of data quality through automated validation, profiling, lineage, observability, and monitoring
- Governance & security
Working closely with AI Engineers, Data Scientists, Platform Engineers, and Solution Architects, you will deliver secure, reliable, and high-performance data solutions while driving engineering best practices, data governance, and operational excellence across AI initiatives
not practised here - Machine learning
As a Senior AI Data Engineer within Mastercard's AI Center of Excellence, you will lead the design, development, and optimization of enterprise data platforms and pipelines that enable scalable AI, machine learning, and Generative AI solutions.
not practised here - Cost & performance
- Build and optimize data architectures supporting LLMs, RAG, embeddings, vector databases, and AI knowledge repositories
Your plan
- The SQL screen: correct, then fast≈ 2 h
Data quality · SQL
- Median delivery time per cityIntermediate
- Bucket deliveries into quartilesIntermediate
- Median order value without a median functionIntermediate
- New and repeat orders by monthIntermediate
- Every order against its customer's averageIntermediate
- Python: the data-wrangling round≈ 3 h
Python
- Diff two snapshots of a tableIntermediate
- Explode an array column into rowsIntermediate
- Flatten nested event payloadsIntermediate
- Pivot a long metrics table to wideIntermediate
- Choose what an incremental run should readIntermediate
- Data modelling: the round most people fail≈ 3 h
Data modelling
- Addresses that stay true to the pastIntermediate
- Seat holds and the release-night raceIntermediate
- Subscription warehouse grainIntermediate
- Campaign efficiencyIntermediate
- Catalogue: products, variants and sellersIntermediate
- Pipeline design: safe to run twice≈ 4 h
Airflow / orchestration · Streaming · Failure handling
- The source will not let youIntermediate
- Parcel tracking pipelineIntermediate
- Is this change safe?Intermediate
- Marketplace transactions at scaleIntermediate
- Five minutes behind the sourceIntermediate
- Spark: read the plan Spark actually ran≈ 2 h
Spark · Cost & performance
- broadcast() with auto-broadcast off, and the case where Spark ignores itIntermediate
- autoBroadcastJoinThreshold compares an estimate: flip a join with select() and one settingIntermediate
- Does Spark really run your EXISTS subquery once per row?Intermediate
- left_semi and left_anti: "customers who did / never did" without a full joinIntermediate
- A self-join on a real key that still multiplies rowsIntermediate
- Say it out loud≈ 1 h
25 drills · Intermediate + Advanced≈ 14 hours
Not covered by the plan: AWS, Azure, GCP, Governance & security, Machine learning.
Readiness
Counted from drills you have completed anywhere on D8LooP.
leaves in 13 dremoved the moment Mastercard closes it