Capital One · capitalone.wd12.myworkdayjobs.com · checked today
Data Engineer 4 (Python, AWS, Kafka, Spark, SQL, Snowflake, Databricks, GenAI)
Data Engineer 4 (Python, AWS, Kafka, Spark, SQL, Snowflake, Databricks, GenAI) Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive and iterative delivery environment?
Skills, with evidence
- Data modelling
At least 1 year of experience in data modeling and designing end-to-end data solutions using both relational and non-relational database systems
must have - Python
At least 2 years of experience with one programming language (Python, Java, or Scala)
must have - SQL
At least 2 years of experience with SQL
must have - Spark
Utilize programming languages such as Python and Spark, along with open-source relational and NoSQL databases
must have - Streaming
4+ years of experience designing, implementing, and operating real-time or streaming data pipelines
must have - Warehousing
4+ years of experience designing and supporting data warehousing solutions (e.g., Snowflake, Redshift)
must have - AWS
4+ years of hands-on experience designing, deploying and operating data workloads in at least one public cloud environment (AWS, Microsoft Azure, or Google Cloud)
must have · not practised here - Airflow / orchestration
2+ years of experience working on data observability (e.g., Monte Carlo, Splunk) or data orchestration tools (e.g., Airflow, Dagster)
- Governance & security
Implement data security standards, including encryption at rest/transit and fine-grained access control, ensuring compliance with data privacy regulations
not practised here - Java
At least 2 years of experience with one programming language (Python, Java, or Scala)
not practised here - Scala
At least 2 years of experience with one programming language (Python, Java, or Scala)
not practised here - Data quality
2+ years of experience working on data observability (e.g., Monte Carlo, Splunk) or data orchestration tools (e.g., Airflow, Dagster)
- Azure
4+ years of hands-on experience designing, deploying and operating data workloads in at least one public cloud environment (AWS, Microsoft Azure, or Google Cloud)
not practised here - GCP
4+ years of hands-on experience designing, deploying and operating data workloads in at least one public cloud environment (AWS, Microsoft Azure, or Google Cloud)
not practised here
Your plan
- The SQL screen: correct, then fast≈ 1 h
SQL · Data quality
- Average order value by countryFoundations
- Line items per orderFoundations
- Revenue by buyer countryFoundations
- Customers who never orderedFoundations
- Hiring funnel by roleFoundations
- Python: the data-wrangling round≈ 2 h
Python
- Clean spreadsheet headers into column namesFoundations
- Events normalization jobFoundations
- List the distinct composite keysFoundations
- Deduplicate rows, first one winsFoundations
- Render a byte count for humansFoundations
- Data modelling: the round most people fail≈ 2 h
Data modelling · Warehousing
- Marketplace core entitiesFoundations
- Cinema seat bookingFoundations
- City parking baysFoundations
- Dating app matchesFoundations
- Food delivery ordersFoundations
- Pipeline design: safe to run twice≈ 3 h
Streaming · Airflow / orchestration
- After the first one finishesFoundations
- Run it for last TuesdayFoundations
- The history nobody keptFoundations
- The spreadsheet is a dependencyFoundations
- Too slow by morningFoundations
- Spark: read the plan Spark actually ran≈ 1 h
Spark
- HAVING vs WHERE: where does a filter after groupBy actually run?Foundations
- countDistinct vs approx_count_distinct: what the extra shuffle buysFoundations
- COUNT(*) vs COUNT(column): the null trapFoundations
- Grouping by two columns: what changes in the shuffle?Foundations
- Does the join type change the join strategy?Foundations
- Say it out loud≈ 1 h
25 drills · Foundations + Intermediate≈ 10 hours
Not covered by the plan: AWS, Governance & security, Java, Scala, Azure, GCP.
Readiness
Counted from drills you have completed anywhere on D8LooP.
leaves in 12 dremoved the moment Capital One closes it