Python defined flat: generators and yield, memoisation with lru_cache, context managers, modules and packages, virtual environments, type hints, duck typing, exception handling, inheritance against composition, the main guard, logging, pytest and mocking, sorting, pickling, text and binary files, locks, repr, properties, enums and __init__.py.
The constructs a pipeline is written from: generators, memoisation, context managers, properties, enums and the two dunder methods that print an object.
Modules, packages & environments
4
How code is found and isolated: imports, packages, the main guard, virtual environments and pinning.
Types, errors & tests
6
What type hints do, how exceptions flow, duck typing, inheritance against composition, and how pytest and mocking fit a pipeline.
Data in and out
4
Sorting, serialization, encodings, and the race condition a lock prevents.
Evergreen · asked verbatim
10
The flat form, in the words interviewers use. The one-line definitions that open a Python screen.
01 / 30
Comprehensions & generators
What is a generator in Python? How is `yield` different from `return`, and why do generators matter for data pipelines?
Why they ask this
Streaming through a file larger than memory is the first Python skill a data engineer needs, and the generator is the construct that makes it possible.
Say this
A generator is a function that contains `yield`; calling it returns an iterator instead of running the body. Each `next()` runs until the next `yield`, hands out that value, and pauses with its local state kept. `return` ends the function and discards its state. Generators let a pipeline process one row at a time, so memory stays at one row no matter how large the file is.
The reasoning
`def rows(path): with open(path) as f: for line in f: yield parse(line)` does nothing when called. It returns a generator object. The body starts on the first `next()`, runs to `yield`, and stops there with `f` still open and the loop position remembered.
`return` in a normal function computes the whole result and hands it back at once. A function that builds a list of 50 million parsed rows needs memory for all of them. The generator version holds one.
Generators compose. `filtered = (r for r in rows(path) if r.amount > 0)` is another generator, and `sum(r.amount for r in filtered)` pulls rows through the chain one at a time. Each stage is a function you can test alone.
The cost is that a generator is consumed once. A second loop over the same generator gets nothing, because its state has reached the end. Call `list()` when you need two passes, a length or random access, and only when the data fits.
The answer most people give
"A generator is a function that returns a list lazily." It returns an iterator, not a list, and it never holds the whole sequence. That is the whole point.
They’ll ask next
A function returns a generator and the caller loops it twice. What does the second loop see, and whose bug is it?
What does `functools.lru_cache` do? When is memoisation unsafe?
Why they ask this
It is a one-line speedup that produces wrong answers when applied without thought, which makes it a good judgment question.
Say this
`@lru_cache` wraps a function so results are stored by argument and returned from the cache on repeat calls; the `functools` docs give `maxsize` a default of 128 and `None` for unbounded. It is unsafe when the function depends on anything other than its arguments, such as time, a database or a file, when the result is mutable and callers change it, or when arguments are unhashable.
The reasoning
The decorator keeps a dictionary from argument tuple to result. A second call with the same arguments skips the body. For a pure function that is expensive, such as parsing a schema or computing a lookup, that turns repeated cost into one cost.
Correctness depends on purity. A cached `get_rate(currency)` that reads a table returns the rate from the first call forever. A cached function returning a list hands the same list to every caller, and one caller's `append` leaks into the others. The docs describe the cache as keyed on the arguments, so a `dict` argument raises `TypeError: unhashable type`.
`maxsize` bounds memory: with 128, the least recently used entries are evicted. `maxsize=None` grows without limit, which on a function keyed by a high-cardinality value such as a user id is a memory leak. `cache_info()` reports hits and misses, and `cache_clear()` empties it.
For values that change, use a cache with a time-to-live from a library, or key the function on a version. For per-instance caching, `functools.cached_property` computes once per object.
The answer most people give
"Cache the expensive function and the pipeline is faster." If the function reads anything external, the pipeline is now faster and wrong. Memoise pure functions only.
They’ll ask next
A cached lookup keyed on `customer_id` runs for 10 million customers with `maxsize=None`. What happens to memory?
What is a context manager? How do you write one with `contextlib.contextmanager`, and what does `with` guarantee?
Why they ask this
Every pipeline holds a file, a connection or a lock, and the context manager is how the release path is made unconditional.
Say this
A context manager is an object with `__enter__` and `__exit__`, used by `with`: `__enter__` runs on entry and its result is bound to `as`; `__exit__` runs on every exit, including an exception. With `@contextlib.contextmanager`, a generator function does the same: code before `yield` is the setup, the yielded value is bound, and code after `yield`, in a `finally`, is the teardown.
The reasoning
`with open(path) as f:` guarantees `f.close()` runs whether the block ends normally, raises, returns or breaks. The guarantee is `__exit__` being called by the interpreter on every path out of the block.
The generator form keeps setup and teardown together. `@contextmanager def connection(): conn = connect(); try: yield conn; finally: conn.close()`. The `try/finally` matters: without it, an exception inside the block skips the code after `yield` and the connection leaks.
`__exit__` receives the exception type, value and traceback. Returning `True` suppresses the exception, which is almost never what a pipeline wants; the default of returning `None` lets it propagate after cleanup.
The standard library ships the common ones: `open`, locks, `tempfile.TemporaryDirectory`, `contextlib.suppress`, and `ExitStack` for a dynamic number of resources. A database transaction wrapped as a context manager that commits on success and rolls back on exception is the idiom pipelines use most.
The answer most people give
"A context manager closes the resource at the end of the block." Only on the normal path, unless the teardown is in `finally`. The guarantee comes from `__exit__`, and a generator-based manager without `try/finally` does not have it.
They’ll ask next
What does `__exit__` returning `True` do, and when would a pipeline want it?
What is the difference between `__str__` and `__repr__`? Which one do logs and debuggers use?
Why they ask this
A small question that reveals whether the candidate has made their own classes debuggable.
Say this
`__repr__` returns the developer-facing representation, ideally one that recreates the object, such as `Order(id=42, amount=10.5)`; it is what the debugger, the REPL and containers show. `__str__` returns the user-facing text, used by `print()` and `str()`. If `__str__` is missing, Python falls back to `__repr__`, so define `__repr__` first.
The reasoning
The language reference describes `__repr__` as the official string representation, which should look like a valid expression when possible, and `__str__` as the informal, readable one. The REPL shows `repr`; `print` shows `str`.
Inside a list or dict, elements are shown with `repr`, so a class with only `__str__` still prints as `<Order object at 0x...>` inside a collection. That is why `__repr__` is the one to write first.
`@dataclass` generates a `__repr__` listing the fields, which is one of its quiet benefits. Logging `%r` or `!r` in an f-string uses `repr`, which makes log lines unambiguous: a string with trailing spaces shows its quotes.
Keep `__repr__` cheap and side-effect free. A repr that queries a database to describe itself runs every time a debugger shows the object.
The answer most people give
"They are the same thing; `__str__` is the newer name." They serve different readers. `__repr__` is for developers and is the fallback; `__str__` is for output.
They’ll ask next
A list of your objects prints as `[<Row object at 0x7f...>, ...]` although you wrote `__str__`. Why?
What is a `property` in Python? When do you use one instead of a plain attribute or a method?
Why they ask this
It is how Python avoids getters and setters, and a good test of whether the candidate knows the data model rather than Java habits.
Say this
A property is a method accessed like an attribute: `@property def total(self): return self.qty * self.price` lets callers write `order.total` with no parentheses. Use it when a value is derived from other attributes, when reading needs validation or a lazy computation, or when a plain attribute must later gain logic without changing callers. Add `@total.setter` to allow assignment.
The reasoning
Python starts with plain attributes because they are simple. A property keeps the same syntax for callers while running code underneath. A class can begin with `self.total = ...` and switch to a computed property later with no caller changes.
The setter form validates: `@amount.setter def amount(self, value): if value < 0: raise ValueError(...)`. Without a setter, assignment raises `AttributeError`, which makes a property a cheap read-only attribute.
A property runs on every access. A heavy computation behind a property surprises callers who loop over it. For an expensive one-time computation, `functools.cached_property` stores the result on the instance after the first access.
Properties signal cheapness and purity. If the operation has side effects or takes arguments, make it a method so the call is visible.
The answer most people give
"Use properties for every attribute, like getters and setters." Python's convention is plain attributes until logic is needed. A property for every field is noise that buys nothing.
They’ll ask next
A property calls a database on every read and a loop calls it a million times. What is the fix?
What is an `Enum` in Python? Why use one instead of string constants for things like a job status?
Why they ask this
Bare strings for statuses are the source of a whole class of pipeline bugs, and the enum is the standard fix.
Say this
An `Enum` is a class whose members are a fixed set of named constants: `class Status(Enum): PENDING = 'pending'; DONE = 'done'`. Members are singletons, compared by identity, iterable, and looked up by value with `Status('done')`, which raises `ValueError` on an unknown value. It replaces string constants so a typo is an error at the point of use, not a silent mismatch downstream.
The reasoning
With strings, `if status == 'succes':` is a bug nobody sees until the branch never runs. With `Status.SUCCESS`, a misspelling is an `AttributeError` immediately. The set of valid states is declared once and the type checker knows it.
`Status('done')` converts an incoming string from a database or an API into the member, and fails loudly on a value that is not in the set. That is exactly the validation a boundary needs. `.value` goes back to the string for storage; `.name` gives the identifier.
`StrEnum`, added in Python 3.11, makes members behave as strings too, so they format into SQL and JSON without `.value`. `IntEnum` does the same for integers. `auto()` assigns values when the exact value does not matter.
Enums are hashable and work as dictionary keys, and `Enum` members can carry methods. So the rule "which states are terminal" lives on the enum instead of in scattered `if` chains.
The answer most people give
"Define constants as module-level strings; that is the same thing." The constants do not validate incoming values and do not stop a typo in a comparison. The enum does both.
They’ll ask next
A row arrives with `status = 'cancelled'` and the enum has `CANCELED`. What happens, and where should it be caught?
What is the difference between a module and a package? How does `import` find them?
Why they ask this
Most "it works on my laptop" failures in pipelines are import-path problems, and candidates should be able to explain the search.
Say this
A module is one `.py` file; a package is a directory of modules, traditionally marked by an `__init__.py`. `import pipeline.extract` searches the directories in `sys.path`, which starts with the script's own directory, then `PYTHONPATH`, then the environment's site-packages, and imports the first match. A package imports its `__init__.py` first, and a module is executed once and cached in `sys.modules`.
The reasoning
`extract.py` is a module; `import extract` runs the file once, binds its names, and stores the module object in `sys.modules` so later imports reuse it. Top-level code in a module runs at import, which is why a module that opens a connection at the top opens it whenever anything imports it.
A directory `pipeline/` with `__init__.py` is a regular package. `import pipeline.extract` runs `pipeline/__init__.py` then `pipeline/extract.py`. Without `__init__.py` it is a namespace package, which works but behaves differently and is easy to confuse with a mistake.
The search order is `sys.path`. The tutorial states it begins with the directory containing the input script, then `PYTHONPATH`, then the installation-dependent default. Running `python src/jobs/run.py` therefore puts `src/jobs` first, and `import pipeline` may fail that it found from the repo root. Running `python -m jobs.run` from the root fixes the path.
Installing the project into the environment, `pip install -e .`, puts the package on `sys.path` the same way everywhere. That removes the class of bugs where the path differs between laptop, container and Airflow worker.
The answer most people give
"Python finds modules relative to the file that imports them." It searches `sys.path`, which depends on how the program was started. Relative imports exist only inside a package.
They’ll ask next
A job runs on a laptop and fails on the Airflow worker with `ModuleNotFoundError`. What differs, and what makes it the same?
What is `__init__.py` for? What should and should not go in it?
Why they ask this
It is a file every repository has and few people can explain, and misusing it creates slow imports and circular ones.
Say this
`__init__.py` marks a directory as a regular package and runs when the package is imported. It can be empty. It may define the package's public names by importing them from submodules and setting `__all__`, so callers write `from pipeline import load`. It should not do work: no connections, no config reads, no heavy imports, because every import of any submodule runs it first.
The reasoning
The tutorial describes `__init__.py` as the file required to make Python treat a directory as a package, and notes it can be empty or execute initialization code. The common useful content is re-exports: `from .extract import extract` so the public API is the package, not its file layout.
Because it runs on the first import of the package or any submodule, side effects in it fire in every process that touches the package. That includes test collection and Airflow's DAG parsing. A database connection there is opened by the scheduler every parse.
Heavy imports in `__init__.py` make importing a small utility expensive, because the whole package tree loads. Circular imports often start here: `__init__` imports `a`, `a` imports the package, which is half-initialised.
Since Python 3.3 a directory without `__init__.py` is an implicit namespace package. It works for simple layouts, but tools and type checkers handle regular packages more predictably, so the convention is to keep the file, often empty.
The answer most people give
"`__init__.py` is where you initialise the package's objects." It is where you declare the package. Initialising resources there runs them on every import, in every process.
They’ll ask next
Airflow's scheduler parses DAG files every 30 seconds and a package `__init__.py` reads a Variable. What happens?
What does `if __name__ == '__main__':` do? Why does a pipeline script need it?
Why they ask this
A one-liner that every script contains and that explains how a module can be both importable and runnable.
Say this
Python sets a module's `__name__` to `'__main__'` when the file is run as a script, and to the module's dotted name when it is imported. The guard runs the block only in the first case, so a file can define functions that other modules import without also running its job, and tests can import it safely.
The reasoning
Running `python load.py` executes the file top to bottom with `__name__ == '__main__'`. Importing `load` from another module executes it with `__name__ == 'load'`. The guard puts the entry point, argument parsing and the call to `main()`, under the first case.
Without it, importing the module to test one function would run the whole job: read the config, connect, load. With it, the import defines functions and stops.
It also matters for multiprocessing on platforms that spawn new interpreters, macOS and Windows. The child re-imports the main module, and without the guard it would re-run the job that started the pool.
The convention is a `main()` function that takes parsed arguments, called from the guard. The same entry point can then be invoked from a console script declared in `pyproject.toml` or from an orchestrator.
The answer most people give
"It is the function Python calls to start the program." It is a condition, not a function. Python runs the file; the guard decides which part is the entry point.
They’ll ask next
A `ProcessPoolExecutor` started at module top level on macOS spawns workers forever. Which line is missing?
What is a virtual environment, and why pin dependency versions? What is the difference between a constraint and a lock file?
Why they ask this
Reproducible builds are a production requirement, and the vocabulary separates people who have shipped from people who have run notebooks.
Say this
A virtual environment is an isolated Python installation with its own `site-packages`, created with `python -m venv .venv`, so a project's dependencies do not collide with another's or with the system. Pinning records exact versions so the same code runs with the same libraries tomorrow and in production. A lock file pins the full resolved tree with hashes; a constraints or requirements file lists what you asked for.
The reasoning
Without an environment, every project shares one `site-packages`, and installing pandas 2.3 for one job breaks the job that needed 1.5. The `venv` module creates a directory with its own interpreter link and packages; activating it puts that on `PATH`.
`requirements.txt` with `pandas` installs whatever is newest today. The same file next month installs a release that changed behaviour, and the nightly job starts failing with no code change. `pandas==2.2.3` pins it. A lock file, from pip-tools, Poetry or uv, pins every transitive dependency too, with hashes, so the environment is byte-for-byte reproducible.
The split is between what you declare and what you resolve. `pyproject.toml` declares `pandas>=2.2`, the range you support. The lock file records the exact set that was tested. CI installs from the lock; a dependency upgrade is a deliberate change to the lock, reviewed like code.
Containers do not replace this. A `Dockerfile` that runs `pip install pandas` has the same drift; the image is reproducible only if the install inside it is pinned.
The answer most people give
"We use Docker, so we do not need to pin." The image builds whatever is newest at build time. Reproducibility comes from the lock file inside the image, not from the image.
They’ll ask next
The job worked for a year and failed last night with no code change. What is the first file you check?
What are type hints? What do `Optional`, `Union` and `TypedDict` mean, and what does Python do with hints at runtime?
Why they ask this
Half of engineers believe hints are enforced. The question finds out, and the three names test practical fluency.
Say this
Type hints annotate parameters, returns and variables with their intended types: `def load(rows: list[Row]) -> int`. Python stores them in `__annotations__` and does nothing else; a static checker such as mypy or pyright reads them, and libraries such as pydantic use them for validation. `Optional[X]` means `X | None`; `Union[X, Y]` means either; `TypedDict` declares the keys and value types of a dict.
The reasoning
At runtime a hint is metadata. `def f(n: int)` called with `'12'` runs, and the string flows through. The checks happen before the program runs, in the editor or in CI, where a checker reports the mismatch. That is where the value is: errors found before the job starts, and code that documents itself.
`Optional[str]`, written `str | None` since 3.10, says the value may be missing, and a checker forces you to handle the `None` branch. `Union[int, str]` says one of several types. `TypedDict` gives a plain dict a declared shape, `class Row(TypedDict): id: int; amount: float`, so a typo in a key is caught, without converting the data to a class.
Validation at runtime is a separate decision. A `dataclass` does not check types; pydantic's `BaseModel` does, by reading the same hints and coercing or rejecting. The usual design validates once at the boundary where untrusted data enters, then trusts the typed objects inside.
Hints have a cost at import time only for complex annotations, which `from __future__ import annotations` defers. Generics such as `list[Row]` and `dict[str, int]` are the everyday forms; `Any` switches checking off and should be rare.
The answer most people give
"Python checks the types when the function is called." It does not. A hint is documentation for a checker, and a string in an `int` parameter runs until something downstream breaks.
They’ll ask next
A dataclass field is `rows: int` and a caller passes `'12'`. What happens, and where should the check live?
What is duck typing? How do `typing.Protocol` and abstract base classes relate to it?
Why they ask this
It is the design principle behind Python's standard library, and the two typing tools are how modern code makes it explicit.
Say this
Duck typing means code uses an object by the methods it has, not by its declared class: anything with a `.read()` can be passed where a file is expected. A `Protocol` writes that expectation down as a structural type a checker can verify without inheritance. An abstract base class makes the expectation nominal: a class qualifies only by inheriting and implementing the abstract methods.
The reasoning
`json.load(f)` works on a real file, a `StringIO`, a gzip stream and an S3 body, because each has `.read()`. The function never checks the class. That is duck typing: if it reads like a file, it is a file.
The weakness is that the contract is implicit. A caller passes something with `.read` that returns bytes where text was expected, and the error appears deep inside. `Protocol` fixes the documentation side: `class Readable(Protocol): def read(self, n: int = -1) -> str: ...`, and a checker verifies any argument structurally.
An ABC takes the other route. `class Source(ABC)` with `@abstractmethod read()` means a class must inherit from `Source` and implement `read`, checked when the class is instantiated. It is explicit and discoverable, at the cost of requiring inheritance that third-party objects cannot have.
In a pipeline codebase, Protocols suit boundaries where many unrelated objects must fit, such as sinks and readers. ABCs suit a family of your own classes that share behaviour and want a common base.
The answer most people give
"Duck typing means Python has no types." Every object has a type. Duck typing means the code does not check the type, only the behaviour, and `Protocol` lets a checker do that check for you.
They’ll ask next
A third-party client has the right methods but does not inherit your ABC. Which tool lets the checker accept it?
What do `try`, `except`, `else` and `finally` each do? What does `raise ... from` add?
Why they ask this
Exception handling is where pipelines either fail loudly or hide errors, and `else` and `from` are the two parts most people have never used.
Say this
`try` wraps code that may raise. `except` catches a matching exception and handles it. `else` runs only if the `try` block raised nothing, keeping the success path out of the protected block. `finally` runs on every exit, for cleanup. `raise NewError(...) from original` records the original as the cause, so the traceback shows both and the chain is explicit rather than the accidental "during handling of the above exception".
The reasoning
Keep the `try` block small: only the statement that can fail. Put what should happen on success in `else`, so an exception raised by the success code is not caught by the same `except` and misreported as the original failure.
Catch specific types. `except Exception` swallows a `KeyboardInterrupt` sibling's cousin and every bug; `except (ConnectionError, TimeoutError)` says what you expect. A bare `except: pass` is the single most damaging line in pipeline code, because it turns a failed write into a successful run.
`finally` is for cleanup that must happen regardless: close, release, record the end time. `return` inside `finally` overrides any exception, which is a trap. Context managers are the cleaner form of most `finally` blocks.
Exception chaining matters for debugging. Raising a domain error inside an `except` without `from` gives the implicit chain, and `raise LoadError('batch 42') from e` gives the explicit one. `from None` suppresses the chain on purpose when the original is noise.
The answer most people give
"Put the whole function body in `try` and catch `Exception` so the job never crashes." A job that never crashes also never reports. The point of an exception is to stop and say why.
They’ll ask next
A retry wrapper catches `Exception` and retries a write that already committed. What goes wrong, and which exceptions should it retry?
What is the difference between inheritance and composition? Which one should a pipeline codebase prefer, and why?
Why they ask this
Deep class hierarchies are a common wrong turn in ETL code, and the question checks design judgment rather than syntax.
Say this
Inheritance makes a class a kind of another, `class S3Source(Source)`, reusing and overriding its behaviour. Composition makes a class have another, `self.client = S3Client()`, delegating to it. Prefer composition for most pipeline code: it keeps behaviours swappable and testable, while inheritance couples the child to the parent's internals and grows into fragile hierarchies.
The reasoning
Inheritance is right when there is a true is-a relationship and a stable base: a family of sources sharing a `read()` contract, with an ABC declaring it. The child gets the parent's methods for free and can override one.
The cost appears over time. A change in the parent changes every child. A child that needs two parents' behaviour gets multiple inheritance and the MRO. Testing a child means constructing the whole chain.
Composition builds objects from parts. A `Loader` holds a `connection`, a `retry_policy` and a `serializer` passed in at construction. Each part has one job, can be replaced by a fake in a test, and can be reused by an unrelated class. This is also what dependency injection means in Python: pass the collaborator in.
Mixins are the middle ground, small classes adding one behaviour through inheritance; they are useful in moderation and the usual source of the MRO question.
The answer most people give
"Make a base `Pipeline` class and subclass it for every job." Jobs differ in parts, not in kind. Compose each job from source, transform and sink objects instead of inheriting a god class.
They’ll ask next
You need to test a loader without a database. Which design makes that trivial, and what do you pass in?
What is the `logging` module, and why use it instead of `print`? What are the levels, and where should configuration live?
Why they ask this
Operability is judged by logs, and `print` in a pipeline is a reliable sign of a script that was never run in production.
Say this
`logging` is the standard library's structured event system: modules call `logger = logging.getLogger(__name__)` and emit records at levels `DEBUG`, `INFO`, `WARNING`, `ERROR` and `CRITICAL`, documented as 10, 20, 30, 40 and 50. Handlers decide where records go and formatters how they look. Unlike `print`, output can be filtered by level, routed to files or collectors, and carries time, module and severity. Configure handlers once, in the entry point, never in library modules.
The reasoning
`print` writes a string to stdout. It cannot be turned down in production, carries no timestamp or source, and goes wherever stdout goes. `logger.info('loaded %d rows', n)` is a record with a level, a logger name, a time and a message, and it is formatted only if a handler wants it.
Levels are thresholds. A logger set to `WARNING` drops `INFO` and `DEBUG` records before formatting them, so verbose debug logging costs nothing when it is off. `logger.exception()` inside an `except` logs at `ERROR` with the traceback attached.
Configuration is the entry point's job: `logging.basicConfig` or `dictConfig` in `main()`. It sets the root level, a handler to stderr or a file, and a format with time, level, name and message. A library module that configures logging overrides the application's choices for everyone who imports it.
Use `%`-style lazy formatting in the call, `logger.debug('row %s', row)`, so the string is built only if the level is enabled. Add context with `extra=` or a structured JSON formatter, so a log aggregator can filter on `job_id` and `partition`.
The answer most people give
"Use `print` for simple scripts and switch to logging later." Later never comes, and the script is already running in production. The two lines of `logging` setup cost nothing.
They’ll ask next
A library module calls `logging.basicConfig` at import. What does that do to the application that imports it?
What are pytest fixtures and `parametrize`? What does `unittest.mock` do, and what should you mock in a pipeline test?
Why they ask this
Testing is where pipeline code quality is decided, and candidates who have only run a job end to end cannot answer this.
Say this
A fixture is a function, marked `@pytest.fixture`, that builds something a test needs and passes it in by argument name: a sample DataFrame, a temporary directory, a fake client. `parametrize` runs one test function over a list of inputs and expected outputs. `unittest.mock` replaces an object with a fake you control, via `patch`, so a test can run without the real database or API. Mock the boundaries, never the logic under test.
The reasoning
Fixtures remove setup duplication and make it explicit. `def test_dedup(sample_rows):` gets the rows from the fixture; pytest resolves the name. Fixtures compose, have scopes from function to session, and clean up with `yield`. `tmp_path` and `monkeypatch` are built in.
`@pytest.mark.parametrize('raw, expected', [('1,2', (1, 2)), ('', None)])` turns a table of cases into separate test runs with separate reports. It is the natural way to test a parser against its edge cases: empty input, a bad row, a boundary value.
`patch('pipeline.load.get_client')` replaces the client for the duration of a test with a `MagicMock`, whose return values you set and whose calls you assert. The rule is to patch where the name is looked up, the module under test, not where it is defined. Mock only what crosses a boundary: network, database, time, randomness.
A test that mocks the function it is testing proves nothing. The shape that works is pure transformation functions tested with small fixtures, boundaries tested with fakes, and one slow end-to-end test on a sample marked to run separately.
The answer most people give
"We test the pipeline by running it against the dev database." That is one integration test with no coverage of edge cases and no way to run in CI. Unit tests on the transform logic with fixtures are what catch the bad row.
They’ll ask next
A test patches `requests.get` but the module does `from requests import get`. Why does the patch not apply, and what do you patch instead?
What is the difference between `sorted()` and `list.sort()`? What is a key function, and what does stable sorting guarantee?
Why they ask this
A small question with three parts that each catch a common mistake: the `None` return, the key, and ties.
Say this
`sorted(iterable)` returns a new sorted list from any iterable and leaves the input alone. `list.sort()` sorts a list in place and returns `None`, so `rows = rows.sort()` sets `rows` to `None`. A key function, `key=lambda r: r.date`, computes the value to compare for each item once. Both sorts are stable, documented as such: items that compare equal keep their original order, which makes multi-key sorts possible by sorting on the secondary key first.
The reasoning
`sorted` works on generators, tuples and dict views and always builds a list. `.sort()` exists only on lists and avoids the copy, which matters for a large list already in memory. The `None` return is deliberate, to make the in-place mutation visible, and it is the source of a classic bug.
The key function replaces comparison functions. `key=str.lower` for case-insensitive, `key=lambda r: (r.region, -r.amount)` for region ascending then amount descending, `key=itemgetter(2)` or `attrgetter('date')` from `operator`. The key is called once per element, not once per comparison, so an expensive key is fine.
Stability is the guarantee that equal keys keep input order. Sort by amount, then by region, and within each region the rows are still ordered by amount. That is the documented way to sort on several keys when one cannot be negated, such as a date string descending: sort by date with `reverse=True` first, then by region.
`reverse=True` reverses the result while keeping stability, so it is not the same as negating a numeric key for mixed-direction sorts; the two-pass approach is what works for any type.
The answer most people give
"`rows.sort()` returns the sorted list." It returns `None`. The list is sorted in place, and the assignment throws it away.
They’ll ask next
Sort rows by region ascending and by a timestamp string descending, where the string cannot be negated. How?
File I/O & formats (CSV/JSON/Parquet)Concurrency (threading vs multiprocessing vs asyncio)
What is pickling? When is JSON the better choice, and why should you never unpickle untrusted data?
Why they ask this
Serialization is everywhere in pipelines, multiprocessing, caches, object storage, and the security line is one the interviewer expects.
Say this
Pickling serializes a Python object graph to bytes that Python can load back into the same objects, including custom classes, through `pickle.dumps` and `loads`. JSON serializes only dicts, lists, strings, numbers, booleans and null, as text any language can read. Use pickle for Python-to-Python transport such as multiprocessing or a local cache; use JSON for anything stored, shared or received. The `pickle` docs warn that loading a pickle can execute arbitrary code, so never unpickle data you did not produce.
The reasoning
Pickle preserves Python types: a `datetime`, a `Decimal`, a dataclass, a set. That is why `multiprocessing` uses it to send arguments to workers, and why a function or lambda that cannot be pickled fails there.
Its limits follow from that. The bytes are tied to Python and often to the class definitions that produced them. A pickle written by one version of your code may not load in the next. No other language reads it. And `pickle.load` can be made to run code, which the documentation states plainly.
JSON is the opposite: a small set of types, human-readable text, universal. Dates become strings by convention, often ISO 8601 and UTC, and `Decimal` needs an explicit choice. `json.dumps(obj, default=str)` is the quick way; a proper encoder is the durable one.
For data at rest in a pipeline, neither is the final answer: Parquet or Avro carry a schema and compress, with JSON for raw landing and exchange. Pickle is for ephemeral, trusted, same-code transport only.
The answer most people give
"Pickle the DataFrame to S3 so the next job can load it." The next job's pandas version may not read it, nothing else can, and anyone who can write to that bucket can run code in your job. Write Parquet.
They’ll ask next
A `ProcessPoolExecutor` call fails with a pickling error on a lambda. Why, and what do you pass instead?
What is the difference between opening a file in text mode and in binary mode? What is an encoding, and why should you always pass `encoding=` explicitly?
Why they ask this
Encoding bugs are the most common data corruption in file ingestion, and the question checks that the candidate knows bytes from text.
Say this
Binary mode, `open(path, 'rb')`, reads raw bytes. Text mode, `open(path, 'r', encoding='utf-8')`, decodes bytes into `str` using an encoding and translates newlines. An encoding is the mapping between characters and bytes; UTF-8 is the standard. The `open` docs say the default encoding is platform-dependent, so a file that reads correctly on a UTF-8 laptop can fail or silently corrupt on a Windows or legacy-locale server unless the encoding is stated.
The reasoning
A file is bytes. To get text, Python must know the encoding. UTF-8 represents every character and is the default for JSON and most modern exports; Latin-1 and Windows-1252 are what older systems and spreadsheets produce. The same byte `0xE9` is `é` in Latin-1 and the start of a multi-byte sequence in UTF-8.
Binary mode is for non-text: Parquet, gzip, images, and anything you will hand to a library that decodes itself. It also lets you sniff the first bytes of a file for a byte-order mark or a magic number before deciding how to decode.
Python 3.15 moves the default to UTF-8, and `python -X utf8` or `PYTHONUTF8=1` does so today; stating `encoding='utf-8'` in every `open` is the portable habit regardless.
The answer most people give
"Open the file in text mode and Python figures out the encoding." It applies the platform default. On the wrong platform that is a silent mojibake, not an error.
Concurrency (threading vs multiprocessing vs asyncio)GIL & memory
What is a race condition? What does a lock do, and why does the GIL not prevent races?
Why they ask this
It is the follow-up to every threading answer, and the GIL half catches the common misconception.
Say this
A race condition is a bug where the result depends on the timing of two or more threads or processes touching shared state: two threads read a counter as 5, both write 6, and one increment is lost. A lock, `threading.Lock`, lets only one thread hold it at a time, so the read-modify-write happens as a unit. The GIL serialises bytecode execution, not whole statements, so a thread switch can still land between the read and the write.
The reasoning
`self.n += 1` compiles to several bytecodes: load `n`, add 1, store `n`. The interpreter can switch threads between any two of them. Two threads that both load before either stores both write the same value, and 400,000 increments end below 400,000.
`with self.lock: self.n += 1` makes the three steps exclusive. A thread that cannot acquire the lock waits. `Lock` is the simple form; `RLock` allows the same thread to re-acquire; `Semaphore` limits concurrency to N.
The GIL protects the interpreter's own structures, reference counts and object internals, so a list does not corrupt itself under concurrent appends. It does not protect your invariants, which span several operations. That is the difference between "thread-safe" for CPython and "correct" for your program.
Locks bring their own hazards: deadlock when two threads acquire two locks in opposite orders, and contention when one lock guards too much. Queues and single-writer designs avoid both, and processes avoid shared memory entirely.
The answer most people give
"The GIL makes Python threads safe, so no locks are needed." It makes the interpreter safe. Your counter still loses updates, which is the easiest race to demonstrate.
They’ll ask next
Two threads each add to a shared dict from different keys. Is a lock needed, and what changes if they update the same key?
EvergreenData structures & complexityConcurrency (threading vs multiprocessing vs asyncio)
Why is Python used so much in data engineering? What are its limits?
Why they ask this
The opening question of a Python screen, and the limits half is where candidates show they have hit them.
Say this
Python is the glue: readable, with a standard library for files, JSON, HTTP and processes, and the language of pandas, PySpark, Airflow, dbt's Jinja and every cloud SDK. Its limits are speed and memory for per-row work in pure Python, the GIL for CPU-bound threads, and loose typing that pushes validation onto the engineer.
The reasoning
A data engineer writes orchestration, extraction, validation and light transformation, and talks to many systems. Python has a client for all of them and reads like pseudocode, so the code is reviewable by the whole team.
The heavy lifting happens elsewhere. Spark, a warehouse or a columnar library does the per-row work in compiled code; Python expresses what to do. That division is why a 10-line PySpark job beats a 10-line pure-Python loop by orders of magnitude on large data.
Pure Python is slow per operation and large per object. A loop over 100 million rows is minutes; 100 million small objects are gigabytes. The GIL means threads do not parallelise CPU-bound code, so parallelism means processes or a different engine.
Types are hints, not checks, so a wrong type travels until it breaks. The mature answer is validation at the boundary, type checking in CI, and vectorised or distributed engines for volume.
The answer most people give
"Python is fast enough for big data because of libraries." The libraries are fast because they are not Python under the hood. Pure-Python per-row work is the thing to avoid.
They’ll ask next
A pure-Python loop parses 40 GB of JSON in 6 hours. What are the two directions to make it faster?
What is pip, and what is `pyproject.toml`? What replaced `setup.py` and `requirements.txt` for a project?
Why they ask this
Packaging vocabulary dates a candidate quickly, and the modern answer is one file.
Say this
pip is Python's package installer; it resolves and installs packages from PyPI or a private index into the active environment. `pyproject.toml` is the standard project file that declares the project's name, version, dependencies and build system in one place, replacing `setup.py` for metadata. `requirements.txt` remains a flat list of pins for an environment; a lock file from uv, Poetry or pip-tools is the stricter form.
The reasoning
`pip install pandas==2.2.3` downloads the wheel and its dependencies and installs them into the environment on `PATH`. `pip install -e .` installs your own project in editable mode so imports resolve to the source tree.
`pyproject.toml` under `[project]` lists `dependencies`, optional groups such as `dev`, the Python version, and entry points that become console commands. Under `[build-system]` it names the backend. Tools read their own tables from the same file: ruff, pytest, mypy.
`setup.py` still works for complex builds but is no longer the place for metadata. `requirements.txt` is not a project definition; it is a snapshot of an environment, useful for deployment and generated from the project's declared dependencies.
uv and Poetry add a lock file and manage the environment and the install in one tool, which is what most new data projects use. pip remains the common denominator every image has.
The answer most people give
"`requirements.txt` is where a project declares its dependencies." It is a pin list for an environment. The declaration belongs in `pyproject.toml`, and the lock file records the resolution.
They’ll ask next
Why would a library declare `pandas>=2.0` in `pyproject.toml` while an application pins `pandas==2.2.3` in its lock?
What is a dictionary in Python, and how is it implemented? Why are lookups O(1), and are dicts ordered?
Why they ask this
The most used structure in data code, and the implementation question separates users from engineers.
Say this
A dict maps hashable keys to values. It is a hash table: the key's `hash()` picks a slot in an array, so lookup, insert and delete are O(1) on average. Since Python 3.7 the language guarantees insertion order is preserved. Keys must be hashable, which excludes lists and dicts and includes strings, numbers, tuples of hashables and frozen dataclasses.
The reasoning
`d['order_42']` computes the hash of the string, takes it modulo the table size, and looks at that slot. On a collision it probes further slots. Equal keys must hash equal, which is the contract between `__eq__` and `__hash__`.
CPython's dict stores a compact array of entries in insertion order plus a sparse index of slots. That is why ordering came for free in 3.6 and became a guarantee in 3.7. Iteration follows insertion; `popitem()` removes the last inserted.
The table resizes as it fills, so memory is more than the entries alone: roughly 50 to 100 bytes per entry for small keys. Ten million entries is around a gigabyte, which is the number to remember before building a lookup for a hundred million keys.
A dict is the right structure for a join index, a lookup table and deduplication by key. `defaultdict`, `Counter` and `OrderedDict` are thin specialisations; `dict.get`, `setdefault` and dict comprehensions cover most everyday needs.
The answer most people give
"Dict order is arbitrary, so never rely on it." That was true before 3.7. It is now a language guarantee, and code that depends on insertion order is correct.
They’ll ask next
Why can a list not be a dict key, and what do you use instead for a composite key?
What is an f-string? Where should you not use one?
Why they ask this
A one-liner with two traps the interviewer knows: SQL and logging.
Say this
An f-string, `f'loaded {n} rows in {secs:.1f}s'`, evaluates the expressions in braces at the point the string is built and formats them inline. It is the clearest way to build text. Do not use it to build SQL with values, which is an injection bug and defeats query caching, and avoid it in log calls, where the string is built even when the level is off.
The reasoning
Expressions inside the braces can be any Python: attribute access, calls, arithmetic. Format specs follow a colon: `{amount:,.2f}` for thousands separators and two decimals, `{dt:%Y-%m-%d}` for dates, `{name!r}` for the repr. Since 3.12, quotes may be reused inside the braces.
For SQL, `f"WHERE id = {user_id}"` puts the value into the text. A value that contains a quote breaks the query or changes it. Pass values as parameters, `cursor.execute('WHERE id = %s', (user_id,))`, so the driver sends them as data. Parameters cannot replace table or column names; those need an allowlist.
For logging, `logger.debug(f'row {row}')` formats the row even when debug logging is disabled. `logger.debug('row %s', row)` defers formatting to the handler and costs nothing when the level is off.
f-strings are evaluated once, at creation. A template that must be reused with different values is `str.format` or a template engine, which is how dbt and Airflow render Jinja.
The answer most people give
"f-strings are just faster string formatting." They are also the easiest way to write an injection bug. The rule is text with f-strings, values to SQL as parameters.
They’ll ask next
How do you let a caller choose which column to sort by, given that a parameter cannot replace a column name?
What is a set in Python? What operations is it built for, and what does it cost?
Why they ask this
Membership and deduplication are everyday data tasks, and the set is the structure that makes them cheap.
Say this
A set is an unordered collection of unique, hashable elements, implemented as a hash table of keys without values. Membership, add and remove are O(1) on average, and sets support union, intersection, difference and symmetric difference with operators. The cost is memory, around 30 to 50 bytes per element plus the elements, and the loss of order.
The reasoning
`ids_seen = set()` with `if x in ids_seen` is the deduplication idiom. A million checks against a million entries is a million hashes, where the same checks against a list are a trillion comparisons.
`a & b` gives common elements, `a - b` those only in `a`, `a | b` the union. Reconciliation between two ledgers is three set operations on their key sets. `frozenset` is the immutable, hashable version, usable as a dict key.
Elements must be hashable. A list cannot go in a set; a tuple can. Rows as dicts cannot be deduplicated by putting them in a set; convert them to tuples or hash a key.
A set of 50 million ints is several gigabytes, which is where exact deduplication in memory stops and sorting, partitioning or a probabilistic structure begins.
The answer most people give
"A set is a list without duplicates." It is a hash table. The absence of duplicates is a consequence of hashing, and so is the O(1) membership that a list cannot give.
They’ll ask next
Deduplicate 50 million UUID strings. How much memory does a set need, and what do you do when it does not fit?
What is slicing in Python? What do `a[start:stop:step]`, negative indexes and `a[::-1]` mean, and does a slice copy?
Why they ask this
A one-liner that still trips people on the exclusive stop and on the copy.
Say this
Slicing selects a range from a sequence: `a[2:5]` returns elements at indexes 2, 3 and 4, with the stop excluded. `step` skips: `a[::2]` every second element, `a[::-1]` reversed. Negative indexes count from the end, `a[-3:]` is the last three. A slice of a list, string or tuple creates a new object; a NumPy slice is a view into the same memory.
The reasoning
The half-open convention, inclusive start and exclusive stop, makes `a[:n]` the first `n` elements and `a[n:]` the rest, with no overlap and no off-by-one. `len(a[i:j])` is `j - i` when in range. Out-of-range bounds are clamped, not errors, unlike single-index access.
Slice assignment edits a list in place: `a[2:5] = [0]` replaces three elements with one. `del a[::2]` removes every second element. Strings and tuples are immutable, so their slices are always new objects.
A list slice copies the references, so `b = a[:]` is a shallow copy: a new list, the same element objects. For nested data that is the shallow-copy trap, and `copy.deepcopy` is the fix.
NumPy and pandas differ: slicing an array returns a view, and writing to the view writes to the original. pandas' chained assignment warning comes from exactly this ambiguity.
The answer most people give
"`a[1:3]` returns three elements." It returns two: indexes 1 and 2. The stop is exclusive, which is what makes `a[:n]` and `a[n:]` partition the list cleanly.
They’ll ask next
`b = a[:]` and then `b[0].append(1)` changes `a[0]`. Why, and what copy avoids it?
What do EAFP and LBYL mean in Python? Which one is idiomatic, and when is the other right?
Why they ask this
The glossary question that reveals whether the candidate writes Python or translated Java.
Say this
EAFP is easier to ask forgiveness than permission: try the operation and catch the exception, `try: v = d[k] except KeyError: ...`. LBYL is look before you leap: check first, `if k in d: v = d[k]`. Python's glossary calls EAFP the common style, because it avoids a race between the check and the use and keeps the happy path clean. LBYL is right when the exception is expensive or expected to be common, or when checking is clearer.
The reasoning
EAFP matches Python's cheap exceptions and its duck typing: ask the object to do the thing, and handle the specific failure. `try: f = open(path) except FileNotFoundError:` has no window in which the file can disappear between a check and the open.
LBYL reads naturally for simple conditions and avoids exceptions in hot loops. Raising and catching costs more than an `if`, so a parser that expects half its rows to be bad should check, not catch.
The pipeline rule is to combine them: validate at the boundary with explicit checks, because bad input is expected there, and use EAFP inside, where failure is the exception.
Both styles must catch the specific exception. `except Exception` around an EAFP block turns a typo elsewhere into a handled case, which is the same bug in different clothes.
The answer most people give
"Always check before you act; exceptions are for errors." In Python exceptions are flow control for the uncommon case, and checking first can race with the action.
They’ll ask next
Why is `if os.path.exists(path): open(path)` worse than `try: open(path)` on a shared file system?
EvergreenData structures & complexityConcurrency (threading vs multiprocessing vs asyncio)
What do `global` and `nonlocal` do? Why does assigning to a variable inside a function make it local?
Why they ask this
The `UnboundLocalError` is a rite of passage, and the two keywords are the vocabulary for explaining it.
Say this
Python decides a variable's scope at compile time: any name assigned anywhere in a function is local to that function, so reading it before the assignment raises `UnboundLocalError`, even if a global of that name exists. `global x` declares that `x` refers to the module-level name. `nonlocal x` declares that `x` refers to the nearest enclosing function's variable, which is how a closure modifies the variable it captured.
The reasoning
`count = 0; def inc(): count += 1` fails, because `count += 1` assigns, which makes `count` local to `inc`. The read on the right happens before any local assignment. Adding `global count` makes both the read and the write hit the module variable.
Nested functions see enclosing variables for reading. Writing needs `nonlocal`: `def counter(): n = 0; def inc(): nonlocal n; n += 1; return n; return inc` keeps `n` alive between calls as the closure's state.
Mutating is different from assigning. `items.append(x)` inside a function changes the global list without any keyword, because no name is rebound. That is also why mutable default arguments and shared module-level lists cause surprises.
In pipeline code, globals are mostly configuration read once, and state that must be shared belongs in an object passed explicitly, not in a `global` declaration inside a task.
The answer most people give
"Python looks up the variable at runtime, so it finds the global." Scope is decided when the function is compiled. An assignment anywhere in the body makes the name local everywhere in the body.
They’ll ask next
A worker function in a process pool updates a global dict and the parent never sees it. Is that a scoping problem?
What do `enumerate` and `zip` do? What does `zip` do with inputs of different lengths?
Why they ask this
Two builtins used in every loop, with one trap the interviewer expects the candidate to name.
Say this
`enumerate(iterable, start=0)` yields `(index, item)` pairs, replacing a manual counter. `zip(a, b)` yields tuples pairing the i-th element of each input, lazily. By default `zip` stops at the shortest input and drops the rest silently; since Python 3.10, `zip(a, b, strict=True)` raises `ValueError` if the lengths differ, which is the setting to use when the inputs are supposed to match.
The reasoning
`for i, row in enumerate(rows, start=1):` gives a line number for error messages without a separate counter variable. It works on any iterable, including a generator, and costs nothing because it is lazy.
`zip(headers, values)` pairs a header row with a data row, and `dict(zip(headers, values))` makes a record from a CSV line. `zip(*rows)` transposes a list of rows into columns, which is the one-line pivot people reach for.
The length trap is silent. Headers with 12 names and a row with 13 fields zip to 12 pairs and the 13th value is gone. `strict=True` turns that into an error at the row that is wrong, which is what a parser wants.
`zip` consumes iterators as it goes, so zipping a generator with itself pairs consecutive elements, and a partially consumed zip leaves its inputs advanced. `itertools.zip_longest` fills the shorter input instead of truncating.
The answer most people give
"`zip` pairs up the lists and errors if they do not match." It truncates to the shortest by default. The error needs `strict=True`.
They’ll ask next
A CSV row has one extra field and `dict(zip(headers, row))` succeeds. What was lost, and what change makes it fail loudly?
What is PEP 8? Why does style matter in pipeline code, and how is it enforced?
Why they ask this
A soft question with a hard answer: tools, not opinions, keep a codebase consistent.
Say this
PEP 8 is the Python style guide: naming conventions such as `snake_case` for functions and variables and `CapWords` for classes, 4-space indentation, line length, import ordering and whitespace. Style matters because pipeline code is read during incidents by people who did not write it. It is enforced by a formatter and a linter in pre-commit and CI, today usually ruff, so reviews discuss logic instead of layout.
The reasoning
PEP 8 is a document, PEP 8 the practice is a toolchain. A formatter such as black or ruff format rewrites code to one style automatically. A linter such as ruff or flake8 flags unused imports, shadowed names and known bug patterns, and isort or ruff orders imports.
Running them in a pre-commit hook means no unformatted code is committed; running them in CI means no unformatted code is merged. The team never argues about brace placement, because the tool decided.
The parts that matter for data code: descriptive names over abbreviations, one statement per line, imports at the top, and no wildcard imports. A reader can then find where `load` came from at 3 a.m.
PEP 8 says to know when to break a rule. A 100-character SQL string is more readable unbroken, and formatters allow a configured line length for exactly that.
The answer most people give
"Style is personal preference, so we do not enforce it." Unenforced style is a tax on every reader. A formatter removes the debate entirely.
They’ll ask next
Which three tools would you add to a repository's pre-commit config first, and what does each catch?
