Loadingone moment…
Sign in to run and submit your work
Reading is open to everyone. Running code and saving drafts need an account so your work is yours and comes back on your next visit.
or
CODE WORKSPACE
The pipeline reads a schema from the source system on every run. When the schema changes, some changes are harmless and some silently break every query downstream — and today the pipeline treats them all the same way, which is to say it ignores them.
Write detect_schema_drift(previous, current). Return every difference and whether it breaks a reader.
Function to write
detect_schema_drift(previous: dict, current: dict) -> dictA dict with changes — each naming the field, the change, both values and whether it breaks — and an overall breaking flag.
How to approach it
Normalise both schemas into the same shape first; then it is a walk over the union of the field names.
Sample cases
+ 2 held back until you submit
a month of upstream changes
A widening, a real type change, a dropped column, a tightened null rule, and two kinds of new column.
Input
Argument 1
{
'order_id': {
'type': 'string',
'nullable': False
},
'amount': {
'type': 'int',
'nullable': True
},
'region': 'string',
'legacy_flag': {
'type': 'int',
'nullable': True
},
'note': {
'type': 'string',
'nullable': True
}
}Argument 2
{
'order_id': {
'type': 'string',
'nullable': False
},
'amount': {
'type': 'float',
'nullable': True
},
'region': {
'type': 'int',
'nullable': True
},
'note': {
'type': 'string',
'nullable': False
},
'channel': {
'type': 'string',
'nullable': True
},
'tax_id': {
'type': 'string',
'nullable': False
}
}Returns
{
'changes': [
{
'field': 'amount',
'change': 'type_widened',
'from': 'int',
'to': 'float',
'breaking': False
},
{
'field': 'channel',
'change': 'added',
'from': None,
'to': 'string',
'breaking': False
},
{
'field': 'legacy_flag',
'change': 'removed',
'from': 'int',
'to': None,
'breaking': True
},
{
'field': 'note',
'change': 'nullable_tightened',
'from': True,
'to': False,
'breaking': True
},
{
'field': 'region',
'change': 'type_changed',
'from': 'string',
'to': 'int',
'breaking': True
},
{
'field': 'tax_id',
'change': 'added',
'from': None,
'to': 'string',
'breaking': True
}
],
'breaking': True
}nothing changed
An identical schema produces no changes and is not breaking.
Input
Argument 1
{
'order_id': {
'type': 'string',
'nullable': False
},
'amount': {
'type': 'int',
'nullable': True
},
'region': 'string',
'legacy_flag': {
'type': 'int',
'nullable': True
},
'note': {
'type': 'string',
'nullable': True
}
}Argument 2
{
'order_id': {
'type': 'string',
'nullable': False
},
'amount': {
'type': 'int',
'nullable': True
},
'region': 'string',
'legacy_flag': {
'type': 'int',
'nullable': True
},
'note': {
'type': 'string',
'nullable': True
}
}Returns
{
'changes': [],
'breaking': False
}the first version
Every field is new, so the non-nullable ones are breaking and the rest are not.
Input
Argument 1
{}Argument 2
{
'order_id': {
'type': 'string',
'nullable': False
},
'amount': {
'type': 'float',
'nullable': True
},
'region': {
'type': 'int',
'nullable': True
},
'note': {
'type': 'string',
'nullable': False
},
'channel': {
'type': 'string',
'nullable': True
},
'tax_id': {
'type': 'string',
'nullable': False
}
}Returns
{
'changes': [
{
'field': 'amount',
'change': 'added',
'from': None,
'to': 'float',
'breaking': False
},
{
'field': 'channel',
'change': 'added',
'from': None,
'to': 'string',
'breaking': False
},
{
'field': 'note',
'change': 'added',
'from': None,
'to': 'string',
'breaking': True
},
{
'field': 'order_id',
'change': 'added',
'from': None,
'to': 'string',
'breaking': True
},
{
'field': 'region',
'change': 'added',
'from': None,
'to': 'int',
'breaking': False
},
{
'field': 'tax_id',
'change': 'added',
'from': None,
'to': 'string',
'breaking': True
}
],
'breaking': True
}Constraints
type and optional nullable, or to a bare type name. A bare name means nullable.type_widened and not breaking when it is one of the known safe widenings: int to bigint, int to float, float to double, string to text. Any other type change is type_changed and breaking.field, change, from, to and breaking, ordered by field name. Also return the overall breaking flag.Worked example
amount goes from int to float. Every value that fit before still fits, so it is a widening and nothing downstream breaks. region goes from string to int in the same release, which is not a widening at all: the values themselves change meaning, and a query comparing it to a string starts returning nothing rather than failing.
The two new fields differ in exactly one respect. channel is nullable, so the rows already loaded can carry a null and nothing has to be backfilled. tax_id is not, which means the table cannot be reloaded without inventing a value for every historical row — a migration, not a schema update. Reporting both as simply 'added' loses the only distinction that changes what anyone does next.
What this tests
Turning a comparison into a classification, and knowing which schema evolutions are safe in a columnar store and which are not. The knowledge is the exercise; the set difference is the easy part.
detect_schema_drift(previous: dict, current: dict) -> dictSubmit for review to find out what your query gets right, what it gets wrong, and how it compares with the best working query for this exercise.
This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.