Loadingone moment…
Sign in to run and submit your work
Reading is open to everyone. Running code and saving drafts need an account so your work is yours and comes back on your next visit.
or
CODE WORKSPACE
A streaming job writes a file every minute, so each daily partition ends up with hundreds of files of a few hundred kilobytes. Query planning spends longer listing them than the scan takes. The compaction job rewrites groups of small files into one, and it needs a plan first.
Write plan_compaction(files, target_bytes, max_files). Return the groups, each saying whether it is worth rewriting.
Function to write
plan_compaction(files: list[dict], target_bytes: int, max_files: int) -> list[dict]One group per rewrite unit, each with its files, total bytes and whether to compact or keep.
How to approach it
Separate the already-large files first, then pack the rest largest-first into open groups.
Sample cases
+ 2 held back until you submit
a day of small files
Five files to pack, one already above the target, and one entry with no path at all.
Input
Argument 1
| path | bytes |
|---|---|
| part-0001.parquet | 400 |
| part-0002.parquet | 700 |
| part-0003.parquet | 250 |
| part-0004.parquet | 5000 |
| part-0005.parquet | 300 |
| part-0006.parquet | 100 |
| 100 |
Argument 2
1000Argument 3
3Returns
[
{
'files': [
'part-0001.parquet',
'part-0003.parquet',
'part-0006.parquet'
],
'bytes': 750,
'action': 'compact'
},
{
'files': [
'part-0002.parquet',
'part-0005.parquet'
],
'bytes': 1000,
'action': 'compact'
},
{
'files': [
'part-0004.parquet'
],
'bytes': 5000,
'action': 'keep'
}
]an empty partition
No files means no plan rather than one empty group.
Input
Argument 1
[] (empty list)
Argument 2
1000Argument 3
3Returns
[] (empty list)
every file is already large
Nothing to compact, so both files are kept as they are and neither is grouped.
Input
Argument 1
| path | bytes |
|---|---|
| a.parquet | 2000 |
| b.parquet | 3000 |
Argument 2
1000Argument 3
5Returns
[
{
'files': [
'a.parquet'
],
'bytes': 2000,
'action': 'keep'
},
{
'files': [
'b.parquet'
],
'bytes': 3000,
'action': 'keep'
}
]Constraints
target_bytes is already large enough. It becomes a group of its own with the action keep, and is never combined with anything.target_bytes and no group holds more than max_files files.keep rather than compact. Rewriting one file costs the same as rewriting several and achieves nothing.compact, and reports the total bytes it would produce.Worked example
With a target of 1000 bytes and at most 3 files per group, the small files sorted largest first are 700, 400, 300, 250 and 100. The 700 opens a group; the 400 does not fit beside it, so it opens the second; the 300 fits with the 700 to make 1000 exactly; the 250 joins the 400; the 100 joins the first group, filling it to three files.
part-0004.parquet is 5000 bytes, already five times the target. Including it in the packing would consume a whole group's budget and produce a rewrite of a file that was fine — compaction costs a full read and write of every byte in the group, so the only files worth touching are the ones whose smallness is the problem.
What this tests
A bin-packing decision with two independent limits, and the judgement to leave work undone. First-fit-decreasing is not optimal and is deterministic and good enough, which is the right trade when the alternative is an exact solver.
plan_compaction(files: list[dict], target_bytes: int, max_files: int) -> list[dict]Submit for review to find out what your query gets right, what it gets wrong, and how it compares with the best working query for this exercise.
This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.