Preparing your workspaceloading the brief, studio & editor…

Loadingone moment…

Deduplicate a stream in bounded memory · Python scenario · D8LooP

Reading is open to everyone. Running code and saving drafts need an account so your work is yours and comes back on your next visit.

CODE WORKSPACE

Deduplicate a stream in bounded memory

An ingestion topic redelivers events, but only ever within a short retry horizon: a duplicate always lands within window events of the original. The same key seen again after that horizon is a genuinely new event — a user clicking the same button an hour later — and dropping it loses real data. Write dedupe_within_window(events, window). Return the events that survive, in input order.

Function to write

dedupe_within_window(events: list[dict], window: int) -> list[dict]

The surviving events, in input order.

How to approach it

You need to know what left the window, not only what entered it.

Sample cases

+ 2 held back until you submit

a stream that comes back

Key 'a' repeats inside the window at seq 3, returns legitimately at seq 7, and repeats again at seq 10.

Input

Argument 1

seq	key
1	a
2	b
3	a
4	c
5	d
6	e
7	a
8	e
9	f
10	a

Argument 2

Returns

seq	key
1	a
2	b
4	c
5	d
6	e
7	a
9	f

a window of one

Only an immediately consecutive repeat is a duplicate — here there are none.

Input

Argument 1

seq	key
1	a
2	b
3	a
4	c
5	d
6	e
7	a
8	e
9	f
10	a

Argument 2

Returns

seq	key
1	a
2	b
3	a
4	c
5	d
6	e
7	a
8	e
9	f
10	a

the same key throughout

One key, six events, window of two: only the first survives.

Input

Argument 1

seq	key
1	a
2	a
3	a
4	a
5	a
6	a

Argument 2

Returns

seq	key
1	a

Constraints

1An event is a duplicate when its key appeared among the previous `window` events of the input, whether or not those events were kept.
2A key that reappears after more than window events is not a duplicate. Keep it.
3Keep the first occurrence and drop the duplicates, preserving input order.
4The stream is unbounded in production, so memory must stay proportional to window — not to the number of events seen.
5A window of 0 or less disables deduplication: every event survives.
6Return the original event dicts, not copies or keys.

Worked example

With window 3: key a at seq 1 is kept. At seq 3 it is a duplicate — seq 1 is still inside the window of the previous three events — so it is dropped. At seq 7 the previous three events are seq 4, 5 and 6, none of them a, so seq 7 is kept as a new occurrence. At seq 10 the previous three include seq 7, so it is dropped again. A set of every key ever seen is the obvious answer, and it drops seq 7 — silently losing a real event. It is also the version that runs out of memory on a stream that never ends, which is the failure you cannot see in a test.

What this tests

Recognising that 'remember everything' is a design decision, not a default — and that on an unbounded stream it is both a memory leak and, once the window is part of the specification, a correctness bug.

PYTHONdedupe_within_window(events: list[dict], window: int) -> list[dict]

AI evaluation

The reviewer reads your query.

Submit for review to find out what your query gets right, what it gets wrong, and how it compares with the best working query for this exercise.

Desktop only

This scenario runs a full workspace — editor, canvas and results side by side. It needs a laptop or desktop to be usable. Open this page on a bigger screen to start building.

Read the library instead Back to modules