The Reference: Every Box on the Canvas
Every tool is a trade between money, attention and lock-in. There is no best; there is only what fits the team, the budget and the requirement you actually have.
This chapter is a lookup, not a reading. Every named technology on the pipeline-builder canvas has an entry here: what it actually is, when to reach for it, what it is good at, what it costs you in money and in attention, how the bill is shaped, and what to compare it against.
It is arranged the way the canvas is — sources and ingestion, storage and lakehouse, processing, orchestration and contracts, serving and intelligence — so you can open the family you are working in and search inside it while you design.
Read the two topics after this one first. The chooser narrows seventy-two options to a shortlist in five questions, and the pricing shapes tell you which of six ways a tool bills you — which is the thing you must know before comparing any two of them.
Every tool is a trade between money, attention and lock-in. There is no best; there is only what fits the team, the budget and the requirement you actually have.
Design decisions get made in meetings, from memory, under time pressure. Having the trade-offs and the shape of the bill in one place is what stops "we chose it because we knew it" being the real reason behind an architecture.
- list price
- The published, undiscounted price. Every figure in this chapter is one. Committed-use discounts, enterprise agreements and region multipliers routinely move real invoices by 30–60%.
- operational surface area
- How many systems somebody has to keep healthy. The most under-counted cost in any architecture, and the one that decides whether a stack survives its second year.
- total cost of ownership (TCO)
- Licence plus infrastructure plus the engineer-hours to run it. "Free" software operated by a person who has another job is frequently the most expensive option available.
- lock-in
- What leaving would cost. Not a reason to avoid a tool — a reason to know the number before you commit, because it is always larger than it looks.
- managed vs self-hosted
- The same capability, bought either with money or with attention. Which is cheaper depends entirely on how many engineers you have, not on the software.
Comparing two tools without first comparing their billing shapes. A per-byte-scanned tool and a per-compute-hour tool cannot be compared on unit price at all. You have to model your own workload under each.
Treating open-source as free. It has no licence cost and a real operating cost. At one engineer that trade is usually bad; at fifteen it is usually good. The software did not change — the team did.
Quoting a figure from this chapter in a procurement conversation. These are list prices from a point in time. Use them to sanity-check an order of magnitude, then get a real quote.
Open the family you are designing in and search it before naming a tool from memory.
Ask which of the six billing shapes a tool uses, then ask whether your workload does the thing that shape punishes.
Count the systems somebody will have to operate, and treat that count as a cost.
Every price here is a published list price in USD for a common region, checked at the time of writing, and they move. Where a vendor does not publish a number, this chapter says "quote-based" rather than inventing one — because that is itself a useful fact about a product. Sanity-check magnitudes with these; never budget from them.
Every entry here is the right answer for somebody. The Postgres-and-cron stack and the Kafka-and-Flink stack are both correct architectures for the companies they belong to, and both are catastrophic in the other’s place. The chapter is arranged to help you find which one you are.
Being able to name a tool is worth little. Being able to say what it bills you for, what that punishes, and what you would compare it against is worth a great deal — and it is the difference between having read a landscape diagram and having paid an invoice.
A reference to every technology on the canvas, arranged by family, with usage, strengths, real costs and billing shape — designed to be opened mid-design rather than read once.
- Open the family you are currently working in and read the entry for the tool you already use. Check whether the cons surprise you.
- Name the billing shape of the three most expensive tools on your platform.