What You'll Master Here
You are a custodian, not just a mover. Know where personal data is, protect it in layers, control who can read it, and be able to find and erase one person on demand.
Pipelines carry the most sensitive data an organisation holds: names, emails, payment details, locations, sometimes health records. With that comes a duty — legal, ethical and reputational — to be a good custodian of it.
A hospital records room is the everyday version. The files are locked, there is a book recording who took what out, only the treating clinician may pull a chart, and a patient can demand to see their own file. There is also a rule people find surprising: some records must be kept for years even if the patient asks you to destroy them. Every one of those maps onto something in this chapter.
None of it is bolted on at the end. A pipeline that leaks personal data or cannot honour a deletion request is a serious liability however fast and reliable it is, and retrofitting the controls is far harder than designing them in.
By the end you will know the words the law actually uses, how to choose a protection per column and why hashing an email does not take it out of scope, what "encrypted at rest" does and does not buy you, how to write a real row and column policy, and how to trace one person’s data through twelve systems when they ask to be forgotten.
You are a custodian, not just a mover. Know where personal data is, protect it in layers, control who can read it, and be able to find and erase one person on demand.
A breach or a compliance failure costs money, trust and engineering time in that order, and instantly outweighs any efficiency gain. This is where data engineering meets real legal and human consequences.
- personal data
- Anything relating to an identifiable living person — including a customer id, a cookie id or an IP address. The legal term, and broader than engineers expect.
- PII
- Personally Identifiable Information — the American term, and narrower than "personal data". Worth knowing that the two are not synonyms when someone uses them as though they were.
- GDPR
- General Data Protection Regulation — the EU law (and its retained UK twin) that most of this chapter is about. It applies to anyone processing EU or UK residents’ data, wherever the company is based.
- data governance
- The policies and controls over how data is classified, accessed and used — a catalog, owners, classifications and an audit trail.
- compliance
- Meeting the obligations those laws impose, and being able to demonstrate that you met them.
- least privilege
- Granting every person and system the minimum access needed for its job, and no more.
Treating security and compliance as a final-step add-on. Personal data spreads unprotected through the pipeline, and the controls you eventually need have to be retrofitted across every copy.
Assuming "PII" and "personal data" mean the same thing. You protect names and emails, leave IP addresses and cookie ids untouched, and are surprised to learn they were regulated all along.
Design the controls in from the first day, not the last sprint.
Know where personal data lives and where it flows.
Default to least privilege for every person and every service account.
You are a custodian of other people’s information. It must be classified, protected in layers, access-controlled and accounted for — by design, because the cost of getting it wrong is measured in fines, trust and a year of retrofitting.
The next topic gives you the vocabulary — data subject, controller, processor, and what the acronyms stand for. It is short, and everything afterwards assumes it.
Pipelines are custodians of personal data; security, governance and compliance have to be designed in, because the legal, financial and trust costs of getting them wrong dwarf any efficiency gained by skipping them.
- Explain why security cannot be a final-step add-on to a pipeline.
- Give three examples of personal data that are not obviously "PII".