Strings, Regex & Text Normalization

In chapter 2 you joined records with a dict lookup. A lookup only works when the key is exactly the same. " Asha " and "Asha" are different keys.

Text from other systems is often messy. It has extra spaces, mixed case, odd characters and values packed into one line.

A string is a piece of text. In Python its type is str. This chapter shows how to clean strings so equal values become the same string.

Here are the customers, and the same five names as a made-up partner app sent them:

customers = [
    {"customer_id": 1, "name": "Asha", "country": "IN"},
    {"customer_id": 2, "name": "Ben", "country": "GB"},
    {"customer_id": 3, "name": "Carla", "country": "US"},
    {"customer_id": 4, "name": "Dev", "country": None},
    {"customer_id": 5, "name": "Omar", "country": "IN"},
]
partner_names = [" Asha ", "BEN", "carla\t", "Dev\xa0", "ＯＭＡＲ"]

id_by_name = {c["name"]: c["customer_id"] for c in customers}
print([id_by_name.get(name) for name in partner_names])

All five people are in both lists, but the lookup finds none of them. By the end of this chapter, one small function finds all five.

You will learn to:

1. Turn bytes into text, and know why the length can change.
2. Remove spaces and fold case.
3. Make look-alike characters the same, where Python can.
4. Cut a line into fields.
5. Find and change text with regular expressions.
6. Build one key function for every join.

A computer stores text as bytes. A byte is a number from 0 to 255.

Python shows bytes with a b in front, such as b'caf\xc3\xa9'. Each \x and two characters after it is one byte.

An encoding is the rule that turns bytes into letters. The most common rule is utf-8. Chapter 4 shows how to read files with the right encoding.

decode turns bytes into a str. encode turns a str back into bytes:

raw = b"caf\xc3\xa9"           # the utf-8 bytes of "café"
text = raw.decode("utf-8")
again = text.encode("utf-8")

print(text)
print(len(text), len(again))

So len() on text counts characters, and len() on bytes counts bytes. Never cut text at a byte position, or you can split one letter in two.

Sometimes the bytes are broken. Here the last byte of é is missing, so utf-8 cannot read them:

bad = b"caf\xc3"

print(bad.decode("utf-8", errors="replace"))
print(bad.decode("utf-8", errors="ignore"))
print(bad.decode("utf-8"))

The errors setting decides what happens to bad bytes. "replace" puts a � in their place, so you can still see the damage. "ignore" drops them without a trace.

Whitespace means characters you cannot see: spaces, tabs and new lines. strip() removes whitespace from both ends of a string:

print([name.strip() for name in partner_names])

Now "Asha" and "Dev" match, but "BEN" and "carla" still differ in case. casefold() makes a string lower case for comparing:

id_by_key = {c["name"].strip().casefold(): c["customer_id"]
             for c in customers}
print([id_by_key.get(name.strip().casefold())
       for name in partner_names])

Clean both sides in the same way. Here the customer names and the partner names both go through strip() and casefold().

Why casefold() and not lower()? casefold() is made for comparing, and it also folds letters that lower() keeps:

print("STRASSE".lower() == "straße".lower())
print("STRASSE".casefold() == "straße".casefold())

Spaces can also hide inside a value. split() with no argument cuts at every run of whitespace, and " ".join() puts the parts back with one space:

city = "  New   Delhi \t"
print(repr(" ".join(city.split())))

Unicode is the list of every character, with a number for each. That number is called a code point.

Some text looks the same but uses different code points. The letter é can be one code point, or an e plus an accent mark:

import unicodedata

one = "caf\u00e9"         # é as one code point
two = "cafe\u0301"        # e, then an accent mark

print(one == two, len(one), len(two))
print(unicodedata.normalize("NFC", one) ==
      unicodedata.normalize("NFC", two))

Normalization rewrites text into one standard form. unicodedata.normalize("NFC", text) joins a letter and its accent into one code point where it can.

NFKC does more. It also turns special forms into plain ones, such as wide letters into normal letters. That fixes the last partner name:

print(unicodedata.normalize("NFKC", "ＯＭＡＲ"))
print(unicodedata.normalize("NFKC", "ＯＭＡＲ").casefold() == "omar")

You cannot undo NFKC. Use it to build keys, and keep the original text for display.

A confusable is a character that looks like another one but is a different letter. Normalization does not fix these:

fake = "\u0410sha"         # starts with a Cyrillic letter

print(fake, fake == "Asha")
print(unicodedata.normalize("NFKC", fake) == "Asha")
print(unicodedata.name(fake[0]))

When two values look equal but do not match, print unicodedata.name() for each character. The hidden difference shows up.

A line of text often holds several fields with a separator between them. split(",") cuts the line at every comma:

line = "105,3,pending,,2026-01-20"
fields = line.split(",")

print(fields)
print(len(fields))

split keeps empty fields, so each field stays in its place. The fields are text, so "105" is not yet the number 105. Chapter 5, Types, Time, Nulls & Numeric Correctness, changes types.

Two more tools cut a line only once. split(sep, 1) cuts at the first separator. partition(sep) gives three parts: before, the separator, and after:

print("note=paid=late".split("=", 1))
print("db:5432".partition(":"))
print("db".partition(":"))
print(",".join(["105", "3", "pending"]))

split(",") breaks when a field itself holds a comma. A CSV file puts such a field in quotes:

line = '101,"Pen, Notebook",paid'
print(line.split(","))

Use split only for simple text that you trust. For CSV, use the csv module. Chapter 4, Files, Paths & Payload Ingestion, shows it.

A regular expression, or regex, is a pattern that describes text. Python has regex in the re module.

A few pattern parts are enough to start:

Write patterns as raw strings, with an r before the quote. In a normal string, Python may change a backslash, so "\b" becomes a backspace character.

Here are three made-up log lines. A log line is one line of text that a program writes about what it did:

import re

logs = [
    "2026-01-20 WARN order=105 status=pending",
    "2026-01-21 INFO order=106 status=paid",
    "2026-01-21 INFO health check ok",
]

print(re.match(r"order=(\d+)", logs[0]))
m = re.search(r"order=(\d+)", logs[0])
print(m.group(1))

re.match only looks at the start of the string. The line starts with a date, so match finds nothing. re.search looks everywhere in the string.

re.findall returns every match as a list:

print(re.findall(r"\d+", "order=105 qty=2"))

A named group gives each part a name: (?P<name>...). Then groupdict() returns a dict, which is a record like in chapter 2:

pattern = re.compile(
    r"(?P<day>\d{4}-\d{2}-\d{2}) (?P<level>[A-Z]+) order=(?P<order_id>\d+)"
)

for line in logs:
    m = pattern.search(line)
    if m:
        print(m.groupdict())
    else:
        print("no order:", line)

The third line has no order, so search returns None. The if m: check skips it.

re.sub(pattern, new, text) replaces every match. In the new text, \1 means "what group 1 matched", \2 means group 2, and so on.

A partner sends dates as day/month/year. This turns them into the shop's year-month-day form:

print(re.sub(r"(\d{2})/(\d{2})/(\d{4})", r"\3-\2-\1", "20/01/2026"))

To validate is to check that a value has the right shape. Here the whole value must match the pattern, not only a part of it.

re.fullmatch checks the whole value. re.search is happy with any part. Compare them on three values:

order_id = re.compile(r"\d{3}")

for value in ["105", "105x", "x105"]:
    print(value, bool(order_id.fullmatch(value)),
          bool(order_id.search(value)))

So use fullmatch to validate. Use search to find a part.

A regex checks the shape, not the meaning. "105" has the shape of an order id, but order 105 may not exist. A lookup in the orders checks that.

Now put the steps into one function. A key function turns a value into the form you compare and join on.

The steps go in this order: normalize with NFKC, then casefold, then collapse the whitespace:

import unicodedata

def key(value):
    value = unicodedata.normalize("NFKC", value)
    value = value.casefold()
    return " ".join(value.split())

id_by_key = {key(c["name"]): c["customer_id"] for c in customers}
print([id_by_key.get(key(name)) for name in partner_names])

split() with no argument also removes whitespace at both ends, so key() needs no strip().

A good key function gives the same result when you run it twice. Test that, and test a few messy values:

for value in partner_names:
    assert key(key(value)) == key(value)

print(key("  New   Delhi \t"), key("ＤＥＶ"))

Sometimes you need a key that is safe in a file name or a web address. A slug keeps only plain letters, digits and dashes:

def slug(value):
    value = unicodedata.normalize("NFKD", value)
    value = value.encode("ascii", "ignore").decode("ascii")
    return re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")

print(slug("Café Mug #2"))

Each exercise opens in the Python workspace with its own data. Write the code, run it, and check the result against the tests.

Each one has a trap in the text, not in the loop.