Open a file saved as UTF-8 in a tool that assumes a different encoding and café turns into café. The bytes on disk did not change. Only the rule used to turn those bytes back into characters did, and the two rules disagree about what byte 0xC3 means.

Most “weird characters” bugs come from the same mismatch: broken filenames, mangled form submissions, emoji that crash a database insert, two strings that look identical but fail an equality check. Behind each one is the same confusion between two separate things, Unicode and UTF-8.

Unicode vs UTF-8: code points are not bytes

Unicode is a single table that assigns every character a number, called a code point, written U+ followed by hex: A is U+0041, é is U+00E9, 😀 is U+1F600. The table runs from U+0000 to U+10FFFF. That covers every script in current and historical use, plus symbols, emoji, and ranges reserved for private use.

Unicode says nothing about bytes. It answers one question: which number does this character have? Turning that number into bytes you can store in a file or send over a socket is a separate job, done by an encoding. UTF-8, UTF-16 and UTF-32 are three encodings of the same table. They write the same code points as different byte sequences.

It’s the same kind of gap that makes 0.1 + 0.2 return something other than 0.3: the abstract value and the bits sitting in memory are two different layers, and the bug shows up the moment you treat them as interchangeable.

How UTF-8 turns a code point into bytes

UTF-8 is variable-width. Each code point becomes 1 to 4 bytes, using only as many as it needs.

Code point rangeBytes usedByte pattern
U+0000–U+007F10xxxxxxx
U+0080–U+07FF2110xxxxx 10xxxxxx
U+0800–U+FFFF31110xxxx 10xxxxxx 10xxxxxx
U+10000–U+10FFFF411110xxx 10xxxxxx 10xxxxxx 10xxxxxx

Every byte that starts with 10 is a continuation byte: a piece of a multi-byte character, never a character on its own. This makes the encoding self-synchronizing. Jump into the middle of a UTF-8 stream, scan backward past any continuation bytes, and you land on the start of a character.

The first row is why UTF-8 won. Code points U+0000–U+007F are exactly ASCII, stored as the same single byte ASCII already used, so any plain ASCII file is already valid UTF-8, byte for byte. Everything outside that range costs extra bytes. é is U+00E9, in the second row, so it takes two:

1
2

>>> "é".encode("utf-8")
b'\xc3\xa9'

😀 is U+1F600, in the fourth row, so it takes four:

1
2

>>> "😀".encode("utf-8")
b'\xf0\x9f\x98\x80'

That 0xC3 0xA9 pair is the culprit from the opening example. Decode it as UTF-8 and you get é. Decode it as Latin-1, which maps every byte to one character, and 0xC3 is à while 0xA9 is ©. Nothing in a raw byte stream records which encoding produced it. The bytes are not wrong. The reader’s assumption about them is.

Why string .length doesn’t count characters

“How long is this string” has three different answers, because “character” can mean three different units.

Python’s len() counts code points:

1
2
3
4

>>> len("😀")
1
>>> len("😀".encode("utf-8"))
4

One code point on the str, four bytes on the encoded bytes. Two different questions, two different answers.

JavaScript strings are UTF-16 internally, and .length counts 16-bit code units, not code points. Code points above U+FFFF, which includes every emoji, need two UTF-16 units, called a surrogate pair:

1
2

"😀".length;              // 2  (a surrogate pair, not one character)
[..."😀"].length;         // 1  (spread iterates by code point)

The family emoji "👨‍👩‍👧‍👦" makes it worse. It is 7 code points: 4 person emoji glued together by 3 zero-width joiners. Each of the 4 emoji is a surrogate pair, so .length reports 11. What a reader sees as one character is a grapheme cluster, and it can span any number of code points. Slice or reverse a string by code unit or by code point and you can cut that emoji in half. Intl.Segmenter counts the way a person would:

1
2

const graphemes = [...new Intl.Segmenter().segment("👨‍👩‍👧‍👦")];
graphemes.length;  // 1

Variable width also has a cost. In a UTF-8 buffer, finding the 50th code point means walking from the start and counting, so indexing by character position is O(n), not the constant-time lookup you get from a fixed-width array. Languages handle this differently. Rust and Go store strings as UTF-8 and index them by byte, so you iterate when you want characters. CPython stores each str at a fixed width per string (1, 2 or 4 bytes per code point, depending on the widest character it contains), which keeps s[i] constant-time at the price of extra memory once a single emoji shows up.

Common UTF-8 encoding bugs in production

Truncating by byte count. Cut a UTF-8 string at exactly 100 bytes and you will sometimes land in the middle of a multi-byte character, leaving a dangling partial sequence that fails to decode. Truncate by code point count instead, or truncate the bytes and decode with errors="ignore", accepting that the last partial character gets dropped. Neither approach respects grapheme clusters: a code-point cut can still separate e from its combining accent or break a family emoji apart. For user-visible text, cut on grapheme boundaries.

Leaving the encoding implicit on file I/O. open("data.txt") without an encoding= argument uses the operating system’s locale encoding. On Linux and macOS that is usually UTF-8. On Windows it has historically been a code page like cp1252. Read a UTF-8 file with the wrong default and you get a UnicodeDecodeError or silent mojibake, depending on the bytes. PEP 686 makes UTF-8 the default everywhere starting with Python 3.15, but plenty of code still runs on earlier versions or sets PYTHONUTF8=0. Pass the encoding explicitly and the question never comes up:

1
2

with open("data.txt", encoding="utf-8") as f:
    text = f.read()

A MySQL column that’s UTF-8 in name only. MySQL’s utf8 character set (now an alias for utf8mb3) stores at most 3 bytes per character. It cannot hold code points above U+FFFF, so no emoji and nothing outside the Basic Multilingual Plane. Insert 😀 into a utf8 column and you get Incorrect string value. The fix is utf8mb4, MySQL’s full UTF-8 implementation and the default since MySQL 8.0. Converting a table is ALTER TABLE t CONVERT TO CHARACTER SET utf8mb4, run per table, and the connection charset has to be utf8mb4 too or the client mangles emoji before they reach the column. On older InnoDB row formats, watch the 767-byte index prefix limit: an indexed VARCHAR(255) fits in utf8mb3 (765 bytes) but not in utf8mb4 (1,020 bytes).

Two byte sequences for the same visible character. é can be the single precomposed code point U+00E9 (NFC form), or the letter e (U+0065) followed by a combining acute accent U+0301 (NFD form). They render identically and differ byte for byte, so two café strings can compare unequal depending on where each was typed or pasted from. macOS filenames and text copied from some PDFs often arrive in NFD. A dictionary keyed by one form silently misses a lookup with the other. Normalize before you compare or store:

1
2
3
4
5
6
7

>>> import unicodedata
>>> a = "café"      # precomposed é (NFC)
>>> b = "café"     # e + combining acute accent (NFD)
>>> a == b
False
>>> unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b)
True

A stray byte-order mark. A UTF-8 file can start with the 3 bytes EF BB BF, a BOM written by some editors (older Windows Notepad among them) to flag the encoding. UTF-8 has no byte order to mark, so it carries no information. It is invisible in most editors, but decoding with plain utf-8 leaves a  character at the start of the string, which breaks strict JSON parsers and keeps a #! shebang from being recognized. Decode with Python’s utf-8-sig codec when you expect one, and don’t write one.

How to stop encoding bugs at the boundary

Pick UTF-8 and state it explicitly everywhere text enters or leaves your program: file reads, database connections, HTTP bodies, subprocess pipes. Decode to a string as soon as the bytes arrive, work on decoded strings internally, and encode back to bytes only at the next boundary. Keep errors="strict" (Python’s default) so a bad byte sequence raises an exception instead of quietly becoming U+FFFD or disappearing.

Then test with input that isn’t ASCII. Fixtures that only contain U+0000–U+007F let encoding bugs reach production, because every code path works by accident at one byte per character until a real user’s name breaks that assumption. Add two cases to whichever test level you already write for any text field, search box, filename handler or export: one string with an accented letter, written in both NFC and NFD, and one with the family emoji. If those survive a round trip through your storage and come back equal, the boundaries are doing their job.