Garbled Characters? Only One Kind Is Repairable

Garbled customer names are not one bug. Four different failures produce text that looks wrong on screen, and they split on the only question that matters before you write a cleanup job: are the original bytes still sitting in the field, or are they gone? A value that reads José still contains every byte of José and can be repaired in place. A value that reads Jos followed by a replacement character has already discarded those bytes, and no script will bring them back. The advice you will find on the first page of search results, which is to save the file as CSV UTF-8 and import it again, is the correct fix for exactly one of the four.

Classify the damage first. It takes about a minute and it decides whether you are writing a repair script or asking the source system for a fresh export.

The four kinds of garbled text

What you see in the field	What happened	Original bytes still present	The actual fix
José, ü, ’, É	UTF-8 bytes were decoded as Windows-1252 or Latin-1	Yes, all of them	Re-encode and re-decode the stored string
A replacement character, drawn by most fonts as a black diamond with a question mark, or a literal ?	A decoder hit bytes it could not map and substituted a placeholder	No	Re-import from the source system
The value stops mid-word, or the write failed with an incorrect string value error	A four-byte character hit a three-byte column	No	Widen the column, then re-import
Nothing. It looks perfect and still fails an exact match	Two systems stored different Unicode normalization forms	Yes, and both spellings are valid	Normalize at comparison time, not in storage

Rows one and four are recoverable without touching the source. Rows two and three are not, and every hour spent writing a transformation for them is wasted. The distinguishing signal is visible in the data itself, which is why this is a triage step rather than an investigation.

Why one is repairable and the other is not

José in UTF-8 is five bytes: 4A 6F 73 C3 A9. A reader that has been told the file is Windows-1252 walks those bytes one at a time, renders C3 as a capital A with tilde and A9 as a copyright sign, and produces José. Nothing was deleted. The byte sequence is intact, and the only thing that went wrong is the interpretation layered on top of it. Encode the visible string back to Windows-1252 and decode the result as UTF-8, and you get José back exactly. That is why the fix is a transformation and not a restore.

The replacement character is the opposite operation. When a UTF-8 decoder meets a byte sequence that is not valid UTF-8, it does not preserve the byte for later inspection. It emits U+FFFD, the replacement character, and moves on. Whatever was there is now a single code point that means "something was here." Reading José as ASCII produces Jos followed by two replacement characters, and the two bytes that distinguished the accent are unrecoverable from the stored value. Restoring it means going back to whatever wrote it.

Here is the part that catches people midway through the repair, and it is worth checking before you run the script across a table. Windows-1252 does not define all 256 byte values. Five positions are unassigned: 0x81, 0x8D, 0x8F, 0x90, and 0x9D. The WHATWG Encoding Standard, which is what browsers implement, maps those five to C1 control characters so every byte decodes. Python's cp1252 codec does not, and neither do several database drivers, so a strict decoder either raises an error or, more commonly in an automation, substitutes a replacement character and carries on.

Exactly five uppercase Latin letters encode to UTF-8 sequences that contain one of those bytes: capital A with acute (C3 81), capital I with acute (C3 8D), capital I with diaeresis (C3 8F), capital eth (C3 90), and capital Y with acute (C3 9D). So a surname written Álvarez in all caps, or a first name of Íñigo, was already damaged at the moment the mojibake was created, while José in the same column was not. Run the two through a strict Windows-1252 decoder and the difference is stark: José comes back as José and repairs cleanly, while Álvarez comes back with a replacement character where the accent used to be and repairs to garbage.

The practical consequence is that a mojibake repair pass is never all-or-nothing. Expect a small residue of names beginning with those five letters that the script cannot fix, and pull those from the source rather than shipping whatever the transformation produced.

Where in an automation the damage happens, and one place it cannot

Encoding is only ambiguous where a format fails to declare it. That gives you a short list of suspects.

Hop	Ambiguous?	What to check
CSV export or import	Yes. A CSV carries no encoding declaration	The export setting, and whether a byte order mark is written
A file opened and re-saved in a spreadsheet	Yes. The save uses whatever the tool defaults to	Whether anyone opened the file to inspect it
An HTTP body sent as JSON	No	Nothing. Look upstream instead
A database read or write	Yes, at the column and connection level	Column character set and the connection charset
A file parse step in the workflow	Yes	Whether the step names an encoding or infers one

That third row is the useful one, because it eliminates a suspect rather than adding one. RFC 8259 states that JSON exchanged between systems that are not part of a closed ecosystem must be encoded using UTF-8, and the media type registration adds that no charset parameter is defined for application/json because adding one "really has no effect on compliant recipients." There is nothing for a receiver to guess. If mojibake arrives in a JSON webhook body, the sender serialized text that was already mojibake, and the damage happened in whatever the sender read from. Stop debugging the webhook and go one hop back. We have watched teams spend a week on connector settings for a problem that lived in a nightly CSV drop three systems away.

The database row is where the third failure class lives. A MySQL column declared as utf8 is not UTF-8. It is an alias for utf8mb3, which MySQL documents as requiring a maximum of three bytes per multibyte character and supporting BMP characters only, with no support for supplementary characters. Accented Latin letters are two bytes and fit fine, which is why the problem stays hidden for years. Anything that needs four bytes, which includes emoji and a large block of CJK ideographs, does not fit. Depending on the server mode you either get an incorrect string value error or a value truncated at the offending character. MySQL has deprecated utf8mb3 and states plainly that the recommended character set is utf8mb4 and all new applications should use it. If your stack has a MySQL column older than the migration, check what it is actually declared as before you conclude the automation mangled anything.

The fourth case: it looks correct and still will not match

This one does not look like an encoding bug at all, which is why it survives cleanup passes. Two records show the same name. Rendered side by side in a browser or a spreadsheet they are pixel identical. The automation's lookup step reports no match, the dedupe run leaves both, and the customer gets two invoices.

The cause is that Unicode gives you more than one legal way to write the same character. An e with an acute accent can be a single code point, U+00E9, or it can be a plain e followed by U+0301, a combining acute accent. Unicode Annex 15 calls these canonically equivalent and says such sequences "when correctly displayed should always have the same visual appearance and behavior." They do. The name José in composed form is four characters and five bytes. The same name in decomposed form is five characters and six bytes. Compare them with an equality operator and the answer is false, and it stays false after trimming whitespace and lowercasing both sides, because none of those operations touch the difference.

Which form you get depends on what typed the text. A form filled in on one operating system, a name pasted out of a PDF, and a value returned by an API that decomposes on the way out can all land in the same column looking identical and comparing unequal. It is the quietest source of the duplicate-customer problem, because every other cause leaves a visible difference you can spot in a spreadsheet and this one does not.

The fix is normalization, and the discipline is to do it on a copy. Build a match key next to the original: normalize to NFC, then apply whatever casing and whitespace rules your matching already uses, and compare on that field. PostgreSQL ships this directly, with a normalize() function that defaults to NFC and an IS NFC NORMALIZED predicate for testing, both available when the server encoding is UTF8. Leave the stored display value alone unless your system is the owner of that field. Rewriting it looks like an edit to anything watching the record, and a two-way sync will push your normalization pass back into a system that did not ask for it.

What to do first

Pull twenty affected records and sort them into the four rows of the first table before you write anything. That single step tells you whether this is a scripting job or a re-import request, and those are entirely different weeks of work. Look specifically for the residue case: if any of the damaged names begin with A-acute, I-acute, I-diaeresis, eth, or Y-acute, set them aside now, because a repair pass will produce plausible-looking wrong values for exactly those rows and you will not notice until a customer does.

Then close the boundary rather than scheduling a recurring cleanup. Name the encoding explicitly on every file parse step in the workflow, move machine-to-machine transfers off CSV and onto JSON where both systems have an API, and audit your database columns for a utf8 that is really utf8mb3. The related failure at the same boundary is type coercion, where a CSV import silently mangles IDs and dates through the same mechanism of an undeclared property being guessed at every hop, and the matching failure downstream is covered in why your CRM shows three of the same customer.

If you are staring at a table where you cannot tell which records are repairable and which need a fresh export, that is a data-quality audit, not a scripting problem, and it is worth doing before the cleanup rather than after. We do this as part of data intelligence work and inside workflow automation builds. Tell us what the damaged values look like and we can usually classify them from a screenshot. Send us a sample.