Using the Convert to Unicode transform

The Convert to Unicode (UTF-8) transform normalizes incoming plain-text files so every downstream transform sees UTF-8. Legacy systems commonly export CSVs as UTF-16 (frequently from Windows tooling) or in a regional single-byte encoding such as Windows-1252 — read as UTF-8, those files show up as mojibake (André instead of André) or fail validation outright.

By default the transform detects the encoding for you. New transforms are created with Auto-detect UTF-8/UTF-16 (recommended) selected.

What auto-detect recognizes

Auto-detect resolves the Unicode family, which can be identified from the bytes themselves with certainty:

When no conversion is needed the file passes through without transcoding, but it is not byte-identical to the input: the transform removes the BOM when present and strips NUL bytes from the content.

Auto-detect does not guess non-Unicode encodings. Windows-1252, Windows-1250, Windows-1257, Windows-1254, Shift_JIS, GBK, and the rest cannot be reliably told apart from bytes alone — the same byte sequence is valid, different-looking text in each of them, so guessing would silently corrupt data. If your files arrive in one of those encodings, pick it explicitly from the Source encoding dropdown; every encoding OneSchema supports is listed there, grouped by region.

Byte-order marks always win

If a file starts with a UTF-8 or UTF-16 BOM, that BOM determines how the file is read — even when the transform is pinned to a specific encoding. A file whose producer declared UTF-16 is never read as Windows-1252 because of a stale setting.

When a BOM overrides an explicitly selected encoding, the run records a warning naming the file, the encoding you selected, and the encoding actually used. The output is still produced; the warning simply tells you the transform's setting no longer matches reality and can probably be switched to auto-detect.

Configure the transform

1. In the MFF builder, click + on a transform and choose Convert to Unicode (UTF-8) under Extract & Convert.
2. Select the input files to convert. The transform accepts any number of text inputs and converts each one independently.
3. Leave Source encoding on Auto-detect UTF-8/UTF-16 (recommended), or choose a specific encoding if your files are not Unicode.
4. Optionally enable Write detection log file (see below).

The detection log

Write detection log file is off by default. When enabled, the transform emits an extra <name>.encoding.log output alongside each converted file, explaining how that specific file was resolved:

File: utf16le_input.csv
Detection method: utf16le_bom
Detected encoding: utf-16le
Configured encoding: auto
Passthrough: false
Replacement count: 0
Evidence: {bom: "FF FE"}

Replacement count is the number of replacement characters emitted during conversion. A non-zero value means some input bytes could not be converted using the selected encoding.

Leave the option off unless you are actively diagnosing an encoding problem: it adds an output per input, and a downstream transform configured to accept files broadly may pick the logs up as data.

Troubleshooting

"Could not determine the source encoding"

Auto-detect could not identify the file as UTF-8 or UTF-16, so the run fails rather than emitting corrupted text. The error names the file and lists byte contexts, each with its offset, raw hex bytes, and a printable rendering with the relevant byte bracketed. These contexts help diagnose the input, but they cannot reliably identify every legacy encoding from bytes alone.

Choose the matching encoding from the Source encoding dropdown. If you are unsure, confirm the encoding with the system that produced the file rather than guessing from the byte contexts.

Output still shows mojibake

If accented characters render as é-style sequences, the file was almost certainly read as UTF-8 when it is really single-byte. Set the encoding explicitly instead of relying on auto-detect — a single-byte file without a BOM is valid UTF-8 often enough that no detector can tell.

Characters replaced with �

The file did not fully match the encoding it was read as, and the unconvertible bytes were replaced. Check the run warnings for the replacement count, and confirm the selected encoding matches what the producing system actually exports.

UTF-32 files fail

UTF-32 is not supported. Convert to UTF-8 or UTF-16 before the file reaches OneSchema.