Text copied out of a web page, a word processor or a PDF frequently carries characters that look like spaces but are not: non-breaking spaces, the ideographic space used in CJK typesetting, zero-width spaces and joiners, and the byte-order mark that leads many exported files. They break searching, comparison and code, and they are invisible while you look for them. Each one found is either turned into an ordinary space or, where it has no width at all, removed outright, and the number found is reported.
The remaining operations run in a fixed order: invisible characters first, then tabs expanded to a chosen number of spaces between one and eight, then runs of spaces and tabs collapsed to a single space, then leading and trailing whitespace trimmed from each line, then blank lines dropped. Collapsing and trimming never touch line breaks, so the shape of the document is only changed by the last option.
Points worth knowing before you run it
-
The reported figure is a net difference:
expanding tabs adds characters, so a document full of tabs can finish longer than it started and the count will reflect that rather than pretending otherwise.
-
Zero-width characters are deleted, not replaced:
they had no width to begin with. Non-breaking and other exotic spaces become ordinary spaces, since something was meant to be there.
-
Code needs care:
collapsing runs and trimming lines will flatten indentation. For source files, expand tabs and clean invisible characters, and leave the other two alone.
-
Line breaks survive everything except the last option:
only dropping blank lines changes how many lines you end up with. Everything else works within the lines you already have.
The character count can go up
That surprises people, and it is correct.
Expanding a tab into four spaces adds three characters. If your text has more tabs than it has runs of redundant spaces, cleaning it makes it longer. The tool reports the net difference honestly rather than assuming the number must fall.
The [msm_tool slug="whitespace-cleaner"] does several distinct jobs, and it is worth knowing which you actually want.
The operations, and when each matters
- Trailing whitespace — spaces at the end of a line. Invisible in almost every editor, and the single most common cause of a diff showing changes you cannot see. Removing it is nearly always safe and nearly always worth doing.
- Leading whitespace — indentation. Safe in prose. Destructive in Python, YAML and Markdown code blocks, where indentation is syntax.
- Multiple spaces to one — tidies the double space after a full stop and the accidental triple space mid-sentence.
- Blank lines — collapsing runs of empty lines to one. Useful on text pasted out of a PDF; destructive on Markdown, where a blank line separates paragraphs.
- Tabs to spaces — or the reverse. A preference in code, and it should match whatever the project already does.
Where whitespace is meaningful
Run the wrong operation on the wrong content and the result is broken rather than tidy:
- Python. Indentation defines block structure. Stripping it destroys the program.
- YAML. Same — nesting is expressed entirely through indentation, and a valid file with the wrong indentation means something different.
- Markdown. Four leading spaces marks a code block; two trailing spaces force a line break; a blank line separates paragraphs. All three are whitespace that carries meaning.
- Fixed-width data. Column-aligned text where position is the delimiter.
- Poetry and formatted quotations, where the line shape is the point.
Invisible characters that are not spaces
The awkward ones. Text copied from a website or a word processor frequently carries characters that look like spaces and are not:
- Non-breaking spaces — common in text pasted from HTML or Word. They look identical to a normal space and behave differently, and they break string comparisons and searches.
- Zero-width characters — invisible entirely, and a known source of mystery bugs in identifiers, passwords and API keys.
- Other Unicode spaces — en space, em space, thin space, all of which display as a gap and are not the space character.
If two strings look identical and refuse to match, one of these is usually why.
The usual reasons to clean
- Text pasted from a PDF, which arrives with line breaks at every visual line and inconsistent spacing throughout.
- Content moving between a word processor and a CMS, which carries non-breaking spaces and stray formatting.
- Preparing text for comparison, where invisible differences would otherwise dominate the result.
- Tidying a code file before committing, where trailing whitespace produces noise in every future diff.
Work on a copy
Cleaning is not reversible, and the operations that are destructive are destructive in ways that are not obvious until later — a YAML file that still parses but nests differently, a Markdown document whose code blocks became paragraphs. Keep the original until you have checked the result.