Text Tools · runs in your browser

Duplicate Line Remover

Strip repeated lines from a list, keeping the first or the last of each.

No sign-up No file limit Nothing uploaded

Duplicate Line Remover

How to use 3 steps
  1. Paste your Lines.
  2. Choose which copy to Keep and the Output order, plus the space and blank-line options.
  3. Press Remove duplicates, then Copy.
Unique lines

How this tool works

Each line is reduced to a comparison key and the keys are tracked as the list is walked. A line whose key has been seen before is a duplicate. The key is what the options change: trimming ignores leading and trailing spaces, and ignoring case folds everything to lower case before comparing. The line that is kept is always written out exactly as you typed it — the options decide what counts as the same, never what the output looks like.

Keeping the first occurrence preserves your original order. Keeping the last moves each surviving line down to the position of its final appearance, which matters when the later copy is the corrected one. Sorting alphabetically, if you ask for it, happens after de-duplication and uses locale-aware comparison, so accented characters land where a reader of that language would expect rather than after Z.

Details that explain a surprising count

  • Comparison is whole-line: two lines that differ by a single trailing space are different lines unless trimming is on. This is the usual reason a list that looks de-duplicated still has pairs in it.
  • Dropped blank lines are counted as removed: the removed figure is the difference between the lines you pasted and the lines you got back, so discarding empty lines inflates it beyond the number of true duplicates.
  • Keeping the last occurrence changes positions: the earlier copy disappears and the survivor stays where the last one was. Choose it when a list has been appended to and the newest entry wins.
  • Sorting is optional and applied last: leave it off and the surviving lines stay in the order you pasted them, which is what you want for logs and anything chronological.

Whole lines, compared exactly

Two lines are duplicates only if every character matches. That sounds obvious and it is the source of nearly every disappointing result.

A trailing space makes two visually identical lines different. So does a capital letter, a curly quote where the other line has a straight one, or a non-breaking space pasted in from a website. The [msm_tool slug="remove-duplicate-lines"] will keep both, correctly, and it will look like it did nothing.

If a deduplication leaves entries you were certain were duplicates, clean the whitespace first and run it again. That fixes it most of the time.

The removed count includes blank lines

Empty lines are lines. If your text has ten blank lines scattered through it, nine of them are duplicates of the first and will be counted as removed.

That inflates the number and is not an error. If the count looks higher than the duplicates you expected, blank lines are usually the difference.

Order is preserved

The first occurrence of each line stays where it was, and later copies are removed. The list does not get sorted as a side effect.

That matters when order carries meaning — a sequence of steps, a chronological log, a priority list. If you want the result sorted, sort it separately afterwards.

Case

By default the comparison is exact, so Apple and apple are two different lines.

For a list of email addresses, tags or keywords that is usually wrong — they are the same entry typed inconsistently. For a list of identifiers or code, case may genuinely distinguish them. Decide which kind of list you have before deciding whether the result is correct.

Where this is genuinely useful

  • Merged mailing lists. Two exports concatenated, with overlap between them.
  • Keyword and tag lists assembled from several sources.
  • Log files, where the same line repeating a thousand times obscures the ones that matter.
  • URL lists for a redirect map or a crawl, where a duplicate entry means duplicate work.
  • Finding out whether there are duplicates at all — the count answers that even if you discard the result.

What it cannot do

It compares whole lines and nothing else. So:

  1. Near-duplicates survive. Two addresses differing by a postcode format, two names with a middle initial on one — both stay. Fuzzy matching is a different problem and a much harder one.
  2. Duplicates by one field do not. A CSV where two rows share an email address but differ elsewhere is two different lines. Deduplicating by a column needs a spreadsheet or a script.
  3. Multi-line records are not handled. If each entry spans several lines, comparing line by line will happily remove a shared line from the middle of two otherwise different records.

Keep the original

Removal is not reversible here, and the count alone will not tell you what went. Everything runs in your browser, so nothing is uploaded — and nothing can be recovered either.

Frequently Asked Questions

Specific to the Duplicate Line Remover. Broader questions about this category are answered on the category page.

No. Case folding is used only to decide what counts as a duplicate. Whichever line survives is printed exactly as you typed it.
Both leave one copy. The first keeps your original ordering; the last moves the survivor to where the final occurrence appeared.
Blank lines that were discarded are included in that figure. Untick the option to drop them and the numbers will line up.
No. Order is preserved unless you choose alphabetical sorting, which is applied after duplicates have been removed.
Not directly — comparison is line by line. Put one word per line first and the same de-duplication applies.

Subscribe to receive notifications about new tools, features, and productivity tips.