دہری سطریں ہٹانے والا

Whole lines, compared exactly

Two lines are duplicates only if every character matches. That sounds obvious and it is the source of nearly every disappointing result.

A trailing space makes two visually identical lines different. So does a capital letter, a curly quote where the other line has a straight one, or a non-breaking space pasted in from a website. The [msm_tool slug="remove-duplicate-lines"] will keep both, correctly, and it will look like it did nothing.

If a deduplication leaves entries you were certain were duplicates, clean the whitespace first and run it again. That fixes it most of the time.

The removed count includes blank lines

Empty lines are lines. If your text has ten blank lines scattered through it, nine of them are duplicates of the first and will be counted as removed.

That inflates the number and is not an error. If the count looks higher than the duplicates you expected, blank lines are usually the difference.

Order is preserved

The first occurrence of each line stays where it was, and later copies are removed. The list does not get sorted as a side effect.

That matters when order carries meaning — a sequence of steps, a chronological log, a priority list. If you want the result sorted, sort it separately afterwards.

Case

By default the comparison is exact, so Apple and apple are two different lines.

For a list of email addresses, tags or keywords that is usually wrong — they are the same entry typed inconsistently. For a list of identifiers or code, case may genuinely distinguish them. Decide which kind of list you have before deciding whether the result is correct.

Where this is genuinely useful

• Merged mailing lists. Two exports concatenated, with overlap between them.
• Keyword and tag lists assembled from several sources.
• Log files, where the same line repeating a thousand times obscures the ones that matter.
• URL lists for a redirect map or a crawl, where a duplicate entry means duplicate work.
• Finding out whether there are duplicates at all — the count answers that even if you discard the result.

What it cannot do

It compares whole lines and nothing else. So:

1. Near-duplicates survive. Two addresses differing by a postcode format, two names with a middle initial on one — both stay. Fuzzy matching is a different problem and a much harder one.
2. Duplicates by one field do not. A CSV where two rows share an email address but differ elsewhere is two different lines. Deduplicating by a column needs a spreadsheet or a script.
3. Multi-line records are not handled. If each entry spans several lines, comparing line by line will happily remove a shared line from the middle of two otherwise different records.

Keep the original

Removal is not reversible here, and the count alone will not tell you what went. Everything runs in your browser, so nothing is uploaded — and nothing can be recovered either.