Remove Duplicate Lines

Clean up lists in seconds

Mailing lists exported from two systems, keyword lists, product codes, log lines, attendee names: lists collect duplicates. Paste the list with one item per line, and the tool keeps the first occurrence of each line, removes the repeats, and tells you how many it removed. The original order is kept, so the list still makes sense.

Case-sensitive or not?

With case-sensitive matching, “Apple” and “apple” are different lines and both are kept. Turn it off to treat them as the same; the first version you wrote is kept. For email addresses, usernames and tags you will usually want case-insensitive matching.

Why some “duplicates” are not removed

• Trailing spaces. “apple” and “apple ” (with a space at the end) are different strings. Run the list through the Text Cleaner with “Trim extra spaces” first.
• Different punctuation, for example “Smith, J.” versus “Smith J”.
• Look-alike characters, such as a non-breaking space copied from a web page, or curly versus straight quotes.

Common workflows

1. Paste data copied from a spreadsheet column (each cell becomes a line).
2. Clean spaces with the Text Cleaner if the data came from several sources.
3. Remove duplicates here.
4. Optionally sort the result with Sort Text Lines, then copy it back.