Remove Duplicate Lines

Paste a list and drop repeated lines while the first copy of each stays in its original position. Exact matching, nothing sorted or uploaded.

The interactive tool loads here when JavaScript runs.

About this tool

What is Remove Duplicate Lines?

Remove Duplicate Lines keeps the first occurrence of every line and deletes later repeats. The surviving lines stay in the order they first appeared, so a list of email addresses, URLs or SKUs keeps its shape; nothing is sorted.

Matching is exact. "apple", "Apple" and "apple " (with a trailing space) are three different lines. Repeated blank lines collapse to a single blank line, because an empty line is itself a value. If your list has stray spaces or mixed case, clean it first or use Find Duplicate Entries, which has trim and case options.

The work happens in your browser with a JavaScript Set, which handles lists of hundreds of thousands of lines in well under a second on a typical laptop.

  • Keeps the first occurrence and its original position
  • Exact, case-sensitive comparison with no hidden normalisation
  • Handles very long lists quickly
  • Copy and Clear buttons
  • Runs in your browser; nothing is uploaded
Read the guide: Cleaning Up Text Lists: Spaces, Blank Lines, Duplicates and Commas

Updated

How to use Remove Duplicate Lines

  1. Paste your list into Input Text, one item per line.
  2. Click Remove Duplicates.
  3. Check Output Text: each line appears once, in the position of its first occurrence.
  4. Click Copy Result, or Clear to start again.

Worked example

Exact matching in action

Input

apple
banana
apple
Apple
banana 

Output

apple
banana
Apple
banana 

The second "apple" is removed. "Apple" survives because case differs, and "banana " survives because of its trailing space. Run Remove Extra Spaces first to catch the second kind.

Background and insight

The classic shell answer is sort | uniq, or sort -u. It works, but it sorts your data, and uniq on its own only removes duplicates that sit next to each other. When order matters (a ranked list, a log, a queue of URLs) the usual idiom is awk '!seen[$0]++', which prints a line only the first time it is seen. This tool does exactly what that awk one-liner does.

Under the hood, deduplication with a hash set is linear time: each line is hashed once and looked up once. Sorting first costs O(n log n) and throws away the original order. For most lists the difference is not noticeable, but keeping order is often the whole point.

Line endings can create invisible duplicates. A file saved on Windows ends lines with CR LF, and if part of a list came from Windows and part from macOS, "abc\r" and "abc" look identical on screen but are not equal. Browsers normalise pasted text to LF, which removes most of these cases, but trailing spaces and non-breaking spaces still survive a paste.

Common use cases

  • Merging mailing lists

    Combine exports from two tools and remove addresses that appear in both, without reordering the list.

  • Cleaning crawl or log output

    Reduce a list of URLs or error messages to the distinct entries while keeping the order they were seen.

  • Deduplicating config values

    Remove repeated hostnames, feature flags or dependency names pasted from several files.

  • Tidying keyword lists

    Strip repeats from a keyword or tag list built up by copying from several sources.

Tips and limitations

  • Trailing spaces and tabs make lines unequal. Run Remove Extra Spaces first if the list came from a spreadsheet or PDF.
  • Matching is case-sensitive. To treat Apple and apple as the same, use Find Duplicate Entries or lower-case the list in the Online Text Processor first.
  • Blank lines are deduplicated like any other line, so one blank line survives. Use Remove Empty Lines to drop it.
  • To see which lines were repeated and how often, rather than just removing them, use Find Duplicate Entries.

Sources and further reading

  1. uniq invocation, GNU Coreutils. uniq compares adjacent lines only
  2. sort invocation, GNU Coreutils. sort -u and its effect on order
  3. awk, The Open Group (POSIX). The associative arrays behind awk '!seen[$0]++'

Frequently asked questions

Does removing duplicate lines change the order of my list?

No. The first copy of each line stays where it was and later copies are deleted, so the result reads in the same order as your input. This is different from sort -u in a terminal, which sorts the list as a side effect of removing duplicates.

Why are some duplicate lines not removed?

They are not byte-for-byte identical. The usual causes are a trailing space, a tab, a non-breaking space copied from a web page or Word, or different capitalisation. Run Remove Extra Spaces on the list first, and lower-case it if case should not matter.

Is the duplicate line remover case-sensitive?

Yes. "Apple" and "apple" are kept as two lines. That is the safe default for data such as file paths, passwords or case-sensitive IDs. For case-insensitive matching, the Find Duplicate Entries tool has a Case sensitive switch that is off by default.

How do I remove duplicate lines in Excel or Google Sheets instead?

Excel has Data > Remove Duplicates, and Google Sheets has Data > Data cleanup > Remove duplicates, or the UNIQUE() function. Those work on cells. Pasting a column here is quicker when the data is already plain text, such as a log or a list from an email.

What happens to blank lines when I remove duplicates?

An empty line is treated as a value, so if your text has several blank lines, the first one is kept and the rest are removed. If you want no blank lines at all, run Remove Empty Lines on the result.

Is there an in-depth guide to Remove Duplicate Lines?

Yes. “Cleaning Up Text Lists: Spaces, Blank Lines, Duplicates and Commas” covers the background, how it works, common pitfalls and worked examples. Read the full guide.