Find Duplicate Entries
See which lines repeat and how many times, with case, trimming and space options plus totals for unique lines and extra copies. Runs in your browser.
The interactive tool loads here when JavaScript runs.
About this toolWhat is Find Duplicate Entries?
Find Duplicate Entries reports which lines in a list appear more than once and how many times each one occurs. It does not delete anything. Use it to audit a list before you clean it: which email addresses were entered twice, which SKUs repeat across two exports, which error message dominates a log.
Five switches control matching and output. Case sensitive (off by default) decides whether "Bob" and "bob" match. Trim spaces (on) ignores leading and trailing whitespace. Remove extra spaces (off) also collapses runs of spaces inside a line. Only duplicates (on) hides lines that appear once. Show count in result (on) appends "(appears N times)".
Blank lines are ignored. Results are listed in the order each entry first appears, along with four totals: total lines, unique lines, duplicate types (distinct entries that repeat) and extra duplicates (copies beyond the first).
- Occurrence count for each repeated line
- Case, trim and inner-space normalisation switches
- Totals for lines, unique lines, duplicate types and extra copies
- Optional full frequency list
- Runs in your browser; nothing is uploaded
Updated
How to use Find Duplicate Entries
- Paste your list into Input Text, one entry per line.
- Set the Options switches: Case sensitive, Trim spaces, Remove extra spaces, Only duplicates and Show count in result.
- Click Find Duplicates.
- Read the totals at the top and the entries under Results, then click Copy Result.
Worked example
Duplicate email addresses with default settings
Input
alice@example.com bob@example.com Alice@Example.com carol@example.com bob@example.com bob@example.com
Output
alice@example.com (appears 2 times) bob@example.com (appears 3 times)
Case-insensitive matching groups the two spellings of Alice. The totals read 6 total lines, 3 unique lines, 2 duplicate types and 3 extra duplicates (one extra Alice and two extra Bobs).
Background and insight
Counting duplicates is a group-by. In SQL it is SELECT value, COUNT(*) FROM t GROUP BY value HAVING COUNT(*) > 1; in a shell it is sort | uniq -cd. Both need the data in a table or a sorted file. Pasting the list here gives the same answer without either.
The normalisation options matter more than they look. Duplicates in real data are rarely exact: one system stores "alice@example.com", another "Alice@Example.com ", and a form adds a double space. Deciding what counts as "the same" is a data-quality decision, and it should be made deliberately. Email local parts are technically case-sensitive under RFC 5321, but almost every provider treats them case-insensitively, so case-insensitive matching is usually right for emails and wrong for things like file paths or base64 tokens.
When case-insensitive matching groups several spellings together, the result shows the first spelling the tool saw. The totals tell you how dirty a list is: if Extra Duplicates is 300 out of 10,000 lines, roughly 3% of the rows are redundant copies.
Common use cases
Auditing sign-up lists
Find people who registered twice with different capitalisation before sending a mailing.
Checking data imports
Confirm that IDs or SKUs in an import file are unique before loading them into a database with a unique constraint.
Spotting noisy log lines
Paste log messages and turn off Only duplicates to see a frequency count of every line.
Reviewing survey answers
Count how often each free-text answer appears after trimming and case folding.
Tips and limitations
- Turn off Only duplicates to get a frequency table of every distinct line, including the ones that appear once.
- Duplicate Types counts distinct repeated entries; Extra Duplicates counts surplus copies. If one line appears 5 times, it adds 1 to the first and 4 to the second.
- Leave Case sensitive off for emails and names, and turn it on for passwords, file paths, tokens and anything else where case carries meaning.
- This tool reports duplicates but does not remove them. Use Remove Duplicate Lines once you have decided what to keep.
Sources and further reading
- RFC 5321: Simple Mail Transfer Protocol, IETF. Section 2.4: the local part of an address may be case-sensitive
- uniq invocation, GNU Coreutils. Counting repeats with uniq -c and -d
- Table Expressions: GROUP BY and HAVING, PostgreSQL Global Development Group. The SQL group-by that this tool reproduces
Frequently asked questions
How do I count how many times each line appears?
Turn off Only duplicates and leave Show count in result on, then click Find Duplicates. Every distinct line is listed once with "(appears N times)" after it, in the order it first appeared. Blank lines are skipped, so they never show up in the counts.
What is the difference between Duplicate Types and Extra Duplicates?
Duplicate Types is the number of distinct entries that occur more than once. Extra Duplicates is the number of surplus copies, which equals total lines minus unique lines. A list with "a" three times and "b" twice has 2 duplicate types and 3 extra duplicates.
Should duplicate detection ignore case for email addresses?
Usually yes. The SMTP standard allows the part before the @ to be case-sensitive, but major providers treat it case-insensitively and domains are always case-insensitive. Leaving Case sensitive off catches "Alice@Example.com" and "alice@example.com" as one person.
Why does the result show only one spelling of a duplicate?
When trimming or case-insensitive matching groups several spellings together, the tool reports the first one it saw. "Apple" followed by " apple " shows as "Apple (appears 2 times)". Turn on Case sensitive if you need the spellings reported separately.
Can I find duplicates across two different lists?
Paste both lists into the input one after the other and anything that appears in both will show as a duplicate, along with repeats inside each list. To separate "in both lists" from "repeated within one list", use the List Comparison tool instead.
Is there an in-depth guide to Find Duplicate Entries?
Yes. “Cleaning Up Text Lists: Spaces, Blank Lines, Duplicates and Commas” covers the background, how it works, common pitfalls and worked examples. Read the full guide.
Related Tools You May Like
Free, in-browser tools that pair well with this one.