Set vs filter vs lodash — Why Your Dedupe Still Leaves Duplicates
Classic r/javascript thread: “Just use [...new Set(arr)]” — then OP replies that duplicates are still there because the array contains objects, lines differ by trailing spaces, or CSV rows look identical but have different hidden characters. The one-liner isn’t wrong; it’s answering the wrong question.
- Set dedupes objects by reference —
[{id:1},{id:1}]has length 2 after Set. You need a key. - Trim matters —
"foo"and" foo "are different strings. Excel exports love trailing spaces. uniqBywithout the right key — Keeps the first{id:1, name:"old"}and drops{id:1, name:"new"}silently.- “Dedupe then sort” vs “sort then dedupe” — Order of operations changes which duplicate survives when you use Map.
String arrays and line lists
The happy path — unique strings, no trim issues:
const tags = ["react", "react", "vue"];
const unique = [...new Set(tags)];
// ["react", "vue"]Real-world export from CRM — trim + lowercase for email cleanup:
const raw = `[email protected]
[email protected]
[email protected] `;
const lines = raw.split("\n").map((l) => l.trim().toLowerCase());
const unique = [...new Set(lines)].filter(Boolean);For 10k+ lines pasted from a spreadsheet, a browser dedupe tool beats writing a one-off script on a locked-down laptop — trim, sort, unique, copy.
Array of objects — pick a key explicitly
const rows = [
{ id: 1, label: "first" },
{ id: 1, label: "duplicate id" },
{ id: 2, label: "ok" },
];
// Keep last occurrence per id
const byId = [...new Map(rows.map((r) => [r.id, r])).values()];| Approach | Keeps | Best for |
|---|---|---|
Set(arr) | N/A for objects | Primitives only |
Map by key | Last wins (above) | Configurable |
filter + findIndex | First wins | Stable “first seen” |
lodash uniqBy | First wins by default | Quick scripts |
When duplicates aren’t duplicates
- Unicode normalization —
"café"vs"café"(composed vs decomposed). Normalize withnormalize("NFC")before Set. - Windows
\r\n— Split on\r?\nbefore comparing lines. - Empty lines — Decide if you want to keep or drop blanks before dedupe.
- Case sensitivity — Usernames vs emails have different rules.
Performance reality check
| Size | Set + trim | Sort + unique in tool |
|---|---|---|
| < 1k lines | Either fine | Either fine |
| 10k–100k | Fine in modern JS | Browser tool OK |
| 1M+ | Stream or DB | Don’t paste in browser |
For one-off marketing list cleanup, correctness beats Big-O notation.
Objects need identity, not Set
Set dedupes by reference for objects — [{id:1},{id:1}] stays length 2. Use Map keyed by id, or serialize stable JSON strings if order doesn’t matter:
const uniqueById = [...new Map(items.map((x) => [x.id, x])).values()];Same trap hits filter((v,i,a)=>a.indexOf(v)===i) on object arrays — indexOf compares references, not deep equality.
Step-by-step: Mailchimp export cleanup
- Export CSV from source system.
- Paste email column into dedupe tool — enable trim, optional sort.
- Scan count: input lines vs output lines — delta should match expected dupes.
- Spot-check 5 removed entries manually (typos vs true dupes).
- Import cleaned list; keep raw export in cold storage for audit.
Tool: dedupe locally
The dedupe tool processes lists in your browser — no upload, no account. Paste lines, toggle trim/sort, copy unique output.
TL;DR
Set works for clean string arrays. Real data needs trim, normalization, and key-based logic for objects. When in doubt, paste into a local dedupe tool, verify counts, then ship.