Fastest Array Dedupe in JS — Benchmarks vs Readability

A benchmark thread crowns a one-liner the “fastest dedupe.” Another insists lodash wins. Meanwhile production arrays have 40 objects and your p95 is dominated by network. Uniqueness is a real problem — nanosecond pride usually is not. Still, picking an O(n²) helper for a 200k-line import will freeze a tab, so the algorithmic class matters more than the brand of one-liner.

Primitives: Set wins the boring race

const unique = [...new Set(list)];
// or
const unique = Array.from(new Set(list));

For strings and numbers, Set is implemented in the engine and keeps insertion order (first wins). The classic slow pattern:

list.filter((item, i, arr) => arr.indexOf(item) === i); // O(n²)

Fine for 10 items. Painful for 100k. If you normalize emails or tags, normalize before the Set or you will keep both "Ada" and "ada".

const uniqueEmails = [
  ...new Set(list.map((e) => e.trim().toLowerCase())),
];

Objects: reference ≠ value

new Set([{ id: 1 }, { id: 1 }]).size; // 2 — different references

Dedupe by business key:

function uniqueBy(arr, keyFn) {
  const seen = new Map();
  for (const item of arr) {
    const k = keyFn(item);
    if (!seen.has(k)) seen.set(k, item); // first wins
  }
  return [...seen.values()];
}

uniqueBy(users, (u) => u.id);

Choose first-win vs last-win explicitly when duplicates disagree on other fields. Last-win is seen.set(k, item) unconditionally, then values at the end.

Composite keys work when id alone is not enough:

uniqueBy(rows, (r) => `${r.orgId}:${r.email}`);

Rough complexity intuition

Approach	Typical cost	Best for
Set spread	~O(n)	Primitives
Map by key	~O(n)	Objects
filter + indexOf	~O(n²)	Tiny lists only
Sort + adjacent skip	~O(n log n)	When you need sorted output anyway

“Fastest” always needs n and element type. A microbench on ten integers will not predict a production import of product SKUs.

When readability beats the leaderboard

• Hot path runs once per request on n<100 → clarity wins
• Library already depends on lodash → uniq / uniqBy is fine
• You need stable multi-key rules → write uniqueBy with tests

Optimize when a profiler shows the unique pass in the flame chart — not because a blog used console.time on 10 elements. Allocation matters too: spreading giant Sets creates a second large array; streaming into a result buffer can help at extreme sizes.

Clipboard and line lists

Product ops often dedupe emails or IDs in text, not in JS arrays. A dedupe tool is enough for one-off lists. In app code, prefer shared helpers so every feature does not invent a slightly different uniqueness rule — especially around trimming, case folding, and empty-line handling.

Pitfalls that look like “Set is broken”

• NaN — Set treats NaN as the same value (usually what you want).
• −0 and 0 — Same in Set.
• Nested arrays / objects — Still reference equality unless you key them.
• Sparse arrays — Prefer dense lists; holes surprise people.
• Stability — Document first-vs-last so product and eng agree which duplicate survives.

A practical policy for the codebase

1. Primitives → Set after any required normalization.
2. Objects → Map keyed by id (or composite key).
3. Add tests for empty input, all-duplicate, and first-vs-last.
4. Benchmark only with production-like n and element shapes.
5. Do not rewrite working code for a 2% win on a cold path.
6. Keep one shared uniqueBy rather than five copy-pasted loops.

Dedupe performance is mostly algorithmic class (n vs n²) and correct equality. Use Set for primitives, keyed maps for records, normalize strings deliberately, and keep microbenchmark theater out of PRs unless the profiler invited it. When the job is a messy clipboard of IDs, use the local tool; when the job is application data, ship a tested helper and move on.