How 1,000+ Links Slowed Down Our Recommendations (and the 2-Line Fix)

We recently debugged a sluggish third-party recommendation system running on a startpage with over 1,000 links. While the main site loaded instantly, the recommendations themselves were taking seconds to appear. When profiling the execution, we found that the script was running a reconciliation loop to track link array updates, and the bottleneck was a nested array difference check.

The Legacy Approach: O(N * M)

The third-party script was calculating array differences using a nested loop structure (essentially written in ES5 style):

// Quadratic time complexity: O(N * M)
var difference = afterLinks.filter(function (x) {
  var found = false;
  for (var i = 0; i < beforeLinks.length; i++) {
    if (beforeLinks[i].href === x.href) {
      found = true;
      break;
    }
  }
  return !found;
});

On a page with 1,000+ links, this nested lookups resulted in up to 1,000,000 comparison checks on every single update. The recommendations crawled to a halt.

The Clean-looking Trap: O(N * M)

Updating it to modern ES6 array methods made the code highly readable, but kept the same performance bottleneck:

// Still quadratic time complexity under the hood
const difference = afterLinks.filter(
  x => !beforeLinks.some(y => y.href === x.href)
);

While elegant, .some() still performs a linear scan behind the scenes. The browser was still executing up to 1,000,000 iterations.

The 2-Line Fix: O(N + M)

To fix the slow loading, we traded a tiny amount of memory for absolute speed by converting the lookup array into a Set. Set lookups in JavaScript are instant hash matches:

// Linear time complexity: O(N + M)
const beforeHrefSet = new Set(beforeLinks.map(link => link.href));
const difference = afterLinks.filter(link => !beforeHrefSet.has(link.href));

By mapping the comparative array to a set of href keys first, checking if a link existed dropped from a linear scan to a sub-microsecond O(1) lookup. The overall iteration checks fell from 1,000,000 down to just 2,000—instantly speeding up the recommendation load time and returning it to a seamless experience.

Lesson: When third-party integrations run on busy pages, lookups must scale linearly. Swap arrays for sets to keep updates fast.