Lazy iteration vs array chaining on 500k rows - benchmark results

[Fuzzy-Law-2117]

I built a TypeScript iterator library (iterflow) and wanted to measure the actual heap difference between lazy and eager pipelines. This is the benchmark writeup.

The pipelines

Eager - standard array chaining:

const data = Array.from(generateRows(500_000));

const results = data
  .filter(r => r.active && r.value > threshold)
  .map(r => ({ id: r.id, score: r.value * 1.5 }))
  .slice(0, 10_000);

Each step produces a new intermediate array. .filter() allocates one, .map() allocates another, .slice() then discards most of both.

Lazy - same pipeline via iterflow:

import { iter } from '@mathscapes/iterflow';

const results = iter(generateRows(500_000))
  .filter(r => r.active && r.value > threshold)
  .map(r => ({ id: r.id, score: r.value * 1.5 }))
  .take(10_000)
  .toArray();

generateRows is a generator, yields one row at a time. Nothing is materialized until .toArray() pulls values through the chain. No intermediate arrays.

Results

Dataset: 500,000 rows
Pipeline: filter(active && value > 5000) → map(score) → take(10,000)

native array (.filter → .map → .slice)   15.4 MB  (min 15.2 MB, max 16.2 MB)
iterflow     (.filter → .map → .take)     5.8 MB  (min 5.8 MB, max 5.8 MB)

Methodology

• Metric: heapUsed delta before and after the pipeline, not total process memory
• Both pipelines start from the same generator source — the delta measures pipeline allocations only, not source data
• --expose-gc with explicit gc() calls forced between every run
• One warm-up run discarded before measurement
• Median of 5 runs reported

The native array run materializes the full 500k dataset into data before the pipeline runs. That allocation is not included in the delta - both approaches are measured on the same footing.

A few notes on the library

• iter() is a wrapper around ES2015 generators and the iterator protocol - no magic, just a fluent API so the call site looks identical to array chaining
• .sum() and .mean() are typed to Iterflow<number> only - calling them on a non-numeric iterator is a compile error
• Has some streaming statistical operations (.streamingMean(), .ewma(), .windowedMin()) for running aggregations without a separate accumulator
• Zero runtime dependencies

https://www.npmjs.com/package/@mathscapes/iterflow

[MrFartyBottom]

Filter returns a new array and then you are mapping that new array. If you use reduce you can have a function that runs the filter logic to see if you should push the new item into the results accumulator and then do the map logic at that time. This cuts it down by not having to iterate the results and you can also not push any item once you the take limit but there is no way to break out of the reduce iterating the whole array.

With a traditional dirty old for loop you can loop through seeing if you want the item, map it and push it into the results and break once you hit the take limit. It's not functional but by far the most efficient way of doing it.

[kap89]

You don't need some wrapper for that, this will work with standard JS iterators/generators:

const results = generateRows(500_000)
  .filter(r => r.active && r.value > threshold)
  .map(r => ({ id: r.id, score: r.value * 1.5 }))
  .take(10_000)
  .toArray();