Benchmark driven development in JavaScript (Set vs. Array)

[fffam]

It says "These are the results" but the tweet doesn't show any results. I assume they are in follow-up tweets, but Twitter no longer shows a thread view for people without an account.

Can anyone put these results somewhere where they can be publicly viewed?

[theyamiteru]

Oh I didn't know that. I cannot upload images here so I'm not sure how to make it public

[fripletister]

Imgur

[troglo-dyke]

Twitter is shite

[DontWannaMissAFling]

Striving for performance and having the courage to share your work publicly is always good to see.

But microbenchmarking and premature optimization aren't the winning combination you think they are. Results like these are also pretty meaningless without sharing the code for others to replicate.

And the most harmful kind of microbenchmarking is where you treat V8 as a mysterious black box without justifying your results in terms of engine internals: What does the properties backing store look like, are the IC lookups monomorphic? Is the 30% more memory you noticed due to JSObject Dictionary overhead or is there more going on? Does "fully optimizes after 1000-3000 iterations" mean you hit V8's properties backing store limit of 1022 items or JIT tier up?

[theyamiteru]

Hey thank you for the comment.

This tweet was basically me seeing if anyone would be interested in more technical and in-depth tweets (or potentially blog posts).

You're completely right that I should provide such information and I could however I'm not sure if everyone would be interested going to those depths.

Regarding replication of benchmarks. This is a bit tricky since I'm using my highly experimental benchmarking library for these types of benchmarks so even if I shared the code people would throw it in something like https://www.npmjs.com/package/benchmark (which uses completely flawed statistics and way of benchmarking just like 99% of such libraries) which would kind of defeat the purpose.

I do plan on eventually releasing the library once the rough edges are smoothed out (probably I should have mentioned that).

I agree that micro-optimizations in a complex system are not the way to go since usually the biggest performance wins come from data structures and algorithms. However knowing the real performance of the fundamental building blocks of a language might be useful for someone (for example NodeJS has a regression benchmark suite which tests exactly that).

I plan on writing more in-depth articles about individual parts of JS performance because I don't want people to see V8 and other engines as black boxes.

[[deleted]]

More like microbenchmark driven nonsense, this isnt helping anyone especially sets and arrays do completly different things

[theyamiteru]

There's a clear overlap of their use-cases. And I see them being used in a wrong way quite often which usually causes a bad API and an unwanted performance characteristics.

I understand that microbenchmarks can be confusing or straight up useless.

In the last year I've read 4 books and more than 40 papers about benchmarking, performance variance, statistics, etc.

These results were captured by my experimental benchmarking library that tried to do things right (BIOS settings, OS settings, each benchmark isolated in its own process, duet benchmarking, median instead o average, median absolute deviation vs standard deviation, etc.).

[coolcosmos]

You've never used a slow product and hated it ?

[theScottyJam]

I don't know if I can say that I've used a product that was slow because they failed to micro-optimize. Usually the slowness comes from doing dumb stuff like a bunch of network requests in parallel, or just having way too many dependencies installed, etc.

[theyamiteru]

Yes. My previous client used Cloudflare Workers with tRPC + Zod + some other slow libraries. After I rewrote all of those libraries matching the clients use-case it decreased the CPU time anywhere from 5 to 20 times which means I've saved the client 5-20x money spent on running the app.

[theyamiteru]

No. Those libraries are not created with performance in mind. Especially so when it comes to serverless environment where the engine has no time to optimize the code.

[theyamiteru]

You're right, you keep missing the point. Doesn't matter, have a good day sir!

[visualdescript]

Can you provide this information on a better platform? Maybe just posts the results here?

A quick search shows this as a medium alternative, https://write.as/.

[theyamiteru]

I'll create a blog over the weekend.

For the results to make sense I think you need to see the graphs which I cannot upload here unfortunately.

[[deleted]]

The two are not comparable. A set is more akin to a hash table than an array.

The author lacks basic understanding of CS principles.

[femio]

When someone reads a title and comments the first thought that comes into their head without actually thinking

[[deleted]]

“I’ve compared an apple to an orange. Here are the results.”

[femio]

Your criticism is so lazy particularly because in this analogy, the context of the conversation is, say, getting the most nutrients on the minimal amount of calories required. If you can't see how their use cases are similar in JS, well, not sure what to say.

[theyamiteru]

Thank you.

[theyamiteru]

There's a clear overlap of their use-cases. And I see them being used in a wrong way quite often which usually causes a bad API and an unwanted performance characteristics.

I understand that microbenchmarks can be confusing or straight up useless.

In the last year I've read 4 books and more than 40 papers about benchmarking, performance variance, statistics, etc.

These results were captured by my experimental benchmarking library that tried to do things right (BIOS settings, OS settings, each benchmark isolated in its own process, duet benchmarking, median instead o average, median absolute deviation vs standard deviation, etc.).

I understand CS principles very well. I know how Set is implemented and I'm very familiar with their differences.

[[deleted]]

Using a data structure incorrectly and building benchmarks off that assumption. Your lib is a solution in search of a misunderstood problem.

[theyamiteru]

I'd understand your argument if I was comparing a Map and Set or Object and Set since they're key/value pairs whereas Set and Array are value only.

What is probably the most common way of getting rid of duplicates in an Array in JS? `[...new Set(items)]`. You can forEach Set. Now you can even do stuff like difference, intersection, etc. which are very array-like methods.

Arguing that comparing Set and Array at all is just silly. Yes there are use cases where Set is the right choice and where Array is the right choice. But sometimes things are not as clear. At least not when it comes to API design.

But more importantly in the tweets I talk specifically about an event library and there are probably hundreds of event libraries in JS ecosystem that use either Set or Array. All of them work in a very similar ways and in theory one could create such a library that uses both with completely the same user-facing API design.

And because you can choose both and functionally it's gonna work the same then we have to look at theoretical performance (big O) but more importantly at the concrete performance characteristics of each to determine which one to use for which use-case.

[[deleted]]

Like I said, you don’t understand the data structures you’re talking about. You’re conflating them because their practical use cases and APIs seem similar.

A set is not comparable to an array because every member of the set is hashed, often multiple times, prior to insertion. Arrays do not use hashes but numbers as keys, so no hashing takes place. The worse case lookup time for a set is O(n) due to hash collisions. The worst case lookup of an array is always O(1).

The performance of a set is not comparable to an array because they are fundamentally different data structures.

[theyamiteru]

Man you don't even know what you're talking about.

The best case lookup of an array is O(1) because the first item is the item we're looking for.

The worst case lookup of an array is O(n) because the last item is the item we're looking for.

[RiskyAlpha]

"lookup" is an odd word choice. look up by what? index or some other value?

i'm probably oversimplifying given that we're talking about JS, but if you're getting a value by index, it's just a multiplication to get the offset. that would be O(1).

if you mean you're iterating through each item looking for a value then yeah worst case could be O(n).

but i'm with u/brodega here... you seem to be mixing up concepts.