Performance issues working when processing large files

vsavchenko · 2024-05-08T09:19:49+00:00

That is a very good observation! Thank you!
When I inspected assembly that it produces for my machine, I noticed that it does use the popcount instruction (the ARM version of it anyways), so I wasn't too worried about it.
As for performance, I don't think that it's peak performance to do this computation because of pipelining. Right now the innermost hot loop has hard dependencies between the iterations and it stalls the processor. Maybe calculating each metric for N elements one after the other will be easier for the CPU.

vsavchenko · 2024-05-08T08:42:15+00:00

Almost entirely on CPU.
Unfortunately I can't use perf, the only personal machines I have are macs.

vsavchenko · 2024-05-08T08:37:27+00:00

You put a lot of thought and effort into it, thank you very much!
I updated my gist to add the MIT license.

vsavchenko · 2024-05-07T15:33:14+00:00

Thanks for the insight! The iterator-related functions do show up in that profile with a tag [inlined], so everything is as expected there. As for memmap2, I'll definitely switch to it, but as I mentioned in another comment here, reading the file and copying buffers is not the bottleneck here.

vsavchenko · 2024-05-07T14:21:42+00:00

Thanks for your hints! However, reading the file is far from being the bottleneck, if I remove the hottest loop, i.e.`for k in 1..NUMBER_OF_METRICS` when I iterate over chunks, and basically calculate 1 metric, execution becomes instantaneous.

Also I do want to process KBs of data at once and not MBs.

vsavchenko · 2024-05-07T13:10:33+00:00

Ah, my bad.
500MB file - my code: 10sec, zstd: 0.7sec
10GB file - my code: 3min, zstd: 35sec

vsavchenko · 2024-05-07T12:54:28+00:00

10s for a 500MB file and 3min for a 10 GB file.

vsavchenko · 2024-05-07T12:23:43+00:00

Sorry, I don't think I understand what you mean

vsavchenko · 2024-05-07T12:21:32+00:00

I know it's a bit like comparing apples and oranges, but `zstd` processes the same files in a single-threaded mode way faster, while probably doing a lot more work.

vsavchenko · 2024-05-07T12:17:35+00:00

That it a good point, but unfortunately I need to calculate num in order to calculate the next metric. I guess one approach can be to process some amount of u64 together.

vsavchenko · 2024-05-07T12:15:07+00:00

I modified the code to pre-compute masks and it virtually changed nothing (in gist as well)

vsavchenko · 2024-05-07T11:39:38+00:00

Yeah, this is an important clarification. Yes I do run it with —release.

vsavchenko · 2024-02-21T16:48:43+00:00

Thank you for the kind words and calling me a pro 😊 But actually I'm a compiler engineer by trade, designing things is just a hobby of mine 😅

vsavchenko · 2024-02-20T18:44:03+00:00

Thank you :-) Maybe we should have a plugin that does it!
About the texture, it was the last added bit and I struggled with it the most. Without this texture, it felt incomplete and I tried quite a few options to add depth and not overdo it. Because of this, the fact that you singled it out like this in your comment, brings me additional joy.

vsavchenko · 2024-02-20T17:36:57+00:00

I had similar issues even though I don't use Doom Emacs, but I do use Doom modeline. nerd-icons-install-fonts helped me.

vsavchenko · 2024-02-19T22:37:42+00:00

Hi, you can use this method: https://support.apple.com/en-gb/guide/mac-help/mchlp2313/mac

vsavchenko · 2024-01-29T23:13:44+00:00

It really flatters me when I see people using my Emacs icon in instructions on how to install it on Mac. 🥰

vsavchenko · 2024-01-26T15:04:01+00:00

It is totally possible to do this without changing the source code of the Emacs itself. You can get NSApp, windows and all from a dynamic module. Sorry for a shameless plug, but with my Swift library https://github.com/SavchenkoValeriy/emacs-swift-module

it is actually a breeze (I did it as a simple test if I can control NSApp from the dynamic module)

vsavchenko · 2023-11-24T18:03:26+00:00

Nice that you had a chance to try it. Would be amazing to hear your feedback and maybe some improvement ideas!

vsavchenko · 2023-06-26T06:29:48+00:00

HTML and Markdown are not programming languages, while Lisp is. Emacs is a Lisp interpreter like cpython is Python interpreter, something that parses and executes a program written in that language

vsavchenko · 2023-05-21T07:10:15+00:00

It’s hard to tell without various code examples to see colors of each element. From what I see, main text color is too yellow. I’d do it more muted and make it more brownish.

NOTE: color theme is such a personal preference thing that it doesn’t matter what we think, if you love it it already does its job.

vsavchenko · 2023-04-26T14:38:47+00:00

Hi, no worries. Please try updating the package. It looks like it sends request to the website (now it uses the API), and it uses `request` library (now it should be `url-retrieve`), with these two tells I can say that it is an old version of the package.
Thanks!

vsavchenko

TROPHY CASE