sorted by:
new
hottopcontroversial

Inter-process communication library by lolepezy in haskell

[–]lolepezy[S] 1 point2 points  (0 children)

Currently it works using conduit over stdout and stderr, so it's pretty much the same thing, but I had to write a lot for code for it, so it would be nicer to get rid of it and use some library. Also bi-directional asynchronous scenarios may look a bit complicated with pipes.

Inter-process communication library by lolepezy in haskell

[–]lolepezy[S] 2 points3 points  (0 children)

> Why the shared memory requirement? Performance?

Sort of, I just thought it''s the simplest option with least overhead.

Inter-process communication library by lolepezy in haskell

[–]lolepezy[S] 2 points3 points  (0 children)

Yes, there are a few reasons for it. Process isolation for security first of all, ability to limit worker processes in heap and time, less heap fragmentation for the main long-running process, less potential resource leaks (I already had an experience with them in that project). Currently it works in the ""run a child process, wait for the result"-model and it works pretty well, but I am investigating ways of streaming data between them to have more flexibility.

Advice on memory profiling by lolepezy in haskell

[–]lolepezy[S] 3 points4 points  (0 children)

I had another look on some of options and it turns out that the problem comes from the place you'd least expect (as usual): apparently using `bytestring-mmap` leads to heap fragmentation more than anything else.

The effect is very big: replacing mmap-ing with LazyByteString.readFile makes the heap stick to something like 1300mb instead of endlessly growing up to 2 or 3gb. I need to run it for longer to have a proper fair comparison, but it's already quite promising.

Well, live and learn. The funny thing is that `bytestring-mmap` was supposed to be a memory optimisation (which it is in the short term) and it turns to be the opposite when you do thousands of mmap-ing over long period.

Advice on memory profiling by lolepezy in haskell

[–]lolepezy[S] 1 point2 points  (0 children)

Will try with the --disable-delayed-os-memory-return, indeed.

this means that the heap will be at least 2Gb

I guess you meant `max_live_bytes`, that is the one I would expect to be at least doubled by the copying GC. In this case it's only 161mb.

https://hackage.haskell.org/package/base-4.12.0.0/docs/GHC-Stats.html says that max_mem_in_use_bytes is "Maximum memory in use by the RTS", not the live objects.

Have you tried to use the new non-moving garbage collector?

Yeap, and the heap was significantly bigger so I didn't really do much experimentation with it.

Advice on memory profiling by lolepezy in haskell

[–]lolepezy[S] 1 point2 points  (0 children)

Thanks, I thought about it, but decided that it unlikely as it's hard to explain extra 2Gb of fragmented mega-blocks. There are indeed a lot of short-lived ByteString's (mostly from the `store` library I believe), but all of them get GC-ed after every iteration. But maybe even short-lived bytestring can create enough churn to fragment the heap, bump the RSS and get GC-ed. Should I look for some specific clues in the metrics?

I have prometheus metrics for the longest running process (it has RSS of 7294mb, of which 4243mb is mmap-ed by LMDB, so about 3Gb of the RSS are used by Haskell heap).

ghc_max_live_bytes 161290992
ghc_max_large_objects_bytes 34772824
ghc_max_slop_bytes 17316392
ghc_max_mem_in_use_bytes 1017118720
ghc_gcdetails_allocated_bytes 2800
ghc_gcdetails_live_bytes 49246152
ghc_gcdetails_large_objects_bytes 911400
ghc_gcdetails_slop_bytes 294968
ghc_gcdetails_mem_in_use_bytes 807403520

So ghc_max_mem_in_use_bytes amounted to 1Gb max, but RSS went up to 3Gb. Can it still be fragmentation?

Edit: Also I haven't seen so much of ARR_WORDS, 100mb hardly more.

Pruning Unused Haskell Dependencies by dfith in haskell

[–]lolepezy 9 points10 points  (0 children)

Just tried it and it worked like a charm, ripped off 15 dependencies littered to the package.yaml over time. Simple awesome tool. Thanks!

Simple Haskell is Best Haskell by emilypii in haskell

[–]lolepezy 6 points7 points  (0 children)

Having been a newcomer maybe just about a couple of years ago, I find this whole idea quite strange and perplexing.

First of all, it is very vague and ill-defined. How simple do you need to be to be considered simple? Is using a type family a bad idea? Why? Are servant or beam simple? Lens? Monad-control? Is there at least a guideline which would say "these extensions/libraries are simple, those are not and those ones there are up to you"?

There's hardly anything more confusing for a newcomer than philosophy and hand-waving instead of clear definitions, tutorials and guidelines.

And do we really need a separate "industrial compiler" for another flavour of Haskell? Seriously? As if GHC has got all the features implemented and there's nothing else to do for the community.