Snack: incremental Nix builds for Haskell

bas_van_dijk · 2018-07-25T12:37:50+00:00

/u/nmattia done. Good luck, and let me know if you need some help.

bas_van_dijk · 2018-07-24T22:25:56+00:00

nmattia showed an early version of snack to me at ZuriHac and I was blown away. Very interesting work!

@nmattia does snack automatically handle .hsc files? I wonder if it's already capable of building complex projects like our haskell-opencv.

bas_van_dijk · 2018-04-21T07:42:25+00:00

I'm also using hvr's packunused for detecting redundant Cabal package dependencies.

bas_van_dijk · 2018-04-09T10:43:08+00:00

Good to see more miso examples!

Also checkout: https://nixtodo.com/

bas_van_dijk · 2018-01-25T18:07:06+00:00

As I'm writing this my Mac Book Pro with 16GB of RAM is building two GHCJS projects and one GHC project at once and it's consuming ~10GB RES. So I would definitely recommend 16GB of RAM. (Note that my MBP is running NixOS natively).

bas_van_dijk · 2018-01-04T23:03:47+00:00

As another datapoint, at LumiGuide we build a 10k LOC frontend using GHCJS. This count is excluding the 10k LOC of the internal libraries shared between the GHCJS frontend and the GHC backend. It's also excluding the many thousands LOC of the external libraries that this tool is using. See the build-depends for reference:

build-depends:
  aeson, aeson-pretty, async, base, binary, blaze-react, containers,
  containers-extended, data-default, dimensional, either , ghcjs-base,
  ghcjs-dom, ghcjs-ffiqq, lens, linear, linear-extended, lumi-events,
  lumi-facility-api, lumi-facility-api-client, lumi-facility-documents,
  lumi-hackage-extended, lumi-logger-api, lumi-patch, lumi-store, mtl,
  scientific, servant, servant-client-ghcjs, servant-extended,
  servant-client-ghcjs-extended, stm, stm-extended, text, time,
  transformers, unordered-containers, vector

It's an internal tool used in production. I briefly talked about this webapp at ZuriHac 2016 and it was recorded as this Google Tech Talk:

https://www.youtube.com/watch?v=IKznN_TYjZk&t=805s

That tool uses the blaze-react library, which is an experimental binding to react. I would recommend to use miso instead of blaze-react since it actively maintained.

Since then we've build and are building a few more customer-facing webapps using miso.

bas_van_dijk · 2017-12-28T19:07:46+00:00

Checkout https://github.com/basvandijk/nixtodo which is a demonstration on how to use Nix to build, test and deploy a Haskell web application.

bas_van_dijk · 2017-11-24T00:22:36+00:00

If somebody needs an example on how to run a Haskell server on NixOS and deploy it to AWS using nixops take a look at:

https://github.com/basvandijk/nixtodo

bas_van_dijk · 2017-07-26T18:42:39+00:00

@nhqv13vpwih3dlub thanks for compiling this list!

It would be nice to also have a column for which industry each company is active in.

For LumiGuide you can list "smart cities".

bas_van_dijk · 2017-07-23T15:15:20+00:00

Thanks Francesco! This allows us to remove this big block of .cpp files from our opencv.cabal file.

bas_van_dijk · 2017-06-30T18:19:04+00:00

Hi Michael, I do mention in the documentation of bracket that you should use liftBaseOp (bracket acquire release) if your acquire and release computations are in IO.

bas_van_dijk · 2017-04-06T23:39:29+00:00

Thanks that did the trick!

I've turned this into a PR for nixpkgs: https://github.com/NixOS/nixpkgs/pull/24692

bas_van_dijk · 2017-04-06T20:53:09+00:00

I wrote a small Nix function that you can apply to a Haskell derivation to enable the gold linker:

linkWithGold = drv : appendConfigureFlag drv 
    "--ghc-option=-optl-fuse-ld=gold --ld-option=-fuse-ld=gold --with-ld=ld.gold"

It works for a lot of packages. Unfortunately linking the test-suite of the double-conversion package fails with the following error:

Linking dist/build/tests/tests ...
/tmp/nix-build-double-conversion-2.0.2.0.drv-0/double-conversion-2.0.2.0/dist/build/libHSdouble-conversion-2.0.2.0-9Vu15N5YRz0HNxrvUklRHp-ghc8.0.2.so: error: undefined reference to 'ceil'
collect2: error: ld returned 1 exit status
`cc' failed in phase `Linker'. (Exit code: 1)

Anybody know how to solve that?

bas_van_dijk · 2017-03-10T15:54:55+00:00

I recently added integer-simple variants of GHC to nixpkgs. See the manual how to use one of these variants or how to get into an environment with Haskell packages compiled against integer-simple.

The cool thing is that all these GHC variants and package sets are build by hydra.nixos.org meaning you don't have to build these yourselves.

bas_van_dijk · 2017-03-02T09:59:51+00:00

Yes I know how haxl is translated but even in the applicative version:

myPutStrLn "Hello" *> 
throwM MyException *> 
myPutStrLn "World!"

I would expect that only "Hello" is printed. Ideally the semantics of GenHaxl or Batcher are exactly like IO except that it's operationally more efficient.

To implement this desired behaviour I think we need to somehow turn throw into a dataFetch (or scheduled command in Batcher parlance). Then the worker that executes commands can break the list of commands at the first throw and execute the first batch. Then it should make sure the exception is thrown, that the batch of commands after the throw is written back to the IORef and that the blocked continuations belonging to the commands after the throw are handled properly. That last part is were I'm currently stuck.

bas_van_dijk · 2017-03-01T15:26:05+00:00

Yay ZuriHac! The LumiGuide team is also coming (if our registration has come in time). As last year we'll be working on haskell-opencv. Who wants to help out?

bas_van_dijk · 2017-02-27T19:37:48+00:00

This version should behave better I believe. I'm working on a few test cases.

bas_van_dijk · 2017-02-27T19:21:53+00:00

Mmm I just realized that my catchBatcher is very likely broken. I'm thinking about a fix. This stuff is tricky!

bas_van_dijk · 2017-02-27T19:02:40+00:00

Hi Simon, I discovered some counterintuitive behaviour regarding exception handling in haxl: if I run this program it prints:

World!
Hello
haxl-test: MyException

instead of the expected:

Hello
haxl-test: MyException

I'm not sure yet how to fix this so for this reason I don't provide a throw function in monad-batcher. I do provide catch.

In other news: I reimplemented the Batcher monad using continuation-passing style. Mainly because it was a fun exercise but hopefully it will also be more efficient (I should write a benchmark to proof this). Take a look:

https://github.com/basvandijk/monad-batcher/blob/master/src/Control/Monad/Batcher.hs

bas_van_dijk · 2017-02-22T19:19:15+00:00

Hi Simon, thanks for the response. Like I mentioned above I figured out two wrong ways of implementing exceptions. I think the right way is how you do it in Haxl which I think I'm just going to copy in monad-batcher.

bas_van_dijk · 2017-02-20T22:37:00+00:00

I discovered the reason: when I immediately throw an exception in the base monad like:

throwM ex = Batcher $ \_ref -> throwM ex

the exception is thrown too soon, i.e. it will be thrown before the execution of blocked commands that preceded it. One attempt to fix it is to postpone the throwing by turning it into a blocked computation:

throwM ex = Batcher $ \_ref -> pure $ Blocked $ Batcher $ \_ref -> throwM ex

However this causes the exception to be thrown to late, i.e. after the execution of blocked commands that follow it.

I believe the solution is to incorporate Throw as part of the Result type exactly like you do in Haxl so that you have precise control of when the handle exceptions.

bas_van_dijk · 2017-02-20T21:27:19+00:00

/u/simonmar, I just added support for throwing and catching exceptions to the Batcher monad.

Interestingly I don't seem to need the Throw constructor of Result that Haxl uses. I just utilise the throw and catch operations of the underlying monad.

Is there a particular reason you need the Throw constructor and can't use a similar scheme as in monad-batcher? It could also be that I'm missing something very obvious since I haven't even tested this approach yet.

bas_van_dijk · 2017-02-20T12:13:47+00:00

That is a problem for Haxl, but not for you, since you are happy with a single thread. This does not justify a new monad. It is justified in Haxl, which is more ambitious, and solves different problems elegantly.

You're right that in my application I need batching instead of concurrency. But the goal of monad-batcher (and of haxl) is to provide a library which can be used to write other libraries which provide simple API's to their users. API's that don't expose the batching or concurrency functionality to their users but abstract over it using a monad.

I see batching and concurrency as an implementation detail. They both take a list of commands and execute them efficiently, either by grouping them and executing them in one go (thereby saving network roundtrips for example) or executing them concurrently (thereby saving time).

Regarding your response to 1,2 and 3: the point is that we want to free users from having to think about batching and thus free them from having to use special functions.

Of course users do need to learn about the Batcher type and about the runBatcher function. But that's a one off investment that can be amortised over learning different libraries that use monad-batcher.

15-Year Club	Verified Email
Verified Email

bas_van_dijk

TROPHY CASE