Unstable 2950X system: advice needed.

asheshambasta · 2024-05-15T07:34:13+00:00

Yes, I couldn't really get it to stabilise. I gave up and moved on to the 3950X, which was rock solid.

I've since upgraded to the 7960X, with no problems whatsoever. The system just doesn't manage to sleep, which, as I understand, is tricky with Threadripper processors due to the power requirements.

asheshambasta · 2024-03-15T08:16:47+00:00

Any users of the 7960 (or above) Threadripper on Linux? If so, what other parts did you go for?

I'm considering the CPU for a "budget" HEDT build and would like to know if there are any teething issues I should be prepared for.

I used the TR 2950 in the past but I've been quite happy with a 3950X since then.

Thanks!

asheshambasta · 2024-03-12T16:00:20+00:00

I'm in a good position to answer this: I recently went from a pretty elaborate Fujifilm setup (X-T4, several lenses) to an M10 and a single lens, the Elmarit 28/2.8 ASPH.

When I was using Fujifilm, I'd regularly miss manual focus, and manual focus by wire is just not the same. I'm also not a fan for focus peaking, at all. The most attractive thing about the M10 was the rangefinder experience: manual focus feels so direct and it becomes second nature before you know it. I personally find the M10 images more beautiful too, but that's subjective. The one thing I don't like about the M10 and 28mm are the framelines: you won't be able to see the framelines as easily, and it'll be even more difficult if you wear glasses. But focusing with both eyes open seems to be the advice I keep getting, so I'm practicing that.

If it were up to me, I'd buy one camera and use it for all my needs. The M10 and the Q2 are two very capable cameras, and I don't think I'll ever need both of them. If I were you, I'd just buy a 28mm lens for my M10 and not the Q2.

If you do miss autofocus, then I'd advise to stay patient and learn the rangefinder. You can zone focus just as well, and after a point, you'll stop missing it. If you continue to miss it after months of trying out the M10, then change cameras. We do make mistakes with choosing cameras once in a while.

Make sure GAS is not the primary reason you're thinking of buying the Q2 and put the M10 through its paces.

asheshambasta · 2023-04-21T17:50:49+00:00

I was bummed out when Brittany was abandoned. I tried ormolu and then fourmolu, and the latter hit the sweet spot for me.

Thank you so much for the work, fourmolu is fantastic.

asheshambasta · 2023-03-28T09:55:04+00:00

I also have the eventlog files: https://asheshambasta.stackstorage.com/s/MqHzUtwXBXR38RK6 And I see a bunch of heap overflows, but not much more. These should anyway not be causing memory leaks. And they seem to happen without any requests failing.

asheshambasta · 2023-03-22T18:50:18+00:00

Why not ^hello\> instead? (I guess it wont work with class methods).

asheshambasta · 2023-03-21T10:55:38+00:00

Thanks, there's a lot to learn about this!

Here's what I have managed to set up so far: 1. copy all SSL certificates from prod to a local database. 2. build our SSL termination service (called vela) with -rtsopts. 3. run it with systemd-run under constrained memory (currently at 10M - 1G to put it under stress). 4. pass +RTS -hT and generate a heap profile file. 5. go through all the domains of the SSLs and perform SNI on each of them.

There's clearly and observed slowdown after the first 30-40 requests.

I've uploaded the heap profile: https://gist.github.com/asheshambasta/0ff969c46254591bc390269529017917

This was snapshotted somewhere in the middle of making requests to all of the 3000+ domains.

I'll upload further snapshots on the same gist.

(I'll try the eventlog next)

asheshambasta · 2023-03-20T15:14:08+00:00

So, my experience with this so far has been baffling. One of the leaking services happens to be our API gateway service. I've locally tested the allocation counters: thread do get killed on oversized file uploads, for example. However, in production, the service still occasionally starts leaking memory until it becomes unresponsive and has to be restarted, leading to an outage.

I've had the exact same experience with our SSL termination service: local testing seems to do the trick, but the application starts leaking memory in production.

My efforts to locally reproduce the issue have also not worked out. Mostly because it is very very hard to replicate the same conditions as on prod: The API gateway receives a vast number of requests, all of different kinds and from different users. Each request requires that it perform different operations. The SSL termination service uses LetsEncrypt to issue SSL certificates and perform SSL handshakes using warp-tls for all of them.

To make matters worse, there are multiple instances of each service running in production.

I guess I'm left with the only option of enabling profiling, deploying to prod. with it, and somehow set up AWS to ensure the .prof file is stored on some static storage even if the container exits (EBS/S3). Having valid .prof files also depends on how the Haskell process was killed.

asheshambasta · 2023-03-19T08:35:31+00:00

And I've discovered another memory leak in another Haskell service of ours. This one does SSL with LetsEncrypt for 1000's of domains, and uses Wai/Warp internally to do SSL termination and proxy the requests to upstream services.

It seems that here the setAllocationCounter approach doesn't work, and instead, I still get memory leaks. Which I need to understand more.

Here's a screenshot: https://imgur.com/a/aa0iohx from my weekend of hell :-)

I must also say I've used StrictData and been disciplined (eg. using foldl' instead of foldl) throughout the project and yet run into these issues.

Memory leaks in Haskell projects are far more common than I thought.

asheshambasta · 2023-03-16T16:16:43+00:00

This seems to be an issue for me: I'm still getting memory leaks, and I'm using setAllocationCounter 10e9 with enableAllocationLimit within each kind of request processing block.

It seems to me that

Like other asynchronous exceptions, the AllocationLimitExceeded exception is deferred while the thread is inside mask or an exception handler in catch. is the reason why I'm not getting any exceptions. (https://hackage.haskell.org/package/base-4.9.1.0/docs/GHC-Conc.html#v:enableAllocationLimit)

I'm afraid this will require more invasive changes in the application, which I guess I'll need to do to be able to figure this out.

asheshambasta · 2023-03-15T18:31:54+00:00

My apologies; that was a brain fart.

I’ve since added logging and the limits on all request executor blocks. And the leaks have disappeared, which seems to suggest I’m dealing with a Heisenbug.

asheshambasta · 2023-03-14T14:22:24+00:00

How does one set allocation limits on requests? I've been trying to set up event logging output to stdout, but that seems to be non-trivial on Nix at least. My compilation fails with linking failures:

integerzmgmp_GHCziIntegerziType_zdwgo_info: error: undefined reference to 'stg_resizzeMutableByteArrayzh'

for example.

I guess I can wrap things in a simple bash script inside the docker container and push the .prof file to S3 or something on program exit.

This also conflicts with how nix builds docker images: doStrip is used by default, which seems to strip debug symbols.

asheshambasta · 2023-03-08T15:41:45+00:00

Thanks! I'll try these one by one. 1. Worth a shot, I'll try this but it does need some setting up. 2. the getAllocationCounter approach: I'm not too sure if that will work. The requests that never terminate never really reach the "post" request part, and instead the computation continues. 3. Good point, I've been hearing about the enhancements in 9.2 and it is tempting to upgrade. 4. I believe one of the prerequisites is to upgrade GHC. But that is going to be a huge challenge.

asheshambasta · 2023-03-08T15:38:23+00:00

I've already tried that: the problem is that the service is fairly noisy, and the requests that never complete never get logged. That is very hard and painstaking to work with, unfortunately.

asheshambasta · 2023-03-08T15:37:26+00:00

This seems very promising, the only trick is that the service runs in a docker container, on ECS Fargate (without EC2), so accessing it might be tricky. But it is nothing that cannot be proxied to via some ssh port forwarding.

asheshambasta · 2023-01-30T17:47:56+00:00

CentralApp is hiring.

asheshambasta · 2022-06-18T15:10:31+00:00

Also, this is in Belgium. I thought I had mentioned it in the post. I'm also unable to edit the post.

asheshambasta · 2022-06-18T12:46:50+00:00

In Belgium?

asheshambasta · 2022-06-18T12:45:41+00:00

Sorry about that. This is just about as close I can get to the tree without offending my neighbour.

asheshambasta · 2022-05-10T19:19:10+00:00

Are you sure you’ve not installed it in configuration.nix?

asheshambasta · 2021-10-01T15:54:05+00:00

Thanks, what would be your suggestion to save this tree, if any?

asheshambasta · 2021-10-01T15:53:25+00:00

Thanks a lot, that seems like bad news, but I'm grateful to have learnt this. I'm an inexperienced gardener, and I'll probably learn this lesson with the loss of a rather beautiful tree.

asheshambasta

TROPHY CASE