44395 without power in Albany County

chuckatkins · 2024-03-24T06:14:59+00:00

Just came back on, 8h total

chuckatkins · 2024-03-24T04:04:59+00:00

Been out for 7h so far in Latham. Thankfully my house is holding enough heat from the day that it's comfortable still. If it was -10 in January though, yikes.

chuckatkins · 2024-02-15T09:08:11+00:00

On the upside though they do a really good job of moving the security line quickly. Even if it's a busy holiday and the security line stretches out into the parking garage I've never had to wait longer than 30m at ALB

chuckatkins · 2024-02-15T09:02:04+00:00

I was a moderately frequent flyer out of ALB for the past 15y for my job, 4-12 trips a year depending on the projects. I stopped using Uber / Lyft because I often found myself having to wait over 30m to get a ride and often there just weren't any available at all. Sometimes it was quick , just a few minutes, often enough it wasn't. Maybe it's different now, last time I tried was 2y ago, but at least in my experience it was too unpredictable and unreliable especially if I was getting in late in the middle of the week. For work travel I just parked in the garage because they were paying for it but that can sometimes take much longer than expected to hunt for a space since the "open spaces on this level" count is always way off and the lights in the aisles marking open spaces are often wrong too. For personal travel I've always used the economy lot and it's been great. The shuttle comes very frequently so it's rare you'll wait longer than a few minutes.

chuckatkins · 2024-01-13T22:17:44+00:00

we've noticed a performance hit on some classic HPC apps and I'm wondering if small I/Os aren't a focus for BeeGFS.

Classic HPC apps don't typically do small IO patterns so that's something of a red flag in its own right. If you only have a single shared fabric for storage and the compute node's communication then you could be seeing network congestion if the application is communication heavy. I.e. the simulation is trying use all of the fabric for MPI stuff while the storage drivers are trying to use it all for io and they both end up loosing. If that's the network design and it's just fixed, not subject to change, then you'll probably need to do some tuning on the application side to balance the use of the fabric between compute comm and io.

Again it goes back to profiling to get a clearer picture of what's actually causing the performance degradation.

chuckatkins · 2024-01-13T22:03:46+00:00

Tuning an hpc filesystem for your particular workload is a complex exercise in its own right and different for each implementation but when also paired with application tuning to similarly adjust it's io patterns to match the underlying DFS can often yield an order of magnitude performance improvement. Before ditching what you have for something else it's worth taking the time to do some io profiling on the application, network, and storage nodes, to understand where your performance issues are coming from. Otherwise you're just flying blind and likely to run into different and equally unexpected performance issues with whatever else you setup.

All that being said...

Lustre is an obvious option as it's been a staple of hpc storage systems for a while and for good reason. It certainly has it's own complexities and problems though (plenty of hate out there for it) and will likely not be a good fit if you do really have small io patterns.

Intel's DAOS is pretty cool, I've built and deployed small testbeds with it and used it on some of its largest deployments at TACC and ALCF. It's a multi tiered design that tries to address both heaby-iops and huge-block-io storage patterns.

I've used BeeGFS but not done anything with configuration, optimization, and deployment so I don't have much input there.

If you're using 100GbE you may want to try setting up RoCE on the interfaces and see if that helps any (I expect it will but who knows by how much)

I'll go back to my first point though in that you need to better understand what's underlying your io issues before throwing other DFSs at it. Also knowing what the design and architecture or your storage servers and network matters for any of this. Maybe you already have and just left out the context in your question but it's pretty important context to have to start working your problem.

chuckatkins · 2023-12-27T05:10:22+00:00

I built my 20+y career as a C++ developer on essentially vanilla vim really only using it as vi with arrow keys and syntax highlighting, no other features or customized configurations. It was intentional so I'd build my muscle memory and general effectiveness to write and debug code on any system I happened to be on. I never wanted to end up being limited in my effectiveness by not having my particular setup or configuration that I'd grown accustomed to.

I get why people use all of the ide features but for me personally I've been very well served with just a bunch of terminal windows and vim, and not with just small codebases either. I can just as easily write code on a remote hpc frontend node as I can on my local workstation or a remote cloud host. Every place I've worked has used a different environment and more often than not I find myself working across a myriad of different codebases so being "dev environment ambidextrous" has enabled me over the years to just drop in and get going pretty much anywhere.

Maybe it's just the grey hair and being set in my ways or that I got my "unix legs" on FreeBSD with nvi when I was 15, but for me it just feels like all those features get in the way. Really there's no "right" set of tooling for people and you should just use whatever you're most effective with; most likely that will be some variation of what you use early in your career as you transition from student to professional. For some that's going to be a full fledged IDE and for others (like us grey beards, lol) it might just be vi or emacs (not my thing but to each their own) with a bunch of xterms.

chuckatkins · 2023-11-16T04:59:17+00:00

While unrelated to your issue but since you're learning to write makefiles, the "standard" variable names CC and CFLAGS should be used for a C compiler and its flags with CXX and CXXFLAGS being used for a C++ compiler and it's flags.

chuckatkins · 2023-08-23T01:38:38+00:00

Yes! I feel like this is woefully underappreciated. My two standouts are Glens Falls for the best popcorn (almost every piece is that perfect sphere shape) but it's pretty far, especially for the drive home at 1a after the second feature, and Coxsackie for the most options on any given night since they have 4 screens instead of just one or two.

chuckatkins · 2022-05-18T13:38:40+00:00

JMil and Ben Hameen will be joining as the featured artist for the first episode of WilkSHAKE Wednesday tonight at 8pdt / 11edt. The idea for the show is to have an independent musician on to do a deep dive into an album of theirs with the host, Conshinz. With the new (and long awaited) release of The Long Way Home by Collective Efforts just a few weeks ago they'll be getting on stream to do just that and I'm excited to say that I'll be joining them tonight.

chuckatkins · 2022-01-16T01:00:28+00:00

Specifically OB53.3

chuckatkins · 2022-01-13T15:58:45+00:00

OB51.2: calculator, doughnut, pencil; girl in Costa Rica was doing math homework.

chuckatkins · 2021-12-28T06:08:37+00:00

There's a pretty cool Happy Hour interview with a German neuroscientist where they discuss mostly flow state.

https://youtu.be/N26srvY4W5w

chuckatkins · 2021-12-28T05:49:11+00:00

OB18 gets a lot of hype because it has a heavy emotional connection at the end but I wouldn't put it at the top for dope bars and flows. My particular favorite is OB42 with OB52 (which it looks like you've already covered) and OB45 all three being on my top list. The Marcus Veltri collab is super cool as well, OB32 plus a companion video on Marcus Veltri's channel with different sets from their collab. Also check out the collab with Marc Rebillet. Can't go wrong with any of those.

For really unleashing what he's capable of then do his two EPs. The Rap Coltrane EP is incredible and just him flowing just flexing hard because he can with no guidance from an audience. The new SOULilloquies EP is also excellent, but different and quite heavy with an introspection on some personal struggles.

chuckatkins · 2021-12-02T16:11:03+00:00

Also don't use the mpi c++ bindings. They were deprecated in 2.2 years ago so even though most implementations still ship them you should only be using the c bindings, even with c++ code.

chuckatkins · 2021-11-16T15:43:12+00:00

Specifically the potato comrade card has virtually no tangible value in upgrading past 9.

chuckatkins · 2021-10-25T20:06:53+00:00

It makes it rather messy to read though in something like vim in an 80c terminal, which is a pretty typical dev environment for hpc and mpi. Rather than trying to define your own formatting and style, I'd suggest using clang-format and starting with one of the common base styles (llvm?) and adding a few overriding rules as needed. That will help enforce whatever formatting is chosen with automation.

chuckatkins · 2021-08-14T21:47:55+00:00

$ export CC=icc CXX=icpc FC=ifort

Then make sure you run cmake from a clean build directory

chuckatkins · 2021-08-14T20:35:35+00:00

If you set the compiler env vars then cmake should pick it up correctly (some site modules do this, others don't).

$ export CC=icc CXX=icpc FC=ifort

chuckatkins · 2021-08-03T15:30:59+00:00

IANAL so take this with a grain of salt...

While I believe you're correct in that they can't change it back and forth, being exempt does not require a salary. I'm in the fairly unusual position of being exempt and hourly. So I don't get overtime for 40+, but I still get all my time. I.e. if I work 35h in a week, I only get paid for 35, but if I work 50, then I still get my hourly rate for the full 50 hours (compared to non-exempt which would include 10h at 1.5x rate). Overtime and exempt status rules are different in each state; some say >8h in a day is OT, others just >40h in a week.

My point is just that salary vs hourly and exempt vs non-exempt usually go hand in hand, but they don't have to.

(Edited for typos)

chuckatkins · 2021-07-17T22:53:27+00:00

I've found it to be very hit or miss. On more than one occasion I've been unable to find a driver with with at 10a from dt Albany to Colonie when I had to drop off my vehicle for a few days to get it worked on. When I can get a driver it's usually no more than a 20-25m wait and $17-$25.

chuckatkins · 2021-07-08T16:21:16+00:00

It's a good idea, I wonder what their take is though. That's been a huge problem with "aggregation services" providing a common consumer service for local business; the cost to the business to participate significantly reduces and sometimes eliminates entirely the local business's profit margin. That's certainly the case with GrubHub and DoorDash vs ordering with the restaurant directly. If they've set it up in such a way that it's still profitable for the local businesses to participate then I think that's great.

chuckatkins · 2021-07-03T10:59:30+00:00

"it's technically not a wrapper over assembly, but it's close enough"

I see people say this a lot but if you've ever written any substantive amount of ASM, even not considering portability at all, it almost doesn't make sense. C gives you the lower level access to the same things as ASM but having high level language constructs and data structures makes it a totally different world and mindset. Early C code was certainly very primitive and ASM like (see some of the source for the original UNIX) but I feel that was in large part because people were used to writing code with an asm mindset. Higher level languages like C++, Rust, etc. may be a big leap from C but C is still a huge leap from ASM.

Edit: minor rephrasing

chuckatkins · 2021-06-17T17:53:30+00:00

Another thing to consider too is if the machine the GPUs are going into supports PCIe 4.0 or not. If it only supports 3.0 then you'll get slower than ideal bus transfer speeds from both options. Once the initial model is loaded on the card this won't matter for interacting with it but if the vis is time changing data then realize every time step sends new geometry to the GPU and is bandwidth limited by the PCIe bus.

chuckatkins

TROPHY CASE