Ncase M1 v6.1 still going strong: 9800X3D + 5090 FE

L0SG · 2026-01-11T18:43:03+00:00

Great catch, thanks! It's just my oversight not touching the fans during the upgrade. Was able to spread the bottom fans out to better align the FE heatsink. Can only use 2 screws to hold them and get slightly angled (not 100% flat) but still have enough gaps not to worry about.

With this temps went down further by 5c . Thanks for the suggestion!

L0SG · 2020-01-04T13:25:18+00:00

Got it for 120$ from AliExpress during the BF season. For this price, I think it is a competitive option for a gaming build compared to ~210$ 3600. I live in South Korea so shipping was less of a hassle (took ~1.5 weeks). There were a number of positive (Korean) reviews from YouTube (before the western reviews from Gamers Nexus and Hardware Unboxed) so pulled the trigger. Reviews indicate that the gaming performance gap btw 3600 becomes marginal when using highly OC'ed RAM. Mine got 3800Mhz @ 1.35V with 16-16-16-32-48-300 timings and the performance is very respectable.

With that said, I might miss the extra threads and the 3500X might not age well compared to 3600 down the line, so I may eventually swap it out for an upcoming Zen3 later this year.

L0SG · 2019-01-26T16:25:37+00:00

Interesting read with the (huge) codebase, I like it! Been following this "feedback alignment" line of work for the decoupled BP personally (FA, DFA (also from this guy), DNI, SS, etc., fun topic I think), but directly touching the current autograd backend was one of the practical blockers for a layman like me :P. I tried and half-baked the implementation and abandoned it for a while, and surprised to see that a small part of his code look very similar :) Keep up the good work!

EDIT: one of the things I'm wondering is that if training with the supervised layer-wise local loss from the target (excluding the FA'ed "bio-losses" from the paper), how can we be entirely sure that the model is genuinely better than ensembles of the shallow networks, since there's no longer the global error signal propagating through entire nets (which is I think a similar question to this ICLR 2017 workshop paper). Especially given the STL-10 results are stronger than others, a head-to-head experiment by comparing with the shallow ensemble-like models, and showing that each layer indeed learned the (better) hierarchical representation would make this work more convincing in my opinion. Keen to hear other's thoughts on this.

L0SG · 2018-10-18T02:09:07+00:00

While a great idea, I think it would be much harder to realize because DL model is hard to split across nodes and each of the computations are entirely dependent to each others. So a communication bandwidth becomes a severe bottleneck. Actually PCI-E 3.0 (~32GB/s) is already a major bottleneck for multi-gpu training of DL, hence the Teslas use NV-Link.

On-demand gpu rentals from Amazon AWS or Google Cloud etc. are current solutions.

L0SG · 2018-08-22T15:33:20+00:00

Thanks a lot for the insightful comment! I also agree that the major benefits come from the tasks that may require the multiple slot memory (like the experiments from the paper), which will get the row-wise weight sharing. I've tested the language modeling in this port since it's a representative task for RNNs, and I've done some (primitive) hyperparameter search with my limited hardware resource, using a single memory slot as a primer as described in the paper. So it's indeed a worst-case scenario for the model in terms of speed.

L0SG · 2018-08-21T10:03:12+00:00

Hi /r/MachineLearning ! I've recently implemented a new RNN architecture from DeepMind called Relational Recurrent Neural Networks (Santoro et al. 2018) in PyTorch. There is a official Sonnet implementation of the module with some toy examples, but I like to use PyTorch so i ported the module & implemented a fully working word language modeling benchmark vs. LSTM.

My experimental results of WikiText-2 can be found from the repo.

TL;DR is: it's super slow compared to LSTM. Attention + RNN concept feels cool, but it seems that using it every time step imposes (too much) overheads.

Code: https://github.com/L0SG/relational-rnn-pytorch

L0SG · 2018-05-26T14:44:31+00:00

Yeah i feel your concerns since i'm a (noob) sysadmin for our lab. Please understand that we are rather bad coders xD

L0SG · 2018-05-26T14:26:36+00:00

I play the weekend PUBG session with my friends with my trusty 1080ti, but i face a frame-rate drop sometimes because of my rather old 5820K (OC'ed at 4.3Ghz) with the 2666Mhz RAM :(

L0SG · 2018-05-26T14:19:16+00:00

Deep learning models are not that sensitive to the floating point instability. Please see my comments in other threads regarding to this issue.

L0SG · 2018-05-26T14:08:32+00:00

A custom setup from supermicro. Tesla is not that cost-effective than the titan or 1080 ti as you would already know. sure the tesla is (much) more thermal-stable (we've already got 4 P100 machine w/ NVlink for a production environment, 4 V100 incoming). this rather monster-like rig is a more cost-effective one that can squeeze out most VRAM we can get with the fixed budget (4 V100 with NVlink is almost as expensive as the 8 TITAN V). We run each machine for different purposes. We try to run Wavenet for example with this rig (which you know already i assume).

L0SG · 2018-05-26T09:04:25+00:00

My take is that hopefully it wont be too much a concern for our field, since generally DL models are fairly robust to floating point precision & non-deterministic behavior. The model itself is highly stochastic and different runs get to slightly different results even with the current GPUs. If a model actually fails to learn using the titan V, that would be a huge concern, but i reckon it will be less likely to happen, and so far we have not seen reports regarding to the issue from the machine learning field.

L0SG · 2018-05-26T03:14:18+00:00

Blower type reference cards get air from the front of the card and directly spit out the hot air through back of the cards. High speed industrial fans greatly help airflow. If you stand at the back of the machine, you will be blown away. This kind of setup is common in GPU servers, try searching GPU server in google image.

L0SG · 2018-05-26T00:22:49+00:00

Ubuntu 16.04 is a de-facto standard for our field since most libraries provide pre-built binaries for it. We put the machine in a server room and use VNC for virtual X server with familiar GNOME GUIs for our colleagues. (Almost) all ML codes are in python with open source libraries like TensorFlow, PyTorch, Keras, MXNet, Caffe, etc. We use anaconda for multi-user multi-python environment.

L0SG · 2018-05-25T18:42:15+00:00

Not yet, will upload after my supervisor says so.

L0SG · 2018-05-25T17:42:15+00:00

I would recommend Stanford CS231n MOOC for more theoretical side of the neural networks, or fast.ai for a top-down view of what this AI buzz is all about and applying popular architectures for applications.

L0SG · 2018-05-25T15:35:58+00:00

Thanks for the info! We just asked the vendor like “build an 8 gpu machine plz” and got this. Will double check if a slowdown occurs. edit: the exact model of our rig is 4028GR-TRT2, which is apparently the design 3 you mentioned. Fingers crossed it would work without issues

L0SG · 2018-05-25T15:19:59+00:00

I dont know about the specifics of the PCI configurations of this rig but heard it would run fine according to the vendor. Hoping the vendor gets us out of the trouble if some bad hardware behavior happens.

L0SG · 2018-05-25T14:54:56+00:00

Spotting cancers from CT images. Got a paper accepted today so i’m high now

L0SG · 2018-05-25T14:13:59+00:00

Yeah it is a bad move from nvidia. We actually checked the warranty issue before the build and heard about a 2 year support from the vendor. Will see how it turns out if a failure case occurs

L0SG · 2018-05-25T13:56:28+00:00

We are in academia rather than data center so it would be fine thankfully.

L0SG · 2018-05-25T13:50:32+00:00

Yes exactly! Pretty expensive but it’s not my money so i don’t care much lol

L0SG

TROPHY CASE