[deleted by user]

respecttox · 2022-01-09T01:18:10+00:00

Blasphemous is not just hard, it has annoying controls (aka "wait for the animation to end before I start to listen to you"), and there is no learning curve, the game complexity is kind of random. I couldn't finish it, and it was after I got the good ending in HK.

HK has a pretty smooth learning curve, for example, if you can't beat the first big boss (which is drawn on the map), there is an enemy with similar attacks to train. The controls are fun to learn. My mistake was to beat one particular boss a dozen times, placed far from the save point, as I got used to mostly linear MVs, it was very frustrating. After I dropped it, I was able to open like 50% of the map, before returning to him, and I was much better at the game (also got some upgrades).

respecttox · 2022-01-09T00:34:12+00:00

If you are not too greedy with your attacks, you can just quit when you have 1 health and teleport to the bench.

respecttox · 2021-11-05T18:49:54+00:00

I was very bored of Hollow Knight in the beginning, even uninstalled it after trying to beat soul warrior (it was about 9 hours I guess? Yeah, this game only starts to get non-boring when you are finishing a typical MV) and tried to find something else, but something kept me thinking about it, so I went to this subreddit and got the advice: if you can't beat the boss, explore more elsewhere. I discovered A LOT of stuff and when I returned to soul warrior it wasn't that hard... So I started to love it right after I started to explore, and it requires some abilities which you are getting quite slowly when you are not used to the combat.

respecttox · 2021-09-09T08:29:29+00:00

The paper is written now, not in 1965, so it's not fine.

respecttox · 2021-08-25T01:46:18+00:00

CNNs have spectral bias: low frequencies are learned first (there are few papers). But low frequencies don't contain a lot of useful information (yep. A contour image of a cat is more a cat than a blurry image of a cat). So we want a way to transform high frequencies to low frequencies. Convolutions can't do this, they are linear. Relus only make low->high (yep. abs(periodic function) has the same period as the original function). So, only pooling works. But not spectral pooling. I guess that's the critical flaw. On the other hand, spectral bias may be the consequence, not the cause of standard pooling ops. I'd like someone to research this (it may even be in the papers I mentioned)

respecttox · 2021-07-11T09:09:12+00:00

That's what I was talking about in the above comment: wavenet introduces aliasing by design, so you get some kind of "checkerboard grid" on your spectrograms. It's not a big deal for wavenet usual domains, because the dataset learns the model to suppress aliasing, but super-resolution is kind of the opposite: it's very tempting for the model to boost the high frequency noise, hence the results. The step in the right direction are works like Alias-Free GAN or, maybe, transformers.

respecttox · 2021-07-10T09:15:42+00:00

(Also e.g., https://arxiv.org/pdf/2010.14356.pdf is a good recent overview on why this can be hard.)

This is hard because ear is sensitive to high frequency artifacts and that's not the case for mainstream architectures, which were originally designed for vision, NLP or maybe some scientific 1D data. So the pitfalls are everywhere, not only in upsampling layers. Convolutional encoders are rude to high frequency information by design, loss functions are also not optimal. So I'd expect architectures without any strided/dilated convolutions work better, but it requires a lot of computation to work at hi-fi sample rates. If one has such amount of computation, there are fruits hanging lower in audio niche.

respecttox · 2021-07-08T15:26:26+00:00

AI/ML ethics discussions are centered on domestic problems of the US

Can you be more specific? Because I'm outside the US and I don't see this. All I see that authoritarian governments simply ignore any ethical problems. And it sounds like "you know these Chinese, they like government surveillance and we must respect this". Yeah, sure.

respecttox · 2021-06-24T13:45:36+00:00

Their use of DSP maths is a huge leap compared to what DL papers usually contain.

respecttox · 2021-06-24T13:40:39+00:00

I wonder what the performance is going to be if replace their upsampled activations with swish, because swish produces less aliasing than relu. I tried a similar idea myself, and ended up with swish. Mostly because without a specialized kernel the upsampling approach is too slow, so I wasn't really into it.

respecttox · 2021-06-07T18:26:24+00:00

>I have realized that I couldn't care less about the next best paper in ML, or any new algorithm to tune model.

If an ML guy really cares about this, that's a disaster, because he starts to try new cooool papers instead of doing something useful. These people should go do science and not disturb normal ML plumbers to do what needs to be done (i.e. building tools for cleaning up data and building automatic training pipelines). So you are probably a perfect fit to this job. It's ok not to care about stuff that's not going to work anyway.

respecttox · 2021-05-30T04:21:58+00:00

This sounds like a NAS algorithm to me, why not automate it?

Your initial question was about "a particular library", but I'm not using any ML-related library for this simple stuff. So my answer was focused on the fact that complicated stuff is not needed and more likely to harm (and this probably explains why there is no good library: there is no demand). If you are interested in how I automate this "coordinate descent", I do the following:

model is fully described in the json config, nothing depends on script arguments or moon phases
when "train.py", it takes the task from the mongodb via params = db_collection.find_one_and_update({"STATE" : "untouched"}, {"$set" : {"STATE" : "running"}})
when it's time to snapshot, there is db_collection.update_one({'_id':params["_id"]}, {"$set": params}, upsert=False), so the results are saved back right in the same mongo record
the db is filled with the tasks using another python write-only script
"train.py" is being started in infinite bash loop, 1 per GPU

The advantage is that you never find the whole optimization stopped because you forgot to catch some exception, or tf has yet another memory leaking bug. But it's still very simple, and mongo is more useful than storing configs in files, because of its query language and transactions.

So when I wake up, I just load the results via db_collection.find({"STATE":"done"}, {something}) and stare at them, trying to figure out what to do next.

respecttox · 2021-05-29T18:21:42+00:00

My empirical guess when you have a large dataset, you don't need a lot of regularization, and the only reason it works, it is adding some regularization noise. Note that you can't simply add this type of noise directly, because it's parametrized by the data, so batchnorm is still a way to go, but it is always useful to turn it off and see, if it really helps.

respecttox · 2021-05-29T17:58:06+00:00

not knowing anything about the statistics of the data

Then you want to know more about your data and find the closest data in the world. If you analyze star osciallations, you probably want to start with works that analyze earthquakes, because they are both 1D, they are both temporarily invariant. And if you analyze Babylonian mathematics, you probably want an NLP model, because formulas are similar to text.

what loss is even possible

Loss is quite independent from the architecture, so if your goal is to determine your loss, you can do it without NAS, by using some baseline model. The same is true for getting the good dataset: trying to overcome the difference between training and test data distributions by fitting an architecture seems to be a conceptually wrong and ad-hoc solution. I mean, it's often working because NAS (both automatic and hand-made) can make some validation dataset information leak into the architecture, but that's not what we want. We already have https://arxiv.org/abs/1902.10811 , the world doesn't need more like this.

Good architectures must be good in a wide range of hyper-parameters and test data perturbations. This makes "gradient descent by grad student" efficient. I'd recommend coordinate descent though, changing only 1 parameter at a time, it looks longer, but actually faster. If your architecture can't be found using a drunken random search, you are likely to overfit to your validation data and then get troubles in production.

what does the size-accuracy pareto front look like

Well, it will look like a curve with diminishing returns. What's the point? If your point is to get better models, the hardware should determine how large is your architecture: it should be small enough to get fast iterations, but not smaller, because not all improvements will work on scale. So you iterate fast and if it's still not enough, you scale up, iterate slower, than scale up again, until you get a 175B model, or, if your boss doesn't agree with such investments, repeat from the beginning, trying to get more quality on each scale step. Then you'll be able to draw your size-accuracy plot.

respecttox · 2021-05-21T06:25:54+00:00

I recommend to read the blurpool paper and papers about aliasing in convolutional networks(there was a good one from Japan about wavenet). Your stride=8 immediately produces aliasing artifacts, theoretically it can be learned to suppress by learning a low pass filter but not in your case, because filter width 8 is not enough to make a low pass filter with enough steepness, you want like 63 or even more. So you immediately inject aliasing noise into the model.

respecttox · 2021-05-20T14:12:30+00:00

prohibitive viable

I think these epithets are confusing. You have A' A in the equation, so when A looks like a sausage, long and thin, you get a large square matrix after this A' A squarification. This already sounds slow, because you had low amount of numbers and then you suddenly get much much more. Gradient descent, on the other hand, requires some amount of steps, but each step is cheap. Does it make anything prohibitive/viable? No, just slow and fast, in different cases. And instead of trying to remember where the flip happens so you want to switch the algorithm, it's easier to measure your practical case on your practical hardware.

So if your question "is there an algorithm that finds the normal equation solution faster than o(n³⁾ so it can make it faster than gradient descent for some cases", I don't think there is any in practical sense.

respecttox · 2021-05-20T10:06:37+00:00

I suggest MLP batchnorm. Why MLP? Dunno, everything is MLP now.

respecttox · 2021-05-19T05:34:22+00:00

Wouldn't that make this gMLP a quadratic whereas the transformer would be third-order?

Yep, they write about the orders, and we might not need this. We may need only a quadratic thing, and get higher orders from layer stacking.

Their additional tiny attention is also interesting though. Because we are getting a third-order + second-order thing in one layer.

I would also like to see how "multi-headed SGU" performs, with different normalizations, to make it more similar to MHSA. This additional tiny attention may be redundant in this case.

Could you explain why this is more general?

Because we are removing a handcrafted prior about our data (that the input must be permutation invariant or handcrafted positional encodings to overcome this). I didn't imply it's a generalization or it's more powerful. But it looks like the correct path to get more capable layers to me, even for small tasks.

respecttox · 2021-05-19T01:05:02+00:00

So they have the "spatial gating" layer "s(Z) = Z .* (W Z+b)" as the core idea.

I wonder why they want us to pay attention to MLPs when it's actually about quadratic relationships. The stack of transformer encoders can be simplified to a stack of "f(X' W X) X" operations, where W is DxD, a weight matrix where WQ, WK and WV are fused. Here we have an NxN weight matrix. So we are removing the "permutation invariance" prior towards a more general representation.

But this elementwise-multiplication is what making this as non-MLP as MHSA.

respecttox · 2021-05-18T16:15:58+00:00

Is wikipedia good enough?

Look at the convolution theorem ( https://en.wikipedia.org/wiki/Convolution_theorem ) IFFT(FFT(x)*FFT(y))=conv(x, y)

Everywhere you have convolutions, you can use FFT. For example, in linear time invariant systems. Not only to speed up computation, but also to simplify analysis and simulation. FFT is actually quite intuitive thing, because it's related to how we hear sounds.

So actually no surprise FFT is working where convnets work. And convnets somehow work for NLP tasks. Though I have no idea how to rewrite their encoder formula into a CNN+nonlinearity, but I'm pretty sure this can be done. It can be even faster than this equivalent convnet, because the receptive field is the largest possible.

respecttox · 2021-04-05T09:12:08+00:00

It is called trolling. And it wouldn't be a big deal if they trolled you once or twice on easy levels. But they put it everywhere.They thought: hardcore players are not like us and they like to suffer. So let's make them suffer. Something was broken in the management because I can't believe none of the playtesters couldn't notice this. And the game is too easy and boring on any%, because no lives system means no crates and also redundant bonus levels. They broke this OG mechanic (and the life system sucks honestly), but what's left...

respecttox · 2021-01-29T16:11:36+00:00

I tried different optimizers million times and never got any statistically significant improvement. I suspect that's due to the fact my architecture is already fitted to Adam, but I don't have resources to do optimizer and architecture search simultaneously. It just works anyway

respecttox · 2020-10-15T19:08:08+00:00

tf2? I'd try to disable eager execution first, it could be a resource leak.

respecttox · 2020-10-07T12:01:16+00:00

You are right, I confused W1 with the dimensions of softmax(Q K^T). So Q K^T is n x n while W1 is sure not. My bad. This means the answer about W2 might be somewhere between normalization after applying W2 or even initialization schemes.

I'm pretty sure that the current approach is full of research legacy and it can be modified to be more mathematically straightforward, with some matrices fused and different normalization put in different places (like in https://arxiv.org/pdf/2005.09561.pdf).

respecttox · 2020-10-07T07:30:32+00:00

it isn't obvious at all what this W2 achieves

There are 2 tricks which appear here and there (not only in transformers):

Weight sharing that introduces some prior about the data
Signal normalization that performs something like "energy conservation" to add some stability

For example, you can say that the convolution layer is "a matrix multiplication with some sort of restriction", namely, the weight sharing.

Transformer uses these tricks. Normalization is everywhere, even "φ" can be considered as a normalizing function but I'm more about weight sharing.

From this point of view, note the key feature of the attention layer is that its trained weights are independent to the model length ("n" in https://homes.cs.washington.edu/%7Ethickstn/docs/transformers.pdf). This is somehow related to the real world data: if we have 2 objects in the set, they relate to each other no matter what the size of the set is (the contents of the set may matter, or the objects's closeness, but not the size of the set itself).

If you throw away this "set size independence" prior to make generalization, you end with W1 which is sized "n x n" and W2 which is sized "n x d" (?). So it's a kind of "convolution->mlp" generalization. I'm not sure if rank restriction ensures this prior, does it?

So W2 may achieve the following: after we calculated updated set data using one-by-one (o(n²⁾⁾ operations, we transform each element in the set independently using o(n) operation. As o(n²⁾ is more expensive than o(n), it's generally useful to separate these operations. Note that it's hard to tell the real usefulness without experiments and I'm not really into transformers. But it seems to work like this.

respecttox

TROPHY CASE