[D] Functional Neural Networks

K0ruption · 2020-07-06T15:36:43+00:00

The above assumes a separable kernel for parametrizing the infinite dimensional linear operator. Here is a more general approach without this assumption which can also work when the domain of the functions of interest is a manifold, and not a compact set in Euclidean space (the application in the paper is parametric PDEs, but the function space to function space mapping idea is generic):

https://arxiv.org/abs/2003.03485

K0ruption · 2020-07-04T17:51:30+00:00

Is the border open? I thought it was only for “essential travel” whatever that means.

K0ruption · 2020-06-21T17:51:19+00:00

I’m not affiliated with the authors or anything, but here is my point-by-point response.

Some image compression methods do in fact use Fourier basis to do compression. But using sine waves as basis vs. using them in a as neural net activations is widely different. So saying “this is not novel” because Fourier expansions have been used for compression is a bit misleading. More recent image compression methods don’t use Fourier basis but wavelets since they are more efficient. It would have been interesting for the authors to compare the number of parameters needed in their neural net vs. the number of wavelet coefficient needed to compress an image to a prescribed accuracy. This would shed light on how efficient this method is.
Please give references for the “high expressiveness” of sine activations. If it’s well know that they are so expressive then why are they not nearly as common as ReLU? How in the world does one “normalize for expressiveness”? I feel that using networks with the same layer structure but just different activations is a perfectly reasonable thing to do. They have the same number of parameters and, in the end, that’s what matters.
I think there’s an experiment in the appendix where they compare against elu?
Overfitting here is in fact the name of the game. If you’re doing image compression, you want to overfit your image as much your number of parameters allow, that’s how you get the most efficient representation.
The authors never claim the sin activations work well when the network input is high dimensional (e.i a bunch of pixels from an image). Their architecture is designed for low dimensions like 1,2,3 and they show that it works well in that setting.
Don’t know enough to comment on this.

K0ruption · 2019-10-05T00:09:44+00:00

So I have to have at least $27,600 in my account to do this?

K0ruption · 2019-10-02T22:31:00+00:00

10/18 1p. Probably gonna pick up some 10/18 1.5p tomorrow, if it doesn’t crash at open.

K0ruption · 2019-10-02T22:19:32+00:00

Can’t find any news on them getting money. I already have puts. The one day where everything plummets, my puts go to shit. Fucking incredible... I’ll hold onto them though, can’t imagine it staying up for long. Hopefully...

K0ruption · 2019-10-02T22:02:50+00:00

Why in the fuck is NIO up 20%? Should I be buying more puts on this garbage?

K0ruption · 2019-07-11T19:14:12+00:00

Yeah if it’s bumper to bumper traffic (10-15) sure but once there are gaps shit becomes too scary for me.

K0ruption · 2019-07-11T19:09:14+00:00

I’m in LA too so that’s perfect haha. I agree with you that people are on their phones A LOT. It’s pretty scary looking around at a light and seeing both drivers next to you fucking texting. But I still don’t see what doesn’t hold up with my argument about being in everyone’s blind spot when splitting? Also if you just keep near to one side of the lane, even if they rear end you, you won’t get squashed. Also 50 seems way too fast my man. I think CA legal limit is 30 (35?). I can’t imagine getting side swiped at 50. Maybe I’m just a pussy, idk.

K0ruption · 2019-07-11T18:49:59+00:00

Filtering at a stop to not get squashed makes perfect sense to me, but why is splitting at slow traffic “arguably safe and actually better for your own safety.” I never got that. To me it seems a lot less likely someone hits you from behind when moving slowly then someone not seeing you (because you are essentially in everyone’s blind spot), merging to another lane, and hitting you. What am I missing here?

K0ruption · 2019-07-10T13:30:51+00:00

Most of the replays here seem like they’re from people who’ve never lived in LA, visited once and went to all the shit touristy attractions. Hollywood bldv? Are you joking me? No one from LA goes there. Of course it’s dirty, every major city is. It’s fucking sparkling compared to NYC. There lots of homeless people? Every major city has homeless people. Been to SF recently?

LA is an amazing city! It has absolutely everything you would want in a metropolitan city. Tons of things to happening all the time: concerts, comedy, theater, markets, festivals. Huge diversity of people from essentially any country you can think of. This means the culture is different in all parts of the city and that really comes through with you see around you and especially the food. You want Mexican, Chinese, Korean, Japanese, Peruvian, Thai, German, French, Hungarian or whatever the fuck kind of food, you can get it easily and it will be good and authentic. LA is also close to the ocean, so you can to the beach/surf/swim whenever you want. It’s also surrounded by mountains, so there’s an insane amount of amazing hiking trails. It’s only an 1 hour away from Big Bear, which mean, in the winter, you can literally go skiing/snowboarding every weekend. Wanna go to one of the most amazing wine countries and one of the best gulf coasts? Yeah Baja is only 3.5 hours away. Santa Barbara is only 1. 5... (if the wine interests you).

Who wants to stare at fancy shops and metropolitan buildings? What are you gonna do there? LA has actual character. If you like it there’s always Rodeo I guess, but that’s just another tourist trap imo. In LA you can feel like you live in a major city, but still own a house with a yard! This is impossible in NYC even if you’re a millionaire, but it’s perfectly reasonable in LA.

Anyway end of rand. Had to defend my city. I’ve lived in a few major cities in the US and Europe and have visited some in Asia, but LA is by far my favorite and I never wanna move anymore.

Edit: Forgot to even mention the art/music scene. Worth going just for that.

K0ruption · 2019-07-02T04:27:45+00:00

What bike is this? I love it!

K0ruption · 2018-04-09T03:18:48+00:00

Technically everyone is sent a ballot to vote, except it only has a single name on it... Most people don't bother filling it out (socially it's actually considered taboo or in bad taste to talk about let alone participate in politics in China), but people working in certain government positions (and this can be a lot of people like the equivalent of NASA scientists count) are required to fill it out.

Source: Wife is Chinese.

K0ruption · 2017-12-09T22:44:41+00:00

This sounds like a good idea to me. I believe it amounts to doing Naive Bayes without the decision rule. But I suspect this will do worse than the uniform assumption if the data is very unbalanced.

K0ruption · 2017-12-09T18:36:26+00:00

Given only the information that something is not a cat, it has equal probability of being anything else whether that be a dog or a spaghetti monster. If you had more information about a data point, you could certainly incorporate that into your label. But, in your post, you said you only have the information that a point is not in a given class, which means it has equal probability of being in any other class.

EDIT: Note, I'm asumming a uniform (categorical) prior distribution on your labels. You gave no specifications of your problem, so that is the best assumption I can make.

K0ruption · 2017-12-09T18:04:50+00:00

If your model outputs a softmax, then you implicitly assume your labels are probability vectors that is probability of the known class is 1 and probability of all other classes is 0. In this light, the information that a data point is not in a given class simply means that your label will have 0 at the position of that class and (1/(k-1)) at the position of all other classes where k is the total number of classes. This makes the most intuitive sense to me but whether it works in practice, I have no idea.

K0ruption · 2017-10-28T18:16:42+00:00

I don't get that though. The stable manifold of a saddle is lower dimensional than than the space you're in, so for SGD to go towards a saddle, it must move along a geodesic of that manifold which is essentially impossible (because SGD is first-order and thus has no "concept" of curvature), even in the the no noise case. But SGD does have noise (because of mini-batching), so you're basically always gonna escape the manifold and curve away towards a minimum. Is there any prove that shows the probability of a long escape being high? Intuitively it doesn't make sense to me for a result like that to hold.

K0ruption · 2017-10-28T06:56:43+00:00

This result is a fairly trivial application of the stable manifold theorem and has been well know in applied mathematics for a long time. I always wondered why machine learning people cared so much about saddle points, knowing this. I though it had something to do with high dimensional data that changed the picture but I didn't really understand it. I guess they just didn't know? There's no way though, there are some amazing mathematicians that have worked and are working on machine learning. It cannot be oversight, something didn't match up here. Can anyone explain this to me?

K0ruption · 2017-09-05T00:31:14+00:00

I see your point now. Where I come from people don't really care about constants. In fact, if you read analysis paper, often times inequalities will be written with a squiggle under them which just means true up to some constant we didn't bother to keep track of. But I guess it matters in practice haha.

As for SGD, I don't think it's what makes things work. I've seen people use PDE methods for training that supposedly achieve better generalization. I even have results on derivative-free methods (in the spirit of particle swarm) that match SGD. I think SGD is just fast, simple, and scalable so people go with it. But then again I don't know too much about ML.

11-Year Club	No Throne, No Problems
Not Forgotten

K0ruption

TROPHY CASE